I'm using express-rate-limit npm package, I deployed my backend on AWS (t2 micro ec2 instance), while limiter is on, requests are blocked from ALL users who try to interact with my API, it works for a couple of minutes and stops for about 10 minutes.
when I comment out the limiter part everything is working fine,I think too many requests should be blocked for only one user who tries to hammer the server with requests but what happens is ALL users get blocked, all users are treated like only 1 user, that's my conclusion.
If that's the case what should I do? I need my rate limiter on, and if there is any other explanation what would it be?
By default, express-rate-limit has a keyGenerator of req.ip. When I log this on my server it is '::ffff:127.0.0.1' which is obviously going to be the same for every request, thus limiting for all IP addresses once it's limited for one.
My solution was to use request-ip to get the correct IP address like so:
const rateLimit = require('express-rate-limit');
const requestIp = require('request-ip');
const app = express();
app.use(requestIp.mw());
app.use(rateLimit({
windowMs: 60 * 1000, // 1 minute
max: 30, // limit each IP to 30 requests per windowMs
keyGenerator: (req, res) => {
return req.clientIp // IP address from requestIp.mw(), as opposed to req.ip
}
}));
keyGenerator: function (req: any) {
return req.headers["x-forwarded-for"] || req.connection.remoteAddress;
}
It blocks based on iP
The express-rate-limit package blocks requests based on IP Address and that's because it provides a very basic configuration for rate-limiting that would be suitable for most applications. If you block based on user, someone can easily configure a bot to hit your APIs until the limit is reached on one user account and make a new account automatically to start hitting your server again. Blocking based on IP avoids such risks as one IP means one Device no matter how many users request from that IP. In most cases, one device is most likely to be used by one person so this solution works pretty well.
Related
I have created an express node.js API, and deployed it to AWS (Elasticbeanstalk with 2 EC2 instances).
I am using the morgan-body package to log the requests and responses on my endpoints, but it seems that tons of bots are "attacking" my API, and this results in millions of logs every months, which cost me a fortune with datadog.
I have used morgan-boday's built-in "skip" feature to filter requests based on the user agents, but new ones seem to appear every day.
Is there a way to skip logging for all kinds of bots, without checking them one by one ?
Here is my code, many thanks for your help ! :)
morganBody(app, {
skip: (req, res) => {
if(req.get('user-agent')){
if (req.get('user-agent').startsWith('ELB-HealthChecker') ||
req.get('user-agent').startsWith('Mozilla') ||
req.get('user-agent').startsWith('Mozlila')||
req.get('user-agent').startsWith('Python')||
req.get('user-agent').startsWith('python')||
req.get('user-agent').startsWith('l9explore')||
req.get('user-agent').startsWith('Go-http-client')
) {
return true
}
}
return false},
logRequestBody:false,
logResponseBody: false
});```
Welcome to internet. Bot/Spam detection is one of most trivial problem to solve. Every logic you add can be negated by reverse logic at the client side.
AWS itself has a tool for it.
https://aws.amazon.com/waf/features/bot-control/
A good strategy to filter traffic will be based on use case.
Some suggestions.
introduce login/session allow only authenticated session
request headers filtering
Ip ranges filter
Amount of traffic from single i.p.
Request rate from different IP etc.
Take service offline when not required.
There should be more material available on internet.
I figured out part of the answer, by simply skipping all GET requests:
if (req.method === "GET") {
return true
}
But I am still getting some POST requests by bots which increase my logs volumes and I still do not know how to filter them...
Thanks if you have an answer !
I can't find a resource for this anywhere online, all I see is references for nginx.
I need help with this quickly as my server is live with users accessing it and somehow google indexed my ip address and users are accessing my site through my ip.
I plan to migrate servers tonight and am aware of why my ip was indexed, but in the meantime need a method to prevent direct access via my ip.
This obviously isn't working, and don't have much room to test, unless I stop the server and kick all of my users off for an extended period of time:
app.get('myiphere', function(req, res){
res.redirect('domain.com');
});
You can implement an application-level middleware which checks that a request host-name isn't anything else but your domain. That way an access to your with an IP address wouldn't cause a processing (on application level).
const SITE_ADDRESS = 'yourwebsite.com';
app.use((req,res,next) => {
if (req.hostname.includes(SITE_ADDRESS))
next();
else
res.status(403).end(`Access with ${req.hostname} is restricted. Use ${SITE_ADDRESS} instead.`);
});
To prevent direct access to your site from IP you can set the loopback IP this way:
app.listen(3000, '127.0.0.1', () => console.log('Server running on port 3000'))
Prevent indexing by creating a robots.txt at your server root directory. See https://stackoverflow.com/a/390379/11191351
I am trying to solve the issue of Slow Post Vulnerability on my application.
Issue: https://blog.qualys.com/securitylabs/2011/07/07/identifying-slow-http-attack-vulnerabilities-on-web-applications
To limit the number of connections from a user, I have used express-rate-limit so that the application does not go unavailable.
const rateLimit = require('express-rate-limit')
const limiter = rateLimit({ windowMs: 60 * 1000, // 1 minute max: 100 // limit each IP to 100 requests per windowMs })
app.use(limiter)
But If I try to test my application with slowtesttool and run a test with 2 connections (with rate 1 connection per sec and follow up data every 10sec), I see the connections never get closed.
I have set timeout to the connection as below, but it doesn't seem to work!
app.use((req, res, next) => {
req.connection.setTimeout(30000, () => {
req.socket.end()
})
next()
})
Is there a way I can limit the rate of accepting data, i.e. specifying the max time I can wait for every next chunk of body?
One solution could be to use the capacities of your front webserver (I assume that you will expose your app behind a server such as nginx, apapche, caddy, ...).
Nginx and caddy have this built-it, others probably too.
I'm writing an api using nodejs and express and my app is hosted by openshift free plan.
I want to protect my routes from brute force. For example if an IP sends more than 5 requests /sec then block it for 5 minutes. :)
There's nothing stopping you from implementing this in Node.js/express directly, but this sort of thing is typically (and almost certainly more easily) handled by using something like nginx or Apache httpd to handle traffic to your app.
This has the added benefit of allowing you to run the app entirely as an unprivileged user because nginx (or whatever) will be binding to ports 80 and 443 (which requires administrative/superuser/whatever privileges) rather than your app. Plus you can easily get a bunch of other desirable features, like caching for static contents.
nginx has a module specifically for this:
The ngx_http_limit_req_module module (0.7.21) is used to limit the request processing rate per a defined key, in particular, the processing rate of requests coming from a single IP address.
There are several packages on NPM that are dedicated to this, if you are using the Express framework:
express-rate-limiter
express-limiter
express-brute
These can be used for limiting by ip, but also by other information (e.g. by username for failed login attempts).
It is better to limit rates on reverse-proxy, load balancer or any other entry point to your node.js app.
However, it doesn't fit requirements sometimes.
rate-limiter-flexible package has block option you need
const { RateLimiterMemory } = require('rate-limiter-flexible');
const opts = {
points: 5, // 5 points
duration: 1, // Per second
blockDuration: 300, // block for 5 minutes if more than points consumed
};
const rateLimiter = new RateLimiterMemory(opts);
const rateLimiterMiddleware = (req, res, next) => {
// Consume 1 point for each request
rateLimiter.consume(req.connection.remoteAddress)
.then(() => {
next();
})
.catch((rejRes) => {
res.status(429).send('Too Many Requests');
});
};
app.use(rateLimiterMiddleware);
You can configure rate-limiter-flexible for any exact route. See official express docs about using middlwares
There are also options for Cluster or distributed apps and many others useful
I need to make download script by NodeJs,which limit connections number and speed for each session.
I can read file and write is to response by NodeJs, maybe I can limit speed by sleep package of npm, but how should I support more than one connection to download file in NodeJs?
for e.g when user download it by IDM he/she can see 7 or 8 connection during download.
In apache it can do it by mode_limitipconn and bw_mod
I mean something like RapidShare which limit speed and connections number for each category.
What you want to do is limit access to a resource to only one IP address at a time.
This is fairly easy to do with Node since all requests are served from a single process.
You can enter IP addresses into an object when the request is made and then check that object when new requests come in for duplicates.
If you're using Express and you have your routes in modules, you can put the IP connection object in the top level of the route module.
var connectedIPs = {};
exports.myDownloadRoute = function(request, response) {
var IP = request.connection.remoteAddress;
if(connectedIPs.IP) {
response.redirect("http://mysite.com/download_rules.html");
return;
}
connectedIPs.IP = true;
// pseudo send a file call, replace this with your code
send_a_file(function(err) {
// done sending or error, remove from connectedIPs
delete connectedIPs.IP;
});
}