my case is simple:
I need an application layer solution to identify and then apply some sort of rule to requests coming from the same origin.
If a guy will request my server from Postman, or from a browser or from a cURL I want to identify this guy and then do something with this information.
In my particular case I want to blacklist a guy who would be attacking my server for sometime.
Is it possible in Node/Express?
There is no uber identifier that comes with a web request that tells you who the user is behind the request, no matter how the request was initiated (browser, cURL, Postman, node.js app, PHP app, etc...).
This question comes up pretty regularly among new web developers. In the end it boils down to two things:
Requiring users to have an account, login to that account in order to use your service, requiring login credentials with every use of the service and then tracking their usage to see if it meets your usage guidelines. If it does not, you can ban that account.
Rate limiting users either by account or by IP address or some combination of both. If they exceed a certain rate limit, you can slow them down or deny access.
A browser provides a cookie so you can attempt to identify repeat users via browser cookies. But, this can be defeated by clearing cookies. Cookies are per-browser though so you can't correlate the same user across multiple devices or across multiple browsers with a plain cookie.
cURL and Postman don't provide any identifying information by default other than the originating IP address. You can attempt to track IP address, but there are some issues with relying only on IP address because corporate users may be going through a proxy which makes them all appear to come from the same IP address. If you ban one user for misbehavior, that may affect lots of other innocent users.
If you look how Google, Facebook, etc... do this, they all require you to create some sort of account and then provide credentials for that account with every request. This allows them to track your usage and manage your traffic if needed. And, for free usage, they generally all have rate limits that limit how frequently you can make API calls. This prevents any single user from using more than an appropriate share of the load of the service. And, it allows them to detect and regulate accounts that are abusing the system.
One step further than this concerns how an account is created because you don't want an abuser to just be able to run a script every 10 minutes to automatically create a new account. There are a variety of schemes for protecting this too. The most common is just requiring some proof that a human is involved in creating the new account (captcha, question/answer, etc...) which prevent automated account creation. Other checks can require a valid credit card, unique email address verification, etc...
Related
I have api which will return user status i.e exists or not exists in the shop against email. Now the challenge here is to rate limit any specific user/bot which is sending multiples request. One solution is to use advanced cloudfare rate limit but its only support IP counting in core subscription which we have already but IP counting is not a good solution at all since it can be send from any corporate LAN having multiple users or proxy servers.
While if I go for normal server side solution we have node modules like express-rate-limit but I still think the user is coming to our server and then we are blocking them.
Not sure if we have any best solution on cdn level. Also how can I track a user request uniquely with IP address which attributes I can use.
I have a little application which needs to retrieve some public pages info by calling instagram.com/[username]/__a=1 .
my application was working for about a week and suddenly instagram changed its way and instead of retrieving JSON data, it just redirected me to login page.
I can run my app in local... so I thought my server IP is banned or marked somehow.
I have used multiple proxies but it seems those IP addresses were banned too.
finally I used a valid sessionid and my application is working properly.
is instagram going to ban my account ?
if it is, how should I get public pages ?
I have read about depreciation of instagram api and new instagram graph api, and it appears to me that old api which have capability of satisfying my need, is going to be disabled in 2020, and new api platform is only for Business and Creator accounts. am I wrong ?
if I am right, what should I do then?
I have experienced exactly the same. Right now I managed to circumvent this issue by letting my applications users be the ones who request the JSON data (through a $.getJSON jquery request), as opposed to my server being the one doing the request. Since every user has a unique IP address, instragram lets all their requests go through. I then take the data their request receives and forward it to my server.
Right now, probably due to extensive testing, Instagram even blocked my home IP address, so when I'm on my home WiFi I cannot access the instagram JSON feed but when I switch to my phones 5G internet, I am able to do JSON requests for public profiles.
It is sad that Instagram has to block requests to public profiles this way, but I guess if you don't make too many requests from the same source or use dynamic IP-addresses you should be fine!
I’m building an application and I’m thinking about asking user phone number to send a verification SMS. Though, imagine if the phone number is cancelled and attributed later to someone else. Then, the new person would be able to connect to my app in the name of the old one...
So is there any way to prevent this behavior ?
I want to make it like tinder : sign up possible by 2 different ways : (facebook connection and phone number) or (phone number and mail)
I have another question : I see that many sms sending services are not free (all of them actually). If I make an api with these services, anyone can send a lot of http request to it and make me pay 0,05€ times 100000000 ? And I can’t rely on IP adresses because with 3G an ip is not associated with a particular person...
To your first question:
You are describing Two Step Authentication (aka Two Step Verification) which you can read about in the Wikipedia page: Multi-Factor Authentication (MFA):
a method of confirming a user's claimed identity by utilizing something they know (password) and a second factor other than something they have or something they are. An example of a second step is the user repeating back something that was sent to them through an out-of-band mechanism.
You are correct that a phone number can change owners (as can an email address though over a longer time period on average). You are using their phone number as that out-of-band mechanism described above.
If the user has recently authenticated with their password, when you send the user an out-of-band code and they re-type that into an input box you have some degree of confidence that the end user both knows the password and has access to the SMS message and are choosing to trust that association.
You will need to consider if, and for how long, you can trust that association within the security context of the use case.
For example, adding two step verification when detecting the end-user has just authenticated on a device you have never seen before is a nice additional protection. However, using the out-of-band SMS verification in account recover could open up a big security hole. You do not want to bypass the authentication with something they know (password) in a password reset flow by simply having access to that SMS number. SMS is also not an appropriate mechanism for one-time-password (OTP).
If you want to offer you users more protections on their accounts look into implementing true MFA with software tokens (eg. Google Authenticator, Authy, etc.) and hard tokens (eg. FIDO U2F devices such as Yubikey, Google Titan, etc.).
To your second question:
You are correct, IP-based limiting is insufficient. With SMS services you are likely going to be making a server-side API call to the SMS provider. First check to see what security features your provider has out of the box. Next, protect your endpoint that is triggering the API calls to the SMS provider.
Rate limit the number of SMS messages to any one given recipient (eg. no more than X SMS messages to a single number per Y minute window)
Rate limit the number of SMS messages one person can make to different numbers (eg. no more than X different phone numbers per user per day).
Do not allow unauthenticated requests. The user should have already completed the first authentication step (something they know eg. username/password) before performing the out-of-band SMS step.
Protect the SMS form from Cross Site Forgery Requests (CSFR). Your back-end should only make the API call to the SMS provider if it knows the request came from your front-end and not another host.
Protect the SMS form from bot attacks. There are many approaches with Google ReCaptcha being one of the more common.
Some time ago, it was commonplace for smartphone apps to open a browser to a registration page with a CAPTCHA, or to require separate signup via web, because API signup was seen as vulnerable.
Now most apps seem to offer registration via native form, though endpoints for this are usually not documented in their public API. I haven't seen many reports of this being abused to create spam accounts.
How is this done? Is there a standard crypto/handshake process to verify real signups, or does signup typically rely on undocumented endpoints and simple API key passing?
Embedding yields a better experience but has the issue you mention. Yes, the service owners on the other end are still worried about this and combating the problem. And undocumented APIs don't help and the service owners know this.
One of the tools in the toolbox these days is keys assigned to devices which can be used for throttling. This would essentially let you limit the amt of service that can be consumed on a per device basis and it would require you have a device (or can steal the key from one) in order to provide service. So long as the process to issue keys to new devices is strong (a solvable problem) then you can offer a CAPTCHA-free signup experience within the confines of what you are willing to give to a device.
I'd also note that there are other well known approaches you can use, like IP throttling and handshakes with other service providers (like a phone carrier). Depending upon the problem domain these are on the table too...
I'm working on a simple web service that allows users to sign up for free and upload a small amount of data. I can easily establish a quota for each user, but malicious users could create fake accounts to upload as much data as they like in a denial-of-service attack.
Obviously, there's no perfect defense against this type of attack, but what can we do to mitigate this problem?
Tie it to a more-or-less unique identifier (phone number, bank account number, facebook/google/etc account) or to a finite resource (such as time, by using a captcha).
use a captcha on account creation to ensure that it's a
human and not an automated process.
require a valid email address and require that they click a link in their email to validate that that's their email address and continue the registration process. This cuts down on their ability to create many throwaway accounts because you can limit them to only having one account per email address and they have to then create a new email address for each account they want to create.
When the user signs up, the user supplies a valid email. Most accounts are not enabled until a response has been received, usually by clicking a link in the body of that email. When that click-through is received, you should be able to grab an IP address. That should help you curtail an abundance of casual DOS attacks.
Consider Phone Number Verification
Requiring phone number for account creation is the best approach I've come across; Creating a new email or cycling an IP address is pretty trivial, but genuine sms phone numbers cost money to activate & grant your service the ability to restrict access by country-code.
An important caveat: Virtual phone numbers (like google-voice), temporary-phone number services, & burner phones can make sms-verification ineffective at preventing duplicate user accounts. Depending on your use case, it might be worthwhile to use a service, like Vonage's Number Insight api, that lets you identify those types of numbers.
Authillo is a passwordless authentication provider that prevents duplicate/fake accounts by leveraging sms verification, liveness detection, & facial recognition. Depending on how critical it is that you prevent fake accounts on your service, their base plan might be what you're looking for.
Just log the IPs and assume the same user if the IP does not change within a time interval. This is bad, because it would prevent multiple users in the same house (same IP) but it is a good start.