I have a website that has been the frequent target of botters who are attempting to scrape our content, and have been trying to devise a better mechanism of prevent bot traffic. I've conceived the following workflow to protect our APIs and webpages, while still allowing legitimate traffic through with a minimal impact on performance:
When user visits the page for the first time, make a request to the backend using recaptcha v3 to tell if the user is human or not. If they pass the captcha, generate a token for the user, store it in our backend with an expiration date, and give it back to the client to store as a cookie.
On the frontend, include the token in all API requests to our backend. Before handling any request, the backend will check the token and make sure that it is valid and not expired.
If/when the token expires, inform the frontend and run step 1 again to make sure the user is human and generate a new token.
I believe this approach would achieve the goals that I have outlined above. There would relatively few requests to recaptcha (basically one request per session, assuming a token duration of 1 hour, let's say), and there would not be any noticeable change to the user workflow. One limitation I see is that we would want verified bots (GoogleBot) to be able to bypass the captcha, but maybe there is a way to do this through Cloudflare by having it tell us when traffic is coming from a verified bot.
Are there any other limitations/downsides to this approach that I haven't thought of? I tried searching for a solution to bot traffic similar to this, but haven't been able to find anything. It seems like this is not a very common solution, and I'm wondering if that may be because there is a flaw to the design that I haven't considered.
Related
I would like to expose a problem to which I just cannot find a solution, although I have been informed several times on the web, the resources I find do not satisfy my curiosity.
The question is the following:
Suppose we have a rest API in node js (express) on the following endpoint -> / stars.
Suppose we want to sell this API with the endpoint/stars to a certain target of customers, the endpoint will therefore only allow customers who buy the API to use it.
The problem arises spontaneously, let's suppose that the pizza company buys my API and that I generate an access token for them, then they would call my endpoint with their token to have the resource, so far very good.
However, all the requests are easily visible.
Example Chrome> dev tools> network and I see not only the endpoint with the full address, but even the payload that is passed!
So as an attacker I could very well (without paying the API) catch the pizza industry using the endpoint/stars with a token, copy everything and slap it on my services by providing the same token and the same endpoint.
I already know the existence of tokens like jwt but they don't solve the problem anyway, as that different token only has the expiration.
Even if it expires after 15 minutes or after 3 minutes, just retrieve another one and provide an identical request with the same token, would anyone be able to direct me to a solution?
The only one I've seen to find a solution to this is Instagram that sends behind a payload of thousands of lines, is it really the only method?
note: it is not even public.
#xVoid
The first thing you can set an encryption/decryption module for your response data with the help of the crypto module in node.js, Here you send encrypted response and the your API client decrypt your response and use it.
You can set a key for your API it means every time your client or user send you a request they have to send that key in the body, not header so other people can't
get your data because they don't have that key, and in express you can set middleware to validate this key is exist or not if not simply return "You are not authorized"
If you aren't getting any point or you want to go deep on particular thing just let me know
You may simply use http-only cookie and send the token in cookie, instead of normal header
A customer using your endpoint should not be sharing their API keys with the end-users.
This means that any customer using your service should create at least a proxy server to your specific endpoint.
CLIENT GET /pizza FROM CUSTOMER -> CUSTOMER GET /pizza?apiToken=<...> FROM SERVICE
Obviously there can be a man in the middle attack between the CUSTOMER and your SERVICE but that's is unlikely to occur using SSL (Related: Are querystring parameters secure in HTTPS (HTTP + SSL)? )
If a CUSTOMER suspects that their api key was leaked they should revoke it and request a new one to your SERVICE.
The frontend of my web app uses an access token (periodically generated by the backend) to issue requests to GCP text-to-speech. Issuing requests from the frontend rather than from the backend is essential to keep down both the costs of my service and the delays experienced by the user.
A not-well-intending user might use my app to synthesize speech for a short text, open the Network tab of the browser tools, and get the access token from the request headers sent by my frontend to GCP. He could thereafter use this access token to synthesize speech for a large corpus of text, with no way for me to catch him. How can I change the way I use access tokens to prevent this kind of fraudulent use?
Here are some directions I have thought about, but am not sure what is supported by GCP or whether there is an even better approach:
Create a separate API key for each user of my app and generate access token for the specific user. Then, even if a user uses his token outside of my app, GCP would have a record of the requests made by that user and I could request that record using the API to charge him.
Make access tokens single-use. This way, even if the user obtains the access token as it is being sent as part of a request, he will not be able to use it for another request.
As John Hanley mention you would rather change your design so the backend makes requests to Text-to-Speech on behalf of authorized clients
I am building a node.js web application with react for the the GUI and graphQL served with Apollo for the back-end connecting to a RDS (MySQL) instance on AWS.
I am authenticating users and then returning JWTs. I have it figured out on how to renew/expire tokens, but now I am being faced with the question where to save it on the client side when a user visits the site...
There are two main concepts with a third being a hybrid model. 1) Store it as localStorage with JavaScript as described on HowToGraphQL 2) Store it in a Cookie with http-only set to true as described in the afore mentioned article as a cationary reference to Randall Degges
There is another alternative to store it in memory only on the client side but then a user would have to login every time the page is refreshed as it would not be persistent anywhere.
Concept 1 is vulnerable to XSS only if there is another XSS vulnerability already exploited. But it is secure to the site only so only scripts running on the site can access it and not scripts on any site. There it a lot of security talk that it should not be stored this way even though it is the common way because a developer cannot trust EVERY JavaScript script they are running on their site and there may be one that reads the localStorage and then sends it offsite.
Concept 2 removes the XSS vulnerable by declaring the http-only to only make it accessible to the server at your site. The problem here lies in that then a separate method has to be created to use the same backend authentication for other uses such as a standard API (for native apps or other sites) where the JWT is sent in the header over https where it is stored securely on another server.
So I researched and found this hybrid method described by Ben Awad 3) use a request token and a refresh token. The request token can then act normally for the standard API but then also on our react app site we can store it only in memory and store a refresh token in a cookie to send back a request token when users refresh or close and reopen browsers.
So theoretically, the best solution is Concept 3 which solves all of the concerns, but it is of course more complicated to setup.
My question: How worried should I be about opening up a JWT to an XSS vulnerability? It is something that down the road I would do the long way when I have more time, but I am pushing for a deadline. My site will be lesser known and not something like Facebook or Sales-Force that hackers would necessarily target. My site is not storing Credit Card data or other highly sensitive data other than a basic CRM and task list. If my site was open to XSS through other code, wouldn't the entire authentication process be vulnerable through keylogging scripts or the likes without even knowing the JWT. I feel like I would be doing a lot of extra work to secure against a possible threat that if occurred, the entire system would be compromised already.
If you are comfortable with your site to not work on Internet Explorer and some older versions of the major browsers, you can take advantage of a new cookies property, called Same-Site (to be more precise, the site will work but the cookie will not be secure).
By defining a cookie as HttpOnly, you are immediately secured from XSS attacks, but you leave yourself open to CSRF attacks.
Now by defining the cookie to have the property Same-Site=Strict, the cookie will be only sent through Http calls and only if the domain matches your site's domain. So for example, if someone creates a form in another site and tries to perform a post request to your own site, the cookie will be never sent.
If you want the cookie to be passed only on GET requests, you can set the Same-Site property to Lax but as you mentioned.
You can find more info about this feature in the following link under the SameSite cookies section:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies
You should also check the browser compatibility of the feature by using the following link:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#Browser_compatibility
This is the issue I spent a lot of time on. How to store the authorization token securely. People have different strategies in dealing with this so I will share what works for me. Users of my apps were targeted by different attacks, all of them where unsuccessful in stealing anything so far. None used XSS.
Here is what I do
In the end I opted for storing authorization token in local storage. Applications that I work usually have WebSocket connections on top of HTTP routes and I want the token to be saved in one place and act as a single source of truth. They are all web applications running in the browser. Most of the applications I build use JWT.
Why I do it like that
First why I don't use refresh tokens. If they are saved the same way as the actual authorization token would be saved negates the reason for refresh token to exist since the attacker can use the refresh token to get the authorization one.
Storing the token in cookies gives no benefits over local storage assuming that the app is secured against attackers being able to inject JavaScript into your app, mostly through forms and api on your app. Make sure all user inputs are JS injection safe. On top of that with cookies there are issues when using WebSockets that you must go around.
There is also the point of one of the accounts being hacked and you want to invalidate that token as soon as possible. JWT by default has no mechanism of being revoked. Implementing this feature negates the scalability of JWT because checking the JWT would require a call to the database to know if that user can do the specific action. There are 2 ways you can go about this. One is just check the user data if the user is frozen from the database, it is less scalable because of the call but if you already pull the user data in a middleware it is good enough TM. Other is to pull the the "is the user frozen" data from the database just when making changes to the database or when the call from the client is important.
In summary
I would store the token in local storage. Secure the app from code injections. And make a kill switch for the accounts if they get compromised in any way.
EDIT THANKS TO THE COMMENTS BY #JerryCauser
It is more secure to keep your token in a secure http only cookie. Don't expect a storage mechanism choice to automatically save your users from being hacked. There are ways to hijack sessions and other exploits including users using web extensions and approving their request to read protected data.
For the example of the betting website below, you wouldn't require user to write their password (or approve the request via automated email) every time they place a bet, but you would every time they want to take a withdrawal for example.
I use local storage because even if it happens for the token to be stolen, or another person got to your user's laptop (like a kid for example) you should never let the account do critical tasks without approval.
There is no magic bullet of anti hack protection. Try your best to keep your users safe with common sense.
EDIT AS ANSWER TO THE COMMENT FROM THE ASKER #amaster
If you are making a trip to the database on every call, maybe JWT is not the best solution. Point of JWT is to have signed claims and the id of the user without calling the database. In this case, maybe opt in for sessions instead of JWT.
Before I proceed with my answer, you may want to check out OWASP for a set of general guidelines regarding XSS and CSRF since you've mentioned cookies.
Cedomir already covered a good deal of the points with storing JWT client side. One thing that's worth mentioning is that if you have Third-Party scripts running in your web app, they also have access to the Storage API. So if a script you had loaded were to be hijacked, they could conceivably steal the token there. As for XSS with inputs, if you make sure to escape every possible user input, then that is largely mitigated as an attack vector. But you only have to screw up once for someone to take advantage of the hole and steal the JWT at that point. (Refer to this blog post for more details)
Now, if you instead store the JWT in a Http-Only, then you largely sidestep the XSS issue as you've already noted. However, now you introduced a new problem, that being Cross Site Request Forgery. Since cookies are sent with every request, a malicious actor could set up a website to make a fraudulent request on behalf of user and execute actions without the user's consent. Now I won't cover the mitigation in detail here as OWASP and other places have done a pretty good job already, but the short of it can be summed up by installing the most popular and well-maintained Anti-CSRF package for your language :-)
As for invalidating the token as Cedomir brought up, having that mechanism can be quite useful. However, to implement it does mean you give up some of the benefits of using JWT gives you. Whether you store the current JWT assigned to user and validate that or a unique key used to sign the JWT for each user, you now have user state to keep track of, eliminating one of the reasons to use JWTs. Depending on your application, you will need to weigh that tradeoff. A much simpler way could be simply to have short-lived tokens so that any token that is stolen potentially won't have a very useful lifetime. However, as you probably recognize a short lifetime would be a potentially a very annoying user experience. You could have your website periodically poll the server for a new token while your user continues to use the website as a way to improve the experience. You can also balance your security concerns with the lifetime of the token, like a 15 minute token lifetime for a e-commerce app vs. a hour or more for a social application.
I would however advise against the use of a refresh token, at least for a Browser-Based Web App. Typically speaking, the browser is just not considered capable of securing sensitive secrets. By using a refresh token, you're just deferring the stealing of credentials to another layer as by the nature of the refresh tokens, they're 1) long-lived and 2) effectively used as credentials to obtain more JWTs. So if the refresh token were to be stolen, an attacker can just get more valid JWTs on behalf of a user. If you have a mobile or desktop app, you have mechanisms you can use to securely store refresh tokens and this advice does not apply.
...Or you could just use sessions ;-)
When logging in on server set JWT token and a random csrf token in the httpOnly cookie
Also send this csrf token in body response of login back to client
On every future request from client send this csrf token via some header (eg. X-CSRF-TOKEN)
On the backend verify if the csrf tokens coming through the cookie and x-csrf-token are the same.
Then verify your JWT token and continue with your app logic.
Putting JWT token in httpOnly cookie prevents XSS attacks, validating CSRF token prevents CSRF attacks. Double sending csrf token in both cookie and header avoids storing stuff in the backend database.
XSS check
CSRF check
Stateless auth check
Auth doesn’t have to be over complicated. If you have clients that only want to pass JWT token in some header other than cookie then it’s better to just make a separate api endpoint for those programs.
While the question is not actually about OAuth / OpenID Connect I still think you can learn a great deal by checking out this Internet-Draft: OAuth 2.0 for Browser-Based Apps (Best Current Practice)
To sum it up: there simply is no secure way to store an access token on the client. If you develop only the frontend you pretty much have to use and store a token on client side - not because it's great but because you have no other choice. However, if you do have full control over Frontend and Backend you do have that choice and should think about using the same domain for both and use a session cookie as described in the Internet Draft. Basically the React application never even sees the acesss token, because your backend is serving a http page and handling the authentication directly, with the final step being a redirect back to your frontend while setting the session-cookie.
A potential XSS attack is pretty bad as it is and you should be careful not to introduce a vulnerability. The thing is: with the JWT-approach a XSS vulnerability leads pretty much to the worst-case scenario: the attacker is able to steal the user authentication and can impersonate the user - this is basically session hijacking.
The same attack against a regular session-cookie simply does not have the same impact (as long as the cookie uses the HttpOnly Flag which is highly recommended). Even though the vulnerability enables arbitrary JavaScript Code to run on the machine (which is really bad obviously) it's still a lot harder for the attacker to do some damage. He is not able to hijack the session in this case, because he is unable to read the cookie.
Just use HTTP only + SSL only cookies to save your JWT. It will make almost impossible to stole user's jwt via a soft or any type of code injections.
Someone said here, what it is no diff between LocalStorage and Cookies. He is not correct, bcs third party libraries and chrome extensions can easily stole LocalStorage data. But they cannot stole HTTP only cookie.
It will protect against any known and most likely new types of attacks.
JWT itself is completely protected. Just don’t store something there that could compromise your architecture or something like that (do not put a hashed password for example)
Upd: Good article about best practices for JWT strategy: https://ducktypelabs.com/5-mistakes-web-developers-should-avoid-when-using-jwts-for-authentication/
For the last few months i've been working on a Rest API for a web app for the company I work for. The endpoints supply data such as transaction history, user data, and data for support tickets. However, I keep running into one issue that always seems to set me back to some extent.
The issue I seem to keep running into is how do I handle user authentication for the Rest API securely? All data is going to be sent over a SSL connection, but there's a part of me that's paranoid about potential security problems that could arise. As it currently stands when a client attempts to login the client must provide a username or email address, and a password to a login endpoint (E.G "/api/login"). Along with with this information, a browser fingerprint must be supplied through header of the request that's sending the login credentials. The API then validates whether or not the specified user exists, checks whether or not the password supplied is correct, and stores the fingerprint in a database model. To access any other endpoints in the API a valid token from logging in, and a valid browser fingerprint are required.
I've been using browser fingerprints as a means to prevent token-hijacking, and as a way make sure that the same device used to login is being used to make the requests. However, I have noticed a scenario where this practice backfires on me. The client-side library i'm using to generate browser fingerprints isn't always accurate. Sometimes the library spits out a different fingerprint entirely. Which causes some client requests to fail as the different fingerprint isn't recognized by the API as being valid. I would like to keep track of what devices are used to make requests to the API. Is there a more consistent way of doing so, while still protecting tokens from being hijacked?
When thinking of the previous question, there is another one that also comes to mind. How do I store auth tokens on client-side securely, or in a way that makes it difficult for someone to obtain the tokens through malicious means such as a xss-attack? I understand setting a strict Content-Security Policy on browser based clients can be effective in defending against xss-attacks. However, I still get paranoid about storing tokens as cookies or in local storage.
I understand oauth2 is usually a good solution to user authentication, and I have considered using it before to deal with this problem. Although, i'm writing the API using Flask, and i'm also using JSON Web tokens. As it currently stands, Flask's implementation of oauth2 has no way to use JWTs as access tokens when using oauth for authentication.
This is my first large-scale project where I have had to deal with this issue and i am not sure what to do. Any help, advice, or critiques are appreciated. I'm in need of the help right now.
Put an API Gateway in front of your API , your API Gateway is publicly ( i.e in the DMZ ) exposed while the actual API are internal.
You can look into Kong..
The application I am designing needs consistent access to a user's inbox. Ideally it would know every time a user received an email to their Inbox, but as a proxy I am instead doing a check every five minutes.
When the user signs up, they grant me access to their account via the google gmail api using oauth. Because offline access is needed, I have it set up to also return a refresh token. As far as I can tell though, this means that I need to request a new access token every hour. That seems off to me. Is there a better way of doing this?
Thanks.
Yes, refresh tokens are the correct way to maintain a valid access token long-term. If you're using one of the Google API libraries, this should all be abstracted for you.
Regarding polling, you still need to poll but I suggest using history rather than constantly querying messages.list() or threads.list() with no parameters.