Need some suggestions about how to handle operations in NodeJs. I have tried to search things but couldn’t find it anywhere, so I thought I will post it here to get the experts feedback.
We have built an E commerce platform using NextJS and NodeJS.
I still have to go test and do a load test confirming how do we perform, but I’m a little worried about all the statements and declarations about how nodejs is not a multi threading language and it is a show stopper and bla bla bla. So I am here to get your suggestion/feedback on what we are doing is okayish or not. This is what I have:
Authentication Microservice: for authenticating requests and finding if the user has access to a particular module
payment microservice: for accepting payments and refunds.
Shopping Service
MySQL and Sequelize
I will try to explain my question with an example of "order processing". So when I get a request to confirm an order, this is what I am doing to authenticate the request and confirm if details are full proof.
Confirm the request is authenticated through jwt token
Make a request to user auth and confirm user is a valid user and has access to order module.
Verify the price of items through database (because I’m assuming a payload can be hacked and wrong prices can be sent)
Verify if the discount is validated correctly from the database
Verify if the coupon is valid and the discount applied is correct or not
Check if the reward points sent are correct or not - a request to user service is sent
Our payment gateway is Razor Pay, here, we confirm if the payment id sent is a valid payment id, and the amount received from the user matches the total amount to be charged.
So this is a long list of validations we do at NodeJS right now.
It’s super fast on a single request. I will definitely tune things around when we go deep into load testing.... but my current dilemma right now is whether we have opted the right approach or not. Is NodeJS good enough to do this kind of execution. I am expecting no more than 200+ concurrent users in 6-7 months down the line so I am not toooooo worried about performance. I can do clustering and dockering to boost up performance. I want to know if I have made the right choice to do this in NodeJS. It’s not a very high cpu intensive algorithm, just some ‘for loops’ and some Database requests.
Should I keep this in Node, or should I move this code of algorithm in another service written in a different language like C++ or may be python.
Big time confused right now. Any help will be greatly appreciated.
Node.js may be single-threaded, but it should be fine with what you are doing. It is well suited for horizontally scaling/clustering and there are plenty of node modules already there to help you out with that.
https://medium.com/iquii/good-practices-for-high-performance-and-scalable-node-js-applications-part-1-3-bb06b6204197
Additionally, Node.js is asynchronous and can handle concurrent requests.
I have also included an older link on how to scale node.js that lists how many users each scaling method can help support. Long story short you can use vertical scaling, horizontal scaling, microservices(which it sounds like you already looking at) and automation to easily scale up your application. Node.js should work fine.
https://adrianmejia.com/how-to-scale-a-nodejs-app-based-on-number-of-users/
Related
I have developed an app for Android/iOS which calculates a value based on the users input. If an event occurs, this calculated value will be sent to my Backend as normal HTTPS payload. My question is now, how can I make sure, that this value is really only calculated by the source code of my app? Is there a way to handle such a problem?
To make it clear: I want to avoid, that somebody is rooting his phone, extract the Auth-Token from the private storage of my app and sends a valid HTTPS-Payload to my Backend with fictitious payload, manually or by manipulating the source code.
From the view of the backend, it's difficult to evaluate the payload based on its values if it is valid or not.
Any suggestions appreciated!
----------EDIT-----------
For the sake of completeness: apart from the answers here, the following are also very interesting:
Where to keep static information securely in Android app?
How to secure an API REST for mobile app? (if sniffing requests gives you the "key")
You can’t trust data coming from the client. Period.
You should consider moving the calculation logic to the server and just sending the raw values needed to perform the calculation. You can easily get sub-second response times sending the data to the server, so the user won’t notice a lag.
If you need offline connectivity, then you’ll need to duplicate the business logic on both the client and the server.
Short of doing everything on the backend, you can't very easily.
I'd recommend some reading around CSRF (Plenty of articles floating around) as that's at least a good mitigation against bots outside of your app domain hitting your backend. The upshot is that your application requests a unique, random, identifier from your backend (which ideally would be tied to the user's auth token) before submitting any data. This data is then submitted with your app's data to perform the calculation on the backend. The backend would then check this against the random identifier it sent for that user earlier and if it doesn't match, then reject it with a 400 (Bad Request), or 404 if you're paranoid about information leakage.
I was recently interviewed by a MNC technical panel and they asked me different questions related to RestAPI , i was able to answer all but below 2 questions though i answered but not sure if those are correct answers. Can somebody answer my queries with real time examples
1) How can i secure my Rest API when somebody send request from Postman.The user provides all the correct information in the header like session id, Token etc.
My answer was: The users token sent in the header of the request should be associated with the successfully authenticated user info then only the user will be granted access if the Request either comes from Postman or application calls these API.(The panel said no to my answer)
2) How can i handle concurrency in Rest API Means if multiple users are trying to access the API at the same given time (For e.g multiple post request are coming to update data in a table) how will you make sure one request is served at one time and accordingly the values are updated as requested by different user request.
2) My answer was: In Entitiy framework we have a class called DbUpdateConcurrencyException, This class takes of handling concurrency and serves one request is served at a time.
I am not sure about my both the above answers and i did not find any specific answer on Googling also.
Your expert help is appreciated.
Thanks
1) It is not possible, requests from Postman or any other client or proxy (Burp, ZAP, etc) are indistinguishable from browser requests, if the user has appropriate credentials (like for example can observe and copy normal requests). It is not possible to authenticate the client application, only the client user.
2) It would be really bad if a web application could only serve one client at a time. Think of large traffic like Facebook. :) In many (maybe most?) stacks, each request gets its own thread (or similar) to run, and that finishes when the request-response ends. These threads are not supposed to directly communicate with each other while running. Data consistency is a requirement of the persistence technology, ie. if you are using a database for example, it must guarantee that database queries are run one after the other. Note that if an application runs multiple queries, database transactions or locks need to be used on the database level to maintain consistency. But this is not at all about client requests, it's about how you use your persistence technology to achieve consistent data. With traditional RDBMS it's mostly easy, with other persistence technologies (like for example using plaintext files for storage) it's much harder, because file operations typically don't support a facility similar to transactions (but they do support locks, which you have to manage manually).
I'm making an angular app that has users log in, make progress, then they are awarded levels/experience points. I'm using a nodejs/express API and I want to be able to make a call from my app to award them exp. I'm using a JWT and server signing with a private key to auth requests, but realized that a user could just pull their token and give themselves experience. My question would be is there anyway to protect my route from that or is that a fundamental flaw in design?
I don't believe this is something you can do specifically with JWT. As commenters have already said, JWT just provides access rights for the given token. As you say yourself, it would be simple enough to just read the traffic and send their own requests to jack up their exp.
While your basic authentication/authorisation mechanism can't solve this, you can handle it in some other fashion within, for example, the request payload itself.
You could encrypt and/or sign your payloads - given that the app would need to know or receive key(s) to use, it's possible that with enough investigation that this is eventually found and duplicated as well. But it's another step someone would have to go through and replicate.
You could employ additional checks and measures - have your requests for [exp increase] be a two-step process; the server responds to the initial request with some minor task to be solved that is then attached to the follow-up request. Assuming the task is done properly, you can be reasonably sure that it came from your app as your app knows how to solve the problems issued (or someone with a serious lack of hobbies outside of deconstructing your entire application).
You could limit the amount of exp that should be reasonably achievable by your users. If you know that people should, at most, be able to gain xyz exp per minute/hour/day/etc, then by monitoring exp growth, you can flag and/or block additional gains past this point.
I have a web application that runs Node.js/Express and I've done some searching and I've come up short.
Basically I have a web app where you can post the usual, comments, ratings, etc. This is all fine, but if somebody uses something like Postman (with their login credentials) they could fire 100/1000s of comments by just clicking POST on Postman.
What's the best way to handle this? Should I just rate limit the user actions server side (comments/ratings after X amount & time) or is there a better way?
Ideally I'd like to take off being able to use software like Postman to submit anything--I want it all through the website but I'm not sure if this is possible because it's a route?
Any advice is greatly appreciated!
Unless this is actively a problem that you are encountering, I would, as general advice, suggest that your time is better spent elsewhere. YAGNI, as they say.
On your actual issue, handling overload attacks has lots of different alternative measures. The easiest option would be if the API you are developing requires authentication - if so, you can apply a per-account rate limit. For instance, 100 requests per minute per user.
You can do that using something like express-rate-limit.
If you are worried about actual large scale denial of service attacks, mitigation these days needs to happen long before requests reach your application server tier.
Businesses like CloudFlare and Akamai provide this as a service, where you configure your domain to go to their DDoS-capable infrastructure, and they will then forward vetted traffic to your Express server.
I use hpp npm package (Express middleware to protect against HTTP Parameter Pollution attacks) to limit call rate
https://www.npmjs.com/package/hpp
Lets say that in a browser based game, completing some action (for simplicity lets say someone clicks on a link that increases their score by 100) clicking on this link which would have a url for example increase_score.pl?amount=100 what kind of prevention is there from someone simply sending requests to the web server to execute this command:
Over and over again without actually doing the task of clicking on the link and
Sending a false request to the server where amount is set to something rediculus like 100000.
I am aware of checking HTTP_REFERER however I know people can get around that (not sure how exactly) and other than some bounds checking for the 2nd option I'm kind of stumped. Anyone ever experience similar problems? Solutions?
Nothing can stop them from doing this if you implement your game how you propose.
You need to implement game logic on the server and assign points only once the server validates the action.
For example: on SO when someone votes your question up, this isn't sent as a command to increase your reputation. The web-app just says to the server user X voted question Y up. The server then validates the data and assigns the points if everything checks out. (Not to say SO is a game, but the logic required is similar.)
Short version: you can't. Every piece of data you get from the client (browser) can be manually spoofed by somebody who knows what they're doing.
You need to fundamentally re-think how the application is structured. You need to code the server side of the app in such a way that it treats every piece of data coming from the client as a pack of filthy filthy lies until it can prove to itself that the data is, in fact, plausible. You need to avoid giving the server a mindset of "If the client tells me to do this, clearly it was allowed to tell me to do this."
WRONG WAY:
Client: Player Steve says to give Player Steve one gazillion points.
Server: Okay!
RIGHT WAY:
Client: Player Steve says to give Player Steve one gazillion points.
Server: Well, let me first check to see if Player Steve is, at this moment in time, allowed to give himself one gazillion points ... ah. He isn't. Please display this "Go Fsck Yourself, Cheater" message to Player Steve.
As for telling who's logged-in, that's a simple matter of handing the client a cookie with a damn-near-impossible-to-guess value that you keep track of on the server -- but I'll assume you know how to deal with session management. :-) (And if you don't, Google awaits.)
The logic of the game (application) should be based on the rule to not trust anything that comes from the user.
HTTP_REFERER can be spoofed with any web client.
Token with cookie/session.
You could make the link dynamic and have a hash that changed at the end of it. Verify that the hash is correct given that period of time.
This would vary in complexity depending on how often you allowed clicks.
A few things to note here.
First, your server requests for something like this should be POST, not GET. Only GET requests should be idempotent, and not doing so is actually a violation of the HTTP specification.
Secondly, what you're looking at here is the classic Client Trust Problem. You have to trust the client to send scores or other game-interval information to the server, but you don't want the client to send illegitimate data. Preventing disallowed actions is easy - but preventing foul-play data in an allowed action is much more problematic.
Ben S makes a great point about how you design the communication protocols between a client and a server like this. Allowing point values to be sent as trusted data is generally going to be a bad idea. It's preferable to indicate that an action took place, and let the server figoure out how many points should be assigned, if at all. But sometimes you can't get around that. Consider the scenario of a racing game. The client has to send the user's time and it can't be abstracted away into some other call like "completedLevelFour". So what do you do now?
The token approach that Ahmet and Dean suggest is sound - but it's not perfect. Firstly, the token still has to be transmitted to the client, which means it's discoverable by the potential attacker and could be used maliciously. Also, what if your game API needs to be stateless? That means session-based token authentication is out. And now you get into the deep, dark bowels of the Client Trust Problem.
There's very little you can do make it 100% foolproof. But you can make it very inconvenient to cheat. Consider Facebook's security model (every API request is signed). This is pretty good and requires the attacker to actually dig into your client side code before they can figure out how to spoof a reqeust.
Another approach is server replay. Like for a racing game, instead of just having a "time" value sent to the server, have checkpoints that also record time and send them all. Establish realistic minimums for each interval and verify on the server that all this data is within the established bounds.
Good luck!
It sounds like one component of your game would need request throttling. Basically, you keep track of how fast a particular client is accessing your site and you start to slow down your responses to that client when their rate exceeds what you think is reasonable. There are various levels of that, starting at the low-level IP filters up to something you handle in the web server. For instance, Stackoverflow has a bit in the web application that catches what it thinks are too many edits too close together. It redirects you to a captcha that you need to respond to if you want to continue.
As for the other bits, you should validate all input not just for its form (e.g. it's a number) but also that the value is reasonable (e.g. less than 100, or whatever). If you catch a client doing something funny, remember that. If you catch the same client doing something funny often, you can ban that client.
Expanding on Ahmet's response, every time they load a page, generate a random key. Store the key in the user session. Add the random key to every link, so that the new link to get those 100 points is:
increase_score.pl?amount=100&token=AF32Z90
When every link is clicked, check to make sure the token matches the one in the session, and then make a new key and store it in the session. One new random key for every time they make a request.
If they give you the wrong key, they're trying to reload a page.
I would suggest making a URL specific to each action. Something along the lines of:
/score/link_88_clicked/
/score/link_69_clicked/
/score/link_42_clicked/
Each of these links can do two things:
Mark in the session that the link has been clicked so that it wont track that link again.
Add to their score.
If you want the game to only run on your server, you can also detect where the signal is sent from in your recieving trick, and ignore anything not coming from your domain. It will be a real pain to tamper with your codes, if you have to run from your dedicated domain to submit scores.
This also blocks out most of CheatEngine's tricks.