I want to setup temporary routes with a unique random string path on ExpressJS. These routes should be dynamically created and should give a 404 if somebody tries to use ie: http://example.com/login/a434bcd34d920bdfe 30 min after that uniqueid was created.
Any ideas on how to do that? I'm pretty new to NodeJS but judging from what I've seen there should be a library that does that :)
Something like this maybe?
// =========== app.js ============
app.get('/generate_url', function(req, res) {
// random string: "a434bcd34d920bdfe"
var extension = randomstring.generate();
var dynamicController = require('./login/'+extension);
dynamicController.init(app);
// Should expire in 20 minutes
dynamicController.expire(20*60)
res.status(200).send();
}
// =========== login.js ============
login.post('/login/:uniqueid', function(req, res) {
// uniqueid should match the extension generated before
var uniqueid = req.query.unique;
var username = req.body.username;
// Do something with this info
}
I think this is far from working fine but at least maybe somebody who's done
You asking two question:
How to create the route.
How to save the data about expired links.
1. How to create the Route:
You have to create a route that receieve the traffic from all the users, and check that the specific URL is valid.
First you create a route that get traffic from all the users.
app.get('/myroute/:id',function(){
/* This route will get any url that start with /myroute/
For example /myroute/abc
/myroute/def
*/
// req.params.id == what the URL is entered
if (is_expired(req.params.id)) return res.end('Sorry your link has expired')
res.send('Great you logged in!')
})
app.post('/login/:uniqueid', function(req, res) {
set_expire(req.params.uniquieid,30*1000*60) //30 minutes = 30*60*1000 miliseconds.
})
2. How to save the data about expired links.
How to implement set_expire and is_expired
You need to implement is using any kind of database. Redis is very good for that. I will show you example how to do it
using setTimeout. It will work. But if the server restart, all the users will be logged out.
users={}
function is_expired(uid){
return users[uid]
}
function set_expire(uid,time){
users[uid]=true
setTimeout(function(){
delete users[uid]
},time)
}
Usually this is done with a generic router handler that then consults some data store to see if the ID sent in the request is valid.
Also, note that a value that comes from a route specification like '/login/:uniqueid' is found in req.params.uniqueid, not in req.query.uniqueid. req.query is for actual query parameters (things after the ? in the URL).
var validIds = {};
// =========== app.js ============
app.get('/generate_url', function(req, res) {
// generate random string: "a434bcd34d920bdfe"
// make sure it's not already in use
var extension;
do {
extension = randomstring.generate();
} while (validIds[extension]);
// save this random string as a valid ID and store the expiration time
validIds[extension] = {expiration: Date.now() + 20 * 60 * 1000};
// return the random value to the caller
res.json(extension);
}
app.post('/login/:uniqueid', function(req, res) {
// uniqueid should match the extension generated before
var uniqueid = req.params.unique;
var username = req.body.username;
// now see if the id is still valid
var idInfo = validIds[uniqueid];
if (!idInfo || idInfo.expiration < Date.now()) {
return res.sendStatus(404);
}
// id is valid, do something with this info
}
Then, you can have some interval timer that regular cleans up any expired ids. This is just housekeeping to keep the validIds object from growing forever - the actual expiration value is still checked before validating the id.
// clean up expired validIds object every 10 minutes
setInterval(function() {
var now = Date.now();
for (var id in validIds) {
if (validIds[id].expiration < now) {
delete validIds[id];
}
}
}, 10 * 60 * 1000);
Note: Since you probably want your uniqueIDs to survive a server restart, you need to regularly persist them to some sort of backing store (database, flat file, etc...) and then you need to read that data in upon server startup.
If you want any sort of user activity to "renew" the timeout on the uniqueid, you can install some middleware that resets the time at an appropriate access.
Related
I have a sample app, user can access some dynamic data via different URL.
The workflow is like this:
when user request get_data?id=1234567
first it checks the DB if there is data for it
if not, generate a random value
then if other users request the same url within a short time (say 10 min), it will return the value that already generated
if one of the users send a clear request, the value will be cleared from DB.
The bug is: if 2 users request the same url at the same time, since it needs time to query the DB, it would do 1 and 2 at the same time, then create different values for each user.
How to make sure that in a short period, it always generate same value for all users?
Although NodeJS is single threaded and does not have the problem of synchronization between multiple threads, its asynchronous event model still can require you to implement some kind of locking mechanism to synchronize the concurrent async operations in certain situations (like in your case).
There are a number of libraries that provide this functionality, e.g. async-mutex. Here's a very basic example of what your code could look like:
const express = require('express');
const app = express();
const Mutex = require('async-mutex').Mutex;
const locks = new Map();
app.get('/get_data', async (req, res) => {
const queryId = req.query.id;
if (!queryId) {
// handle empty queryid ...
}
if (!locks.has(queryId)) {
locks.set(queryId, new Mutex());
}
const lockRelease = await locks
.get(queryId)
.acquire();
try {
// do the rest of your logic here
} catch (error) {
// handle error
} finally {
// always release the lock
lockRelease();
}
});
app.listen(4000, function () {
console.log("Server is running at port 4000");
});
I have a project I build with React and Strapi headless CMS (using nodejs). The backend, which is handled using Strapi, resides in port 443. When I send a GET request to any url in backend using http://site:443 I get a valid response, but a POST request always returns an error 404.
I've used an external tool https://reqbin.com/ to test it and got the same results.
The interesting thing is that even though I get 404 response, the route is being handled and executed, and yet gives a 404 response back.
Why could this be happening?
Thanks in advance
Here is the code upon reaching this route:
/**
* Create a/an orders record.
*
* #return {Object}
*/
create: async (ctx) => {
let params = ctx.request.body;
if (params.Sum) { // only if we got a good POST response
// check secret code
let a = params.UniqueID;
var res = a.substr(a.length - 12, a.length);
if (res !== 'secret') {
ctx.throw(500, 'SECURITY BLOCK', { expose: true });
return null;
}
var username = a.substr(0, a.length - 12);
// create validity
var date = new Date();
if (params.Sum === '250') { // one month membership
date.setTime( date.getTime() + 31 * 86400000 );
} else if (params.Sum === '390') { // one month membership
date.setTime( date.getTime() + 180 * 86400000 );
}
// create order in database
let today = new Date();
// extract username
var n = params.CustomerName.indexOf("_");
var name = params.CustomerName.substr(0, n);
var instId = params.CustomerName.substr(n+1, params.CustomerName.length);
const order = await strapi.services.orders.add({
userId: username,
username: name,
institutionId: instId,
sum: params.Sum,
transactionDate: today,
validity: date,
cardNum: params.CardNum,
cardName:params.CardName,
});
let axiosArr = {
institutions: [{_id: instId}],
validity: date,
secret: 'secret'
}
axios.put(apiUrl+'/users/'+username, axiosArr).then(() => {return order})
// return order
Most of the code is irrelevant to the question, I was thinking maybe the reason that the response is 404 is because I'm not returning the order object right away, but only after the axios.put?
Unfortunately I cannot test it right now
If the route is well executed but it returns a 404, it's because the response body is empty.
In your code your commented the return order.
Add in your code something like ctx.send({order}) or un comment your return
You will no longer have a 404.
I'm not familiar with strapi but it doesn't sound like an auth issue to me. Are you sure these routes are set up to accept POST requests and not just GET requests? It's common to return 404s if a user is missing a permission needed to access a resource, to not let the person know that resource exists but without any sample code I can't really say for sure what's happening in your case. Could you link more code?
I am currently exploring solutions to limit access to an API endpoint on NodeJS by total number of requests per month.
For instance, I want the free plan users to access the /api endpoint up to a total of 100 requests per month, and the premium plan users to have 5000 requests per month.
The naïve way to get around it is by implementing a passport middleware to get the user's plan and then keep track of the count:
app.get("/api", requireAuth, async (req, res, next) => {
try {
// Check if user ran out of requests
if (req.user.apiRequestsLeft === 0) {
res.send("You ran out of API requests!")
} else {
// Decrement the allocated requests
req.user.apiRequestsLeft--;
await req.user.save();
res.send(user)
}
} catch (err) {
next(err);
}
});
My concerns are:
Performance/Scalability issues of having to update a MongoDB document each time there's a request - is this feasible or will I hit a problem when the app grows?
Resetting the count - should this be a daily cronjob that looks at the timestamp of 'registration' of each and every user, compute if a month has passed and reset allotted requests accordingly, or is there a better way of designing something like this?
Performance/Scalability issues of having to update a MongoDB document
each time there's a request - is this feasible or will I hit a problem
when the app grows?
Definitely. You will soon experience heavy mongoDB traffic and it will hit performance bottleneck. In my opinion, you should use a faster in-memory database like Redis to handle the situation. You can even use the Redis as the session-store which will reduce the load on MongoDB. That way, MongoDB can be utilized for other Business queries.
Resetting the count - should this be a daily cronjob that looks at the
timestamp of 'registration' of each and every user, compute if a month
has passed and reset allotted requests accordingly, or is there a
better way of designing something like this?
A better way would be to achieve the resetting part in the middleware itself.
Here is some code that explains my solution.
Sample design of Quota object would be:
{
type: "FREE_USER", /** or "PREMIUM_USER" */
access_limit: 100, /** or 5000 */
exhausted_requests: 42 /** How many requests the user has made so far this month */
last_reset_timestamp: 1547796508728 /** When was the exhausted_requests set to 0 last time */
}
With that design. Your middleware that checks the quota would look something like:
const checkQuota = async (req, res, next) => {
const user = req.user;
const userQuotaStr = await redis.getAsync(user.id)
let userQuota;
/** Check if we have quota information about user */
if (userQuotaStr != null) {
/** We have previously saved quota information */
userQuota = JSON.parse(userQuotaStr);
/**
* Check if we should reset the exhausted_requests
* Assuming that all the requests are reset on the First Day of each month.
*/
if ( isStartOfMonth() ) {
/**
* It is First Day of the month. We might need to reset the `exhausted_requests`
* Check the difference between `Date.now()` and `userQuota.last_reset_timestamp`
* to determine whether we should reset or not
*/
if ( shouldResetTimeStamp(userQuota.last_reset_timestamp) ) {
userQuota.exhausted_requests = 0
userQuota.last_reset_timestamp = Date.now()
}
}
} else {
/** We do not have previously saved quota information. Prepare one */
userQuota = {
type: user.type,
access_limit: user.access_limit,
exhausted_requests: 0,
last_reset_timestamp: Date.now()
}
}
/** Incredement the counter to account the current request */
userQuota.exhausted_requests++
/** Update in database */
redis.set(user.id, JSON.stringify(userQuota))
if ( userQuota.exhausted_requests >= userQuota.access_limit ) {
/** User has reached the quota limit. Deny the request. set with 401 or 403 status code */
} else {
/** User can access the API. call next() */
}
}
Of course, the snippet is is incomplete. It just gives you the idea about how to go about writing that middleware.
Here is how you can use the middleware for your APIs:
/** If requests to routes are under the quota */
app.get("/api/quota-routes", requireAuth, checkQuota, /** Mount the actual middleware here */)
/** If requests to routes are unlimited, just remove the checkQuota middleware */
app.get("/api/unlimited-routes", requireAuth, /** Mount the actual middleware here */)
rate-limiter-flexible package helps with counters and automatically expire counters.
const opts = {
storeClient: mongoConn,
points: 5000, // Number of points
duration: 60 * 60 * 24 * 30, // Per month
};
const rateLimiterMongo = new RateLimiterMongo(opts);
const rateLimiterMiddleware = (req, res, next) => {
// req.userId should be set before this middleware
const key = req.userId ? req.userId : req.ip;
const pointsToConsume = req.userId ? 1 : 50;
rateLimiterMongo.consume(key, pointsToConsume)
.then(() => {
next();
})
.catch(_ => {
res.status(429).send('Too Many Requests');
});
};
app.use(rateLimiterMiddleware);
Note, this example is not bound to calendar month, but counts events from the first event within the next month after it. You could set custom duration with
block to strictly connect counters expiry to calendar months.
This code should easily handle about 1k-2k requests per second on basic server. You could also use Redis limiter or Mongo limiter with sharding options.
Also, it provides In-memory Block strategy to avoid too much requests to MongoDB/Redis/any store.
Alternatively, play with get method from rate-limiter-flexible to decrease amount of unnecessary counters updates. get method is much-much faster than increment.
I've installed Loopback and enabled ACL for a couple of models. I noticed that the Access Token is valid for ever, I would like to change this period somehow to, for example, an hour. But even better would be to reset this period when activity occurs (sliding expiration)
I've checked the documentation but couldn't fine anything about this subject. Any help/guidance would be appreciated!
When you call the login method you can specify a ttl property in seconds(I believe by default it's 2 weeks if you don't specify). Then you can have sliding expiration by having the following middleware:
app.use(loopback.token()); // You should have this already
app.use(function(req, res, next) {
// Make sure this middleware is registered after loopback.token
var token = req.accessToken;
if (!token) {
return next();
}
var now = new Date();
if ( now.getTime() - token.created.getTime() < 1000 ) {
return next();
}
req.accessToken.created = now;
req.accessToken.ttl = 604800; //one week
req.accessToken.save(next);
});
Sometimes we need restrict executing repeating requests from a user if the first request has not been finished. For example: We do want register user at some service, and only after that put him into database with external id. I would like to have a service where I can set protected routes.
I have solved this problem by checking flag at request.start() -> and removed it after the request has been completed. Anyway I'm looking for your suggestion guys.
I think you need to build this into the route handlers yourself:
var inProgress = []
var handleRegisterRoute = function create(req, res, next) {
var id = req.user.id
var found = _.find(inProgress, function(item){
return item = id
})
if(found){
res.send('Dont think twice, its aright')
return
} else {
inProgress.push(id)
completeRegister()
inProgress= _.without(inProgress, id);
req.send()
return
}
}
Again this is psuedo code - to the gist of what I would write. You may need to store the "inprogress" in a better data store referenceable by your entire server farm - some sort of DB.