So I'm writing an application in node.js+express which I want to achieve the following goal.
User POST many requests at a nearly same time (like with command curl...& which & makes it run at the background)
Process each request at one time, hold other requests before one is finished. The order can be determined by request arrive time, if same then choose randomly. So if I POST 5 requests to add things in the database at nearly the same time, the first request will be added into database first, while other requests will be held (not responding anything yet) until the first request is been processed and respond with a 200 code then follow on processing the second request.
Is it possible to achieve this with express, so when I send couple requests at one time, it won't occur issue like something isn't add into MongoDB properly.
You can set up middleware before/after your routes to queue up and dequeue requests if one is in progress. As people have mentioned this is not really best practice, but this is a way to do it within a single process (will not work for serverless models)
const queue = [];
const inprogress = null;
app.use((req, res, next) => {
if (inprogress) {
queue.push({req, res, next})
} else {
inprogress = res;
}
})
app.get('/your-route', (req, res, next) => {
// run your code
res.json({ some: 'payload' })
next();
})
app.use((req, res, next) => {
inprogress = null;
if (queue.length > 0) {
const queued = queue.shift();
inprogress = queued.res;
queued.next();
}
next();
})
Related
I have a script that plays music and because of the nature of the script it seems to be running 2 connections for every 1 listener.
What that means is it is creating 2 documents inside Mongodb.
I am wondering is there a way to tell mongoose to not insert document within 5 seconds if the insert has already ran?
I have run a console.log(req.url)
and I get the following
/dance?uuid=6f70c645-4ef4-4042-854d-a6e87b61e45f
/dance?uuid=6f70c645-4ef4-4042-854d-a6e87b61e45f
Which means it is requesting the file twice on load.
Is there away to only allow 1 request to go through and discard the other?
I have tried
router.get('/:stationname', (req, res, next) => {
console.log(req.url)
});
Also tried the following
var doCSUpdateAlreadyCalled = false;
router.get('/:stationname', (req, res, next) => {
if (!doCSUpdateAlreadyCalled) {
doCSUpdateAlreadyCalled = true
console.log(req.url)
}
});
But it still runs twice.
I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.
I want to be able to exit execution of a post route when an event is sent from the client-side. I'm using socket.io but I'm not sure it can do what I want. I am using the uploads route to process a file, but if the user deletes the file, I want the app.post execution to end, similar to either a res.end() or return statement.
My app in the front-end receives a file from the user and immediately is sent to the post route for processing. If the user deletes the file and uploads a new one, the previous post route is still going. I want to make sure the previous one was terminated, cancelled, etc.
I'm currently using socket.io to communicate front-end to back-end.
How can I achieve this?
app.post('/uploads', async (req, res) => {
// async func1
// async func2
// if we receive an event from the front end while processing here, how can I exit the post route?
// async func3
});
You can add UUID for each request you make and return it to the front-end. The request will be resolved with the 202 ACCEPTED status code meaning the request was accepted and being handled but the HTTP request will be resolved.
Now you can implement a resourceManagerServeic that will allow APIs (http or ws) to change the state of a resource (like canceling it).
app.post('/uploads', async (req, res) => {
const resourceUuid = resourceManagerServeic.createResource();
res.status(202); // ACCEPTED
res.send({ uuid: resourceUuid });
// start besnise logic
await function1();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function2();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function3();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
});
app.del('/uploads/:resourceUuid', async (req, res) => {
resourceManagerServeic.cancle(req.params.resourceUuid);
res.end() // handle response
});
I guess that your are using Express. Take a look at express-async-handler
You can invoke it
const asyncHandler = require('express-async-handler')
app.post('/upload', asyncHandler(async(req, res) => {
await firstfunc()
await secondfunc()
}))
I am writing the middleware for API endpoints in my app that respond to webhooks from other applications, and am relatively new to Koa, so am not completely familiar with its patterns.
I would like to structure my middleware as follows:
exports.updateReceived = async (ctx, next) => {
// Respond to server issuing the webhook
ctx.res.body = "ok";
ctx.res.statusCode = 200;
// Grab what we need from the request
const { headers, state, request } = ctx;
const { body } = request;
// Do some async work
const { example } = await doSomeAsyncWork(ctx);
// Prepare a database query
const query = { aValue: anId };
// Run the DB query
const result = await Thing.findOne(query);
// Add data to the request
state.thing = result;
// Move on...
return next();
};
However, this does not appear to be working, as an error in any of my async methods can cause the route to error out.
My goal is for this endpoint to always respond "yep, ok" (immediately), meaning it is simply up to the application to handle any error states.
I have researched this fairly well, and have come across this pattern:
app.use(async ctx => {
db.fetch() // Assuming a Promise is returned
.then(() => { ... })
.catch(err => {
log(err)
})
// status 200 will be returned regardless of if db.fetch() resolves or rejects.
ctx.status = 200
})
However, this does not meet my needs as the middleware makes no use of next, so it is not really a useful pattern, so far as I can tell.
Could someone tell me what I am overlooking?
next() invokes the downstream middleware chain and returns a promise that resolves after all downstream middleware/handlers have finished.
That means you can simply implement your own upstream error handler that catches any errors and always ensures a 200 OK response.
const Koa = require('koa')
const app = new Koa()
app.use(async (ctx, next) => {
next().catch((err) => {
// Print error for our own records / debugging
console.error(err)
// But ensure that outgoing response is always a smile
ctx.status = 200
ctx.body = ':)'
})
})
app.use(async (ctx) => {
// Do your webhook / biz logic here, but for demonstration
// let's make it always throw an error. Thus upstream next()
// will be a rejected promise.
throw new Error('this middleware will always bubble up a rejected promise')
})
app.listen(3000, () => {
console.log('listening on 3000')
})
Note: We are not awaiting next(), so we can end the request immediately. However the next handler in the chain will still have the opportunity to process the logic
app.use((ctx, next) => {
next()
ctx.status = 200
})
app.use( async ctx =>{
db.fetch()
.then(() => { ... })
.catch(err => log(err))
}
}
Just to divert the solution in a different side, You could consider adding your work to some kind of MessageQueue and then let another process do that task for you. Basically asynchrously but you will still be important. This kind of pattern suits for your requirement.
There are many messaging system availble like AWS SQS which you could consider. This way your api will be very light weight and it will do thing which it needs to and send a command to your messaging system to do extra stuff. You are basically separting your core logic and the doing things in background which scales very nicely as well.
How to run function within a function
If the first function throw any error then 2nd function should run
server.post('/User', (req, res,next) =>
server.post('/submit', (req, res) =>
server.post('/User', (req, res,next) => {
// Some Task
if(){
//Do something
}
else {
server.post('/submit', (req, res) => {
If the first function returns a promise you could run those functions after catching the error like this:
firstFunction()
.catch(e => {
secondFunction();
});
Otherwise you could use a try-catch statement like this:
try {
firstFunction();
} catch (e) {
secondFunction();
}
In Express, the next parameter allows a route/middleware to pass processing to the next route/middleware (actually, all routes are merely middlewares that don't call next). Therefore you can implement passing your route processing by just calling next():
function submitHandler (req, res) {
// ...
}
server.post('/User',
(req, res, next) => {
if(/* all ok */){
//Do something
}
else {
next(); // continue processing
return;
}
},
submitHandler // continue with this function
)
server.post('/submit', submitHandler); // also have a separate endpoint where
// submitHandler can be accessed directly
Remember that server.post(...) doesn't actually carry out that operation in that route. It just registers a route handler for some future route. So, you don't generally do things like:
server.post("/firstRoute", function(req, res, next) {
if (some condition) {
server.post("/secondRoute", ...);
}
});
This would configure a second route handler for all users based on what one user did in their /firstRoute request.
Remember a server contains a set of route handlers that work for ALL users that will use the server now and in the future. What your code proposes is that based on some data in one particular incoming request, you would change the routes that the server supports for all users. That's not how you would code a server.
If you need to maintain server-side state for a particular user such that data from one request influences how some future request works, then you would typically create a user session object, store the state for that user in the session and then one some future request, you can consult the data in the session to help you respond to the future request.
A simple example would be a shopping cart. Each time you add something to the cart, the data is added to the session object. Then, when the user asks to view the cart, you render a page by consulting the items in that user's session object. If they then ask to "check out", you purchase the items in the cart for them. You don't register and deregister routes for "view cart" and "checkout". Those routes are always registered. Instead, you use the data in the session object to help you process those requests.
If you want to run the same code from several different routes, then you just move that code into a shared function and call that shared function from more than one route handler.