Sometimes we need restrict executing repeating requests from a user if the first request has not been finished. For example: We do want register user at some service, and only after that put him into database with external id. I would like to have a service where I can set protected routes.
I have solved this problem by checking flag at request.start() -> and removed it after the request has been completed. Anyway I'm looking for your suggestion guys.
I think you need to build this into the route handlers yourself:
var inProgress = []
var handleRegisterRoute = function create(req, res, next) {
var id = req.user.id
var found = _.find(inProgress, function(item){
return item = id
})
if(found){
res.send('Dont think twice, its aright')
return
} else {
inProgress.push(id)
completeRegister()
inProgress= _.without(inProgress, id);
req.send()
return
}
}
Again this is psuedo code - to the gist of what I would write. You may need to store the "inprogress" in a better data store referenceable by your entire server farm - some sort of DB.
Related
I have a sample app, user can access some dynamic data via different URL.
The workflow is like this:
when user request get_data?id=1234567
first it checks the DB if there is data for it
if not, generate a random value
then if other users request the same url within a short time (say 10 min), it will return the value that already generated
if one of the users send a clear request, the value will be cleared from DB.
The bug is: if 2 users request the same url at the same time, since it needs time to query the DB, it would do 1 and 2 at the same time, then create different values for each user.
How to make sure that in a short period, it always generate same value for all users?
Although NodeJS is single threaded and does not have the problem of synchronization between multiple threads, its asynchronous event model still can require you to implement some kind of locking mechanism to synchronize the concurrent async operations in certain situations (like in your case).
There are a number of libraries that provide this functionality, e.g. async-mutex. Here's a very basic example of what your code could look like:
const express = require('express');
const app = express();
const Mutex = require('async-mutex').Mutex;
const locks = new Map();
app.get('/get_data', async (req, res) => {
const queryId = req.query.id;
if (!queryId) {
// handle empty queryid ...
}
if (!locks.has(queryId)) {
locks.set(queryId, new Mutex());
}
const lockRelease = await locks
.get(queryId)
.acquire();
try {
// do the rest of your logic here
} catch (error) {
// handle error
} finally {
// always release the lock
lockRelease();
}
});
app.listen(4000, function () {
console.log("Server is running at port 4000");
});
I am working on building a blog API for a practice project, but am using the data from an external API. (There is no authorization required, I am using the JSON data at permission of the developer)
The idea is that the user can enter multiple topic parameters into my API. Then, I make individual requests to the external API for the requested info.
For each topic query, I would like to:
Get the appropriate data from the external API based on the params entered (using a GET request to the URL)
Add the response data to my own array that will be displayed at the end.
Check if each object already exists in the array (to avoid duplicates).
res.send the array.
My main problem I think has to do with understanding the scope and also promises in Axios. I have tried to read up on the concept of promise based requests but I can't seem to understand how to apply this to my code.
I know my code is an overall mess, but if anybody could explain how I can extract the data from the Axios function, I think it could help me get the ball rolling again.
Sorry if this is a super low-level or obvious question - I am self-taught and am still very much a newbie!~ (my code is a pretty big mess right now haha)
Here is a screenshot of the bit of code I need to fix:
router.get('/:tagQuery', function(req, res){
const tagString = req.params.tagQuery;
const tagArray = tagString.split(',');
router.get('/:tag', function(req, res){
const tagString = req.params.tag;
const tagArray = queryString.split(',');
const displayPosts = tagArray.map(function(topic){
const baseUrl = "https://info.io/api/blog/posts";
return axios
.get(baseUrl, {
params: {
tag: tag
}
})
.then(function(response) {
const responseData = response.data.posts;
if (tag === (tagArray[0])){
const responseData = response.data.posts;
displayPosts.push(responseData);
} else {
responseData.forEach(function(post){
// I will write function to check if post already exists in responseData array. Else, add to array
}); // End if/then
})
.catch(function(err) {
console.log(err.message);
}); // End Axios
}); // End Map Function
res.send(displayPosts);
});
Node.js is a single thread non-blocking, and according to your code you will respond with the result before you fetching the data.
you are using .map which will fetch n queries.
use Promise.all to fetch all the requests || Promise.allsettled.
after that inside the .then of Promise.all || promise.allsettled, map your result.
after that respond with the mapped data to the user
router.get('/:tag', function (req, res) {
const tagString = req.params.tag;
const tagArray = queryString.split(',');
const baseUrl = "https://info.io/api/blog/posts";
const topicsPromises=tagArray.map((tobic)=>{
return axios
.get(baseUrl, {
params: {
tag: tag
}
})
});
Promise.all(topicsPromises).then(topicsArr=>{
//all the data have been fetched successfully
// loop through the array and handle your business logic for each topic
//send the required data to the user using res.send()
}).catch(err=>{
// error while fetching the data
});
});
your code will be something like this.
note: read first in promise.all and how it is working.
I've currently written middleware to verify an ID exists in an external services (Salesforce). I initially wrote it when it was a single use app, but now I'm trying to make it work with different routes, so I want it to be fairly generic.
I don't even know if middleware is the right way to go, or if I should just call the function before saving the specific form.
I've got a form where someone puts in some information about a project, and the salesforce ID. For background, the salesforce ID is actually an auto-increment number, and I need to convert that to the actual salesforce system ID before I use jsForce to create a new object linked to that ID.
My route looks like this:
router.post('/invoice/add', ensureLoggedIn, invoiceController.validateInvoice, catchErrors(sfdc.validateSFID), catchErrors(invoiceController.saveInvoice))
So, I've got a middleware that does this:
exports.validateSFID = async(req, res, next) => {
const salesforceProjectNumber = req.body.SF_Opportunity
const sfResult = await conn.search(`FIND ... long query`, (err, result) => {
if (err || result.searchRecords.length !== 1) {
req.flash('error', 'Unable to find a Salesforce Job with that ID number.')
console.error(`ERROR: ${req.user.displayName} errored when looking up job number ${salesforceProjectNumber}.`)
return result
}
})
if (sfResult.searchRecords.length > 0) {
req.body.salesforce_Opportunity_id = sfResult.searchRecords[0].Id //Create a generic variable to hold the salesforce opportunity so it works regardless of the custom object name
res.locals.Opportunity_Clean_Name = sfResult.searchRecords[0].Name
}
next()
}
The query rarely throws an error, but in this case, an error is basically returning !1 records.
When that happens, I want to flash a message on the screen saying the ID wasn't found, but keep the form filled in.
When an ID is found, I want to proceed to save it and NOT display the form fields anymore.
This middleware needs to work regardless of the form I'm using, I want to be able to pipe in the middleware from any form that might require a user to enter a salesforce job as a field.
Any thoughts on how best to handle it all?
You can use your middleware by using app.use() function
app.use((req, res, next) => {
// Every time a request has made, this middleware will fire
console.log('Howdy');
})
I want to setup temporary routes with a unique random string path on ExpressJS. These routes should be dynamically created and should give a 404 if somebody tries to use ie: http://example.com/login/a434bcd34d920bdfe 30 min after that uniqueid was created.
Any ideas on how to do that? I'm pretty new to NodeJS but judging from what I've seen there should be a library that does that :)
Something like this maybe?
// =========== app.js ============
app.get('/generate_url', function(req, res) {
// random string: "a434bcd34d920bdfe"
var extension = randomstring.generate();
var dynamicController = require('./login/'+extension);
dynamicController.init(app);
// Should expire in 20 minutes
dynamicController.expire(20*60)
res.status(200).send();
}
// =========== login.js ============
login.post('/login/:uniqueid', function(req, res) {
// uniqueid should match the extension generated before
var uniqueid = req.query.unique;
var username = req.body.username;
// Do something with this info
}
I think this is far from working fine but at least maybe somebody who's done
You asking two question:
How to create the route.
How to save the data about expired links.
1. How to create the Route:
You have to create a route that receieve the traffic from all the users, and check that the specific URL is valid.
First you create a route that get traffic from all the users.
app.get('/myroute/:id',function(){
/* This route will get any url that start with /myroute/
For example /myroute/abc
/myroute/def
*/
// req.params.id == what the URL is entered
if (is_expired(req.params.id)) return res.end('Sorry your link has expired')
res.send('Great you logged in!')
})
app.post('/login/:uniqueid', function(req, res) {
set_expire(req.params.uniquieid,30*1000*60) //30 minutes = 30*60*1000 miliseconds.
})
2. How to save the data about expired links.
How to implement set_expire and is_expired
You need to implement is using any kind of database. Redis is very good for that. I will show you example how to do it
using setTimeout. It will work. But if the server restart, all the users will be logged out.
users={}
function is_expired(uid){
return users[uid]
}
function set_expire(uid,time){
users[uid]=true
setTimeout(function(){
delete users[uid]
},time)
}
Usually this is done with a generic router handler that then consults some data store to see if the ID sent in the request is valid.
Also, note that a value that comes from a route specification like '/login/:uniqueid' is found in req.params.uniqueid, not in req.query.uniqueid. req.query is for actual query parameters (things after the ? in the URL).
var validIds = {};
// =========== app.js ============
app.get('/generate_url', function(req, res) {
// generate random string: "a434bcd34d920bdfe"
// make sure it's not already in use
var extension;
do {
extension = randomstring.generate();
} while (validIds[extension]);
// save this random string as a valid ID and store the expiration time
validIds[extension] = {expiration: Date.now() + 20 * 60 * 1000};
// return the random value to the caller
res.json(extension);
}
app.post('/login/:uniqueid', function(req, res) {
// uniqueid should match the extension generated before
var uniqueid = req.params.unique;
var username = req.body.username;
// now see if the id is still valid
var idInfo = validIds[uniqueid];
if (!idInfo || idInfo.expiration < Date.now()) {
return res.sendStatus(404);
}
// id is valid, do something with this info
}
Then, you can have some interval timer that regular cleans up any expired ids. This is just housekeeping to keep the validIds object from growing forever - the actual expiration value is still checked before validating the id.
// clean up expired validIds object every 10 minutes
setInterval(function() {
var now = Date.now();
for (var id in validIds) {
if (validIds[id].expiration < now) {
delete validIds[id];
}
}
}, 10 * 60 * 1000);
Note: Since you probably want your uniqueIDs to survive a server restart, you need to regularly persist them to some sort of backing store (database, flat file, etc...) and then you need to read that data in upon server startup.
If you want any sort of user activity to "renew" the timeout on the uniqueid, you can install some middleware that resets the time at an appropriate access.
I have this route that queries an API to get information about the project. It uses super agent to post a get request, passes along some headers with the .set and project_results should contain the data.
Now here is the issue: I can load the page for this particular route/project 20 times... 19 of the times it will work perfectly but randomly it will throw an error:
TypeError: Cannot read property 'creator' of null
This error points out the line: with if(project.creator == req.signedCookies.user_3rb._id) {
So I know it is making it past the: if(project_results.status == '200') {
and since I am looking at the same project over and over and over (and I know all projects have a creator I checked the DB)
my question would be why sometimes does it not find this property of the project_results variable? Its very inconsistent.. I would think project_results is completely populated before going through the code... since its passing the status check we know there is at least some data in the project_results variable..
app.get('/user/projects/:project_id', function(req, res, next) {
var agent = superagent.agent();
var project = {};
project.id = req.params.project_id;
agent
.get(apihost + '/api/project/'+project.id)
.set('api_key', apikey)
.set('access_token', user.access_token)
.end(function(project_error, project_results) {
if (project_error) {
console.log(project_error);
}
if(project_results.status == '200') {
project = JSON.parse(project_results.text);
// Check if we are the owner of the project
if(project.creator == req.signedCookies.user_3rb._id) {
project.owner = true;
}
......
I'm one of the contributors to superagent.
I'd recommend isolating a test case instead of trying to debug directly in your app.
One simple way is to create a very short express app that always returns a hardcoded, perfect response to the route you're trying to reach - then see if project is still null 1 in 20 times:
var app = express();
app.use(function(req, res, next) {
res.send({
// your data hardcoded here
});
});
app.listen(3000);
If it's still null 1 in 20 times, then please submit it as an issue and I'll check it out. If not, then likely something else is going on - perhaps the data isn't being consistently fetched from the db, or there's a race condition between the fetch and the rendering.