I'm running a small server that needs to receive webforms. The server checks the request and sends back "success" or "fail" which is then displayed on the form (client screen).
Now, checking the form may take a few seconds, so the user may be tempted to send the form again.
What is the corret way to ignore the second request?
So far I have come out with this solutions: If the form is duplicate of the previous one
Don't check and send some server error back (like 429, or 102, or some other one)
Close directly the connection req.destroy();res.destroy();
Ignore the request and exit from the requestListener function.
With solution 1 and 2 the form (on client's browser) displays a message error (even if the first request they sent was correct, so as the duplicates). So it's not a good one.
Solution 3 gives the desired outcome... but I'm not sure if it is the right way around it... basically not changing req and res instead of destroying them. Could this cause issues, or slow down the server? (like... do they stack up?). Of course the first request, once it has been checked, will be sent back with the outcome code. My concern is with the duplicate requests, which I don't destroy nor answer...
Some details on the setup: Nodejs application using the very default code by the http module.
const http = require("http");
const requestListener = function (req, res) {
var requestBody = '';
req.on('data', (data)=>{
requestBody += data;
});
req.on('end', ()=>{
if (isduplicate(requestBody))
return;
else
evalRequest(requestBody, res);
})
}
Related
I am sorry that I can't come up with a better title.
I always have this problem (when coding in node.js also python) but I think my solution is kind dirty.
I am here to seek a better solution for this problem.
Here is the scenario:
Your server is doing a very very heavy task upon a special http request (like generating browser screenshot for an URL/generating game server banner with statistics). Whoever did a HTTP request to your server will get the same response. The response will be cached for a long time.
For example, in the browser screenshot generating HTTP request, your server is expected to spawn a phantomjs, capture the screenshot, save it and cache it for a long time, then respond with the PNG captured. The HTTP request after this should hit the cache.
The pseudo code to scenario:
server.get(":urlname.png", function(req, res, next) {
var cached = cache.get(req.params_urlname);
if (cached) {
res.send(cached);
return;
}
// This will take very long time
generateScreenshot(req.params_urlname, function(pngData) {
cache.set(req.params_urlname, pngData, LONG_TIME);
res.send(cached);
});
});
Here is the problem:
Imagine that you have a screenshot generating URL
(http://yourserver.com/generate-screenshot/google.png). The screenshot
is not generated nor cached yet.
Your posted the URL in a very popular forum, and there are 1000 HTTP requests to the that URL at the same time! It means that your server will have to spawn 1000 phantomjs and all of them together will generate screenshot of google.com at the same time, which is crazy!
In other words, the heavy function should be executed only once for generating cache.
My current code solution to the problem:
var pendingResponse = {};
server.get(":urlname.png", function(req, res, next) {
var cached = cache.get(req.params_urlname);
if (cached) {
res.send(cached);
return;
}
// The screenshot is currently generating for other request. Let's mark this response as pending.
if (req.params_urlname in pendingResponse) {
pendingResponse[req.params_urlname].push(res);
return;
}
// The screenshot needs to be generated now. Let's mark the future response as pending.
req.params_urlname[req.params_urlname] = [];
// This will take very long time
generateScreenshot(req.params_urlname, function(pngData) {
cache.set(req.params_urlname, pngData, LONG_TIME);
res.send(cached);
// Let's respond all the pending responses with the PNG data as well.
for (var i in pendingResponse[req.params_urlname]) {
var pRes = pendingResponse[req.params_urlname][i];
pRes.send(cached);
}
// No longer mark the future responses as pending.
delete pendingResponse[req.params_urlname];
});
});
This solution works. However, I consider this solution dirty, because it not reusable at all. Also, I think it may cause resource leak. Is there any better solution / library?
Here's a proof-of-concept server doing this result caching using a memoizee package (not only removes the necessity to cache computations in progress, but also allows to remove the "cache" altogether):
var express = require('express');
var memoize = require('memoizee');
function longComputation(urlName, cb) {
console.log('called for ' + urlName);
setTimeout(function () {
console.log('done for ' + urlName);
cb();
}, 5000);
}
var memoizedLongComputation = memoize(longComputation, {async: true, maxAge: 20000});
var app = express();
app.get('/hang/:urlname', function (req, res, next) {
memoizedLongComputation(req.params.urlname, function () {
res.send('hang over');
});
});
app.listen(3000);
Here we make the result be cached for 20 seconds.
When I start the server and then run in the shell
for i in `seq 1 10`; do curl http://localhost:3000/hang/url1; done
(or just open several browser tabs and quickly navigate them all to http://localhost:3000/hang/url1), I see one "called for url1" and in 5 s one "done for url1" message in the console, meaning only one "real" longComputation call was made. If I repeat it shortly after (less than in 20 s), there are no additional messages, and results are returned instantaneously, because they are cached. If I repeat the command later (in more than 20 s), there's again one call only.
I have 2 clients and one node.js server url - localhost:8888/ServerRequest. The First client GETs from this url and waits for 20 seconds to see if the Second client has POSTed some data for the first client within the 20 second timeout period or not.If the second client did POST before the timeout, then that value is returned to the GET request, else a default value is returned for the GET request. I am not sure what is the best way to implement this. I am trying something like this, but it is not working as desired -
function ServerRequest(response, postData , request)
{
var id;
if(request.method == "GET")
{
id= setTimeout(function( )
{
// handle timeout here
console.log("Got a timeout, sending default value");
cmd = "DefaultVal";
response.write("<?xml version=\"1.0\" encoding=\"UTF-8\"?><list id=\"20101001\"><com type=\"" + cmd + "\"></com></list>")
response.end()
},20000);
}
else if(request.method == "POST")
{
console.log("Received POST, sending POSTed value");
cmd = postData;
//Cancel Timeout
clearTimeout(id);
console.log(" \n Received POST")
response.write("<?xml version=\"1.0\" encoding=\"UTF-8\"?><list id=\"20101001\"><com type=\"" + cmd + "\"></com></list>")
response.end()
}
}
Another approach in my mind was to use 2 separate URLs - One for GET Request (/ServerRequest) and the other for POST Request (/PostData). But then how will I pass the POSTed data from one URL to the other if received before the timeout?
EDIT: I think I know now what I exactly need. I need to implement a longpoll, where a client sends a GET request, and waits for a timeout period (the data might not be immediately available to consume, so it waits for 20 seconds for some other client to POST some data for the first client to consume). In case timeout occurs, a default value is returned in response to the GET request from the first client. I'm working on the longpoll implementation I found here, I'll update if I am able to succeed in what I'm trying. If someone can point me or provide me with a better example, it will be helpful.
Edit: removed my original code after a more careful reading of the question.
The best solution would probably be websockets the browser will appear to hang waiting for 20 seconds.
Using a library like socket.io you can do this
var io = require('socket.io').listen(8888);
function postHandler(req, data, res){
io.sockets.emit("response" , data)
}
then client side
<script src="/socket.io/socket.io.js"></script>
<script>
var socket = io.connect('http://localhost:8888');
socket.on('response', function (data) {
console.log(data);
});
</script>
I'm really new to node.js so please bear with me if I'm making a obvious mistake.
To understand node.js, i'm trying to create a webserver that basically:
1) update the page with appending "hello world" everytime the root url (localhost:8000/) is hit.
2) user can go to another url (localhost:8000/getChatData) and it will display all the data built up from the url (localhost:8000/) being triggered
Problem I'm experiencing:
1) I'm having issue with displaying that data on the rendered page. I have a timer that should call get_data() ever second and update the screen with the data variable that stores the appended output. Specifically this line below response.simpleText(200, data); isn't working correctly.
The file
// Load the node-router library by creationix
var server = require('C:\\Personal\\ChatPrototype\\node\\node-router').getServer();
var data = null;
// Configure our HTTP server to respond with Hello World the root request
server.get("/", function (request, response) {
if(data != null)
{
data = data + "hello world\n";
}
else
{
data = "hellow world\n";
}
response.writeHead(200, {'Content-Type': 'text/plain'});
console.log(data);
response.simpleText(200, data);
response.end();
});
// Configure our HTTP server to respond with Hello World the root request
server.get("/getChatData", function (request, response) {
setInterval( function() { get_data(response); }, 1000 );
});
function get_data(response)
{
if(data != null)
{
response.writeHead(200, {'Content-Type': 'text/plain'});
response.simpleText(200, data);
console.log("data:" + data);
response.end();
}
else
{
console.log("no data");
}
}
// Listen on port 8080 on localhost
server.listen(8000, "localhost");
If there is a better way to do this, please let me know. The goal is to basically have a way for a server to call a url to update a variable and have another html page to report/display the updated data dynamically every second.
Thanks,
D
The client server model works by a client sending a request to the server and the server in return sends a response. The server can not send a response to the client that the client hasn't asked for. The client initiates the request. Therefore you cannot have the server changing the response object on an interval.
The client will not get these changes to the requests. How something like this is usually handled as through AJAX the initial response from the server sends Javascript code to the client that initiates requests to the server on an interval.
setTimeout accepts function without parameter which is obvious as it will be executed later in time. All values you need in that function should be available at the point of time. In you case, the response object that you are trying to pass, is a local instance which has scope only inside the server.get's callback (where you set the setTimeout).
There are several ways you can resolve this issue. you can keep a copy of the response instance in the outer scope where get_data belongs or you can move the get_data entirely inside and remove setTimeout. The first solution is not recommended as if getChatData is called several times in 1sec the last copy will be prevailing.
But my suggestion would be to keep the data in database and show it once getChatData is called.
I am writing a webapp, using express.js.
My webapp achieves the following
User posts 100 json objects
Each json object is processed via a service call
Once the service call is completed, a session variable is incremented
On incrementation of the session variable, a server side event must be sent to the client to update the progress bar
How do i achieve listening on a session variable change to trigger a server-sent event?
Listening to a variable change is not the only solution I seek?
I need to achieve sending a server-sent event once a JSON object is processed.
Any appropriate suggestion is welcome
Edit (based on Alberto Zaccagni's comment)
My code looks like this:
function processRecords(cmRecords,requestObject,responseObject)
{
for (var index = 0; index < cmRecords.length; index++)
{
post_options.body = cmRecords[index];
request.post(post_options,function(err,res,body)
{
if(requestObject.session.processedcount)
requestObject.session.processedcount = requestObject.session.processedcount + 1;
else
requestObject.session.processedcount = 1;
if(err)
{
appLog.error('Error Occured %j',err);
}
else
{
appLog.debug('CMResponse: %j',body);
}
var percentage = (requestObject.session.processedcount / requestObject.session.totalCount) * 100;
responseObject.set('Content-Type','text/event-stream');
responseObject.json({'event':'progress','data':percentage});
});
};
}
When the first record is updated and a server side event is triggered using the responseObject (express response object)
When the second record is updated and I try triggering a server side event using the same responseObject. I get an error saying cannot set header to a response that has already been sent
It's hard to know exactly what the situation is without seeing the routes/actions you have in your main application...
However, I believe the issue you are running into is that you are trying to send two sets of headers to the client (browser), which is not allowed. The reason this is not allowed is because the browser does not allow you to change the content type of a response after you have sent the initial response...as it uses that as an indicator of how to process the response you are sending it. You can't change either of these (or any other headers) after you have sent them to a client once (one request -> one response -> one set of headers back to the client). This prevents your server from appearing schizophrenic (by switching from a "200 Ok" response to a "400 Bad Request," for example).
In this case, on the initial request, you are telling the client "Hey, this was a valid request and here is my response (via the status of 200 which is either set elsewhere or being assumed by ExpressJS), and please keep the communication channel open so I can send you updates (by setting your content type to text/event-stream)".
As far as how to "fix" this, there are many options. When I've done this, I've used the pub/sub feature of redis to act as the "pipe" that connects everything up. So, the flow has been like this:
Some client sends a request to /your-event-stream-url
In this request, you set up your Redis subscriber. Anything that comes in on this subscription can be handled however you want. In your case, you want to "send some data down the pipe to the client in a JSON object with at least a data attribute." After you have set up this client, you just return a response of "200 Ok" and set the content type to "text/event-stream." Redis will take care of the rest.
Then, another request is made to another URL endpoint which accomplishes the task of "posting a JSON object" by hitting /your-endpoint-that-processes-json. (Note: obviously this request may be made by the same user/browser...but the application doesn't know/care about that)
In this action, you do the processing of their JSON data, increment your counters, or do whatever...and return a 200 response. However, one of the things you'd do in this action is "publish" a message on the Redis channel your subscribers from step #1 are listening to so the clients get the updates. Technically, this action does not need to return anything to the client, assuming the user will have some type of feedback based on the 200-status code or on the server-sent event that is sent down the pipe...
A tangible example I can give you is this gist, which is part of this article. Note that the article is a couple years old at this point so some of the code may have to be tweaked a bit. Also note this is not guaranteed to be anything more than an example (ie: it has not been "load tested" or anything like that). However, it may help you get started.
I came up with a solution please let me know if this is the right way to do stuff ?
Will this solution work across sessions ?
Server side Code
var events = require('events');
var progressEmitter = new events.EventEmitter();
exports.cleanseMatch = function(req, res)
{
console.log('cleanseMatch Inovked');
var progressTrigger = new events.EventEmitter;
var id = '';
var i = 1;
id = setInterval(function(){
req.session.percentage = (i/10)*100;
i++;
console.log('PCT is: ' + req.session.percentage);
progressEmitter.emit('progress',req.session.percentage)
if(i == 11) {
req.session.percentage = 100;
clearInterval(id);
res.json({'data':'test'});
}
},1000);
}
exports.progress = function(req,res)
{
console.log('progress Inovked');
// console.log('PCT is: ' + req.session.percentage);
res.writeHead(200, {'Content-Type': 'text/event-stream'});
progressEmitter.on('progress',function(percentage){
console.log('progress event fired for : ' + percentage);
res.write("event: progress\n");
res.write("data: "+percentage+"\n\n");
});
}
Client Side Code
var source = new EventSource('progress');
source.addEventListener('progress', function(e) {
var percentage = JSON.parse(e.data);
//update progress bar in client
App.updateProgressBar(percentage);
}, false);
I have been working on the answer from this question located here How to make a socket a stream? To connect https response to S3 after imagemagick. As per loganfsmyth recommendation I commented the req.end(image) line however when I attempt to upload a file the server simply times out. I experience similar behaviour when I uncomment the req.end(image) line with the exception that the image successfully uploadsto S3. Can someone clarify for me which way is correct also if it is right to uncomment the req.end(image) line what is the best way to send a response to the browser to prevent it from timing out?
https.get(JSON.parse(queryResponse).data.url,function(res){
graphicsmagick(res)
.resize('50','50')
.stream(function (err, stdout, stderr) {
ws. = fs.createWriteStream(output)
i = []
stdout.on('data',function(data){
i.push(data)
})
stdout.on('close',function(){
var image = Buffer.concat(i)
var req = S3Client.put("new-file-name",{
'Content-Length' : image.length
,'Content-Type' : res.headers['content-type']
})
req.on('response',function(res){ //prepare 'response' callback from S3
if (200 == res.statusCode)
console.log('it worked')
})
//req.end(image) //send the content of the file and an end
})
})
})
Basically the page was being requested twice which caused the image to be overwritten because of the favicon. node.js page refresh calling resources twice?
In the question you link to, the user was using putStream, so calling req.end() is incorrect, however in your case you are using put directly, so you need to call req.end(). Otherwise with it commented out, you never actually use the image value, except for the length, so you never send the image data.
It is hard to tell without seeing the server handler that actually runs this code, but you need to (optionally) return some response, and then .end() the actual connection to the browser too, or it will set there waiting.
So if you have something like this
http.createServer(function(req, browserResponse){
// Other code.
req.on('response',function(s3res){ //prepare 'response' callback from S3
if (200 == s3res.statusCode) console.log('it worked')
// Close the response. You also pass it data to send to the browser.
browserResponse.end();
})
// Other code.
});