I'm working on two node.js tutorials at the moment and while I understand what is going on within each tutorial, I clearly don't understand what's going on that well.
The following code listens for "data" events and then adds new chunks of data to a variable named postData. Another listener sends this data along with other stuff to my route.js file.
request.addListener("data", function (postDataChunk) {
postData += postDataChunk;
console.log("Received POST data chunk '" + postDataChunk + "'.");
});
request.addListener("end", function () {
route(handle, pathname, response, postData);
});
The following code creates a variable, tailChild, that spawns the shell command 'tail' on my system log and then attempts to add this data to my postData variable:
var spawn = require('child_process').spawn;
var tail_child = spawn('tail', ['-f', '/var/log/system.log']);
tail_child.stdout.on('data', function (data) {
postData += data;
console.log("TAIL READING: " + data);
});
tail_child.stdout.on('end', function () {
route(handle, pathname, response, postData);
});
Now my console is updated in realtime with system.log data but my browser times out with a "No data received error."
I've tried tweaking the code above to figure what is going wrong and as near as I can tell node is telling me that var data is null so it is adding nothing to var postData. This doesn't make sense to me since console.log("TAIL READING: " + data) gives me the results of spawn('tail', ['-f', '/var/log/system.log']) in my terminal window. Clearly var data is not null.
Edit:
Here's a pastebin link to my server.js code
tail -f won't trigger the end callback so you never respond to the user.
Related
So I'm fairly new to node js, and am having trouble wrapping my head around asynchronous programming. I'm trying to get a JSON from a website and pass it to a variable for use later, to test I have been using this code:
var https = require("https");
var a;
function getter(url){
var request = https.get(url, function(response){
var body = "";
response.on("data", function(chunk){
body += chunk;
});
response.on("end", function(){
if(response.statusCode === 200){
try{
a = JSON.parse(body);
}catch(err){
console.log(err);
}
}
})
})
};
getter('https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY');
console.log(a);
When I run this I get a as undefined, which seems to make sense from what I've read. But I'm unclear as to what to do from here. How would I go about passing this JSON into a variable?
http.get is asynchronous and executes the event handlers when the events occur. When you call getter() this function immediately returns, ie it does not wait for the events and the next statement console.log(a) is executed.
Furthermore, js is single threaded, and the current execution stack is never interrupted for any other event/callback or whatsoever. So the event handlers can only run if the current execution has come to an end, ie contains noch more statements. Thus, your console.log() will always be executed before any eventhandler of the request, thus a is still undefined.
If you want to continue after the request finished, you have to do it from the eventhandler.
See this excellent presentation for some more details https://youtu.be/8aGhZQkoFbQ
I work with node-red and develop a custom node at the moment that uses websockets to connect to a device and request data from it.
function query(node, msg, callback) {
var uri = 'ws://' + node.config.host + ':' + node.config.port;
var protocol = 'Lux_WS';
node.ws = new WebSocket(uri, protocol);
var login = "LOGIN;" + node.config.password;
node.ws.on('open', function open() {
node.status({fill:"green",shape:"dot",text:"connected"});
node.ws.send(login);
node.ws.send("REFRESH");
});
node.ws.on('message', function (data, flags) {
processResponse(data, node);
});
node.ws.on('close', function(code, reason) {
node.status({fill:"grey",shape:"dot",text:"disconnected"});
});
node.ws.on('error', function(error) {
node.status({fill:"red",shape:"dot",text:"Error " + error});
});
}
In the processResponse function I need process the first response. It gives me an XML with several ids that I need to request further data.
I plan to set up a structure that holds all the data from the first request, and populate it further with the data that results from the id requests.
And that's where my problem starts, whenever I send a query from within the processResponse function, I trigger an event that results in the same function getting called again, but then my structure is empty.
I know that this is due to the async nature of nodejs and the event system, but I simply don't see how to circumvent this behavior or do my code in the right way.
If anybody can recommend examples on how to deal with situations like this or even better could give an example, that would be great!
There were already a few questions here about node.js executing commands and outputting the data, but I still can't get this working. What I want is that using node.js I want to execute a python script that runs for a long time and produces some intermediate outputs. I want to stream these outputs to the client as soon as they are produced. I have tried the following, but what I get is that I get the whole output only once the command has finished. How can I make it pass on the data into the socket in real time? Thanks.
function run_cmd(cmd, args) {
var spawn = require('child_process').spawn,
child = spawn(cmd, args);
return child;
}
io.sockets.on('connection', function (socket) {
var foo = new run_cmd('python', ['test.py']);
foo.stdout.setEncoding('utf-8');
foo.stdout.on('data', function(data) {
console.log('sending data');
io.sockets.emit('terminal', {output: data});;
});
);
all your node.js code is okay.your code sends data only once because your code gets data only once.
The point is puts or print commands are not enough to trigger foo.stdout.on
try adding '$stdout.flush' at the point you want to send chunk in ruby code.
You need to order explicitly to output data.
here is my test code.
js
var spawn = require('child_process').spawn;
var cmd = spawn('ruby', ['testRuby.rb']);
var counter = 0;
cmd.stdout.on('data', function(data) {
counter ++;
console.log('stdout: ' + data);
});
cmd.stderr.on('data', function(data) {
console.log('stderr: ' + data);
});
cmd.on('exit', function(code) {
console.log('exit code: ' + code);
console.log(counter);
});
testRuby.rb
def execute_each_sec(sleep_sec)
yield
sleep sleep_sec
end
5.times do
execute_each_sec(1) do ||
puts "CurrentTime:#{Time.now}"
$stdout.flush
end
end
as you can see I'm calling $stdout.flush to output data explicitly in testRuby.rb.
if I remove that,node.js won't get anything until execution of testRuby.rb's finished.
edited
lol my bad. I used ruby instead of python.
in the case of python, use sys.stdout.flush() like svkk says
Edit:
In python you can also use -u flag to force it to flush after each print.
I need to allow the user of my app to download a file with Meteor. Currently what I do is when the user requests to download a file I enter into a "fileRequests" collection in Mongo a document with the file location and a timestamp of the request and return the ID of the newly created request. When the client gets the new ID it imediately goes to mydomain.com/uploads/:id. I then use something like this to intercept the request before Meteor does:
var connect = Npm.require("connect");
var Fiber = Npm.require("fibers");
var path = Npm.require('path');
var fs = Npm.require("fs");
var mime = Npm.require("mime");
__meteor_bootstrap__.app
.use(connect.query())
.use(connect.bodyParser()) //I add this for file-uploading
.use(function (req, res, next) {
Fiber(function() {
if(req.method == "GET") {
// get the id here, and stream the file using fs.createReadStream();
}
next();
}).run();
});
I check to make sure the file request was made less than 5 seconds ago, and I immediately delete the request document after I've queried it.
This works, and is secure(enough) I think. No one can make a request without being logged in and 5 seconds is a pretty small window for someone to be able to highjack the created request URL but I just don't feel right with my solution. It feels dirty!
So I attempted to use Meteor-Router to accomplish the same thing. That way I can check if they're logged in correctly without doing the 5 second open to the world trickery.
So here's the code I wrote for that:
Meteor.Router.add('/uploads/:id', function(id) {
var path = Npm.require('path');
var fs = Npm.require("fs");
var mime = Npm.require("mime");
var res = this.response;
var file = FileSystem.findOne({ _id: id });
if(typeof file !== "undefined") {
var filename = path.basename(file.filePath);
var filePath = '/var/MeteorDMS/uploads/' + filename;
var stat = fs.statSync(filePath);
res.setHeader('Content-Disposition', 'attachment; filename=' + filename);
res.setHeader('Content-Type', mime.lookup(filePath));
res.setHeader('Content-Length', stat.size);
var filestream = fs.createReadStream(filePath);
filestream.pipe(res);
return;
}
});
This looks great, fits right in with the rest of the code and is easy to read, no hacking involved, BUT! It doesn't work! The browser spins and spins and never quite knows what to do. I have ZERO error messages coming up. I can keep using the app on other tabs. I don't know what it's doing, it never stops "loading". If I restart the server, I get a 0 byte file with all the correct headers, but I don't get the data.
Any help is greatly appreciated!!
EDIT:
After digging around a bit more, I noticed that trying to turn the response object into a JSON object results in a circular structure error.
Now the interesting thing about this is that when I listen to the filestream for the "data" event, and attempt to stringify the response object I don't get that error. But if I attempt to do the same thing in my first solution(listen to "data" and stringify the response) I get the error again.
So using the Meteor-Router solution something is happening to the response object. I also noticed that on the "data" event response.finished is flagged as true.
filestream.on('data', function(data) {
fs.writeFile('/var/MeteorDMS/afterData', JSON.stringify(res));
});
The Meteor router installs a middleware to do the routing. All Connect middleware either MUST call next() (exactly once) to indicate that the response is not yet settled or MUST settle the response by calling res.end() or by piping to the response. It is not allowed to do both.
I studied the source code of the middleware (see below). We see that we can return false to tell the middleware to call next(). This means we declare that this route did not settle the response and we would like to let other middleware do their work.
Or we can return a template name, a text, an array [status, text] or an array [status, headers, text], and the middleware will settle the response on our behalf by calling res.end() using the data we returned.
However, by piping to the response, we already settled the response. The Meteor router should not call next() nor res.end().
We solved the problem by forking the Meteor router and making a small change. We replaced the else in line 87 (after if (output === false)) by:
else if (typeof(output)!="undefined") {
See the commit with sha 8d8fc23d9c in my fork.
This way return; in the route method will tell the router to do nothing. Of course you already settled the response by piping to it.
Source code of the middleware as in the commit with sha f910a090ae:
// hook up the serving
__meteor_bootstrap__.app
.use(connect.query()) // <- XXX: we can probably assume accounts did this
.use(this._config.requestParser(this._config.bodyParser))
.use(function(req, res, next) {
// need to wrap in a fiber in case they do something async
// (e.g. in the database)
if(typeof(Fiber)=="undefined") Fiber = Npm.require('fibers');
Fiber(function() {
var output = Meteor.Router.match(req, res);
if (output === false) {
return next();
} else {
// parse out the various type of response we can have
// array can be
// [content], [status, content], [status, headers, content]
if (_.isArray(output)) {
// copy the array so we aren't actually modifying it!
output = output.slice(0);
if (output.length === 3) {
var headers = output.splice(1, 1)[0];
_.each(headers, function(value, key) {
res.setHeader(key, value);
});
}
if (output.length === 2) {
res.statusCode = output.shift();
}
output = output[0];
}
if (_.isNumber(output)) {
res.statusCode = output;
output = '';
}
return res.end(output);
}
}).run();
});
I've read about nodejs event-loop, but what i don't understand well is:
i've created a simple http-server that logs the whole request post-data to a file .
i used apache-ab to flood it with a 700 kb file for-each request it does .
i imagined that each request will write some chunks after each other each tick in the event-loop, but i found that the full post data is written completely after each request and i don't know why , and i cannot understand it .
i'm using something like this
stream = require('fs').createWriteStream('path/to/log.file', {flags: 'a'})
log = function(data){
return stream.write(data)
}
require('http').createServer(function(req, res)
{
// or req.pipe(stream)
req.on('data', function(chunk){
log(chunk.toString() + "\r\n")
})
req.on('end', function(){
res.end("ok")
})
}).listen(8000)
sorry for my bad English :)
I edited the code to output the chunk size and also use an easily identifiable word 'SQUIRREL' to search for in the log file. I also sent the image file using curl instead of apache ab, mainly because I do not have it setup.
If you look at the output of http in the terminal you are running it, you will see the chunk size for each chunk as it is processed, which in my case was 65536 (as you alluded to in your comments we should see). In a text search, I see the word SQUIRREL one time for each chunk that was processed.
There is nothing wrong with your code, hopefully making these changes will allow you to see what you are expecting to see.
stream = require('fs').createWriteStream('./logfile.txt', {flags: 'a'});
log = function(data){
return stream.write(data);
};
require('http').createServer(function(req, res)
{
// or req.pipe(stream)
req.on('data', function(chunk){
log(chunk.toString() + "SQUIRREL ");
console.log(chunk.length);
})
req.on('end', function(){
res.end("ok");
})
}).listen(8000)
curl --data-binary "#IMAG0152.jpg" http://localhost:8000
thanks to everyone tried to help, i found the main issue
the stream.write was returning false i've added the following code to use drain event with .pause() and .resume() and the
problem solved
ok = stream.write(chunk)
if ( ! ok )
{
req.pause()
stream.once('drain', function(){
req.resume()
stream.write(chunk)
})
}