I'm trying to implement a simple HTTP endpoint for an application written in node.js. I've created the HTTP server, but now I'm stuck on reading the request content body:
http.createServer(function(r, s) {
console.log(r.method, r.url, r.headers);
console.log(r.read());
s.write("OK");
s.end();
}).listen(42646);
Request's method, URL and headers are printed correctly, but r.read() is always NULL. I can say it's not a problem in how the request is made, because content-length header is greater than zero on server side.
Documentation says r is a http.IncomingMessage object that implements the Readable Stream interface, so why it's not working?
'readable' event is wrong, it incorrectly adds an extra null character to the end of the body string
Processing the stream with chunks using 'data' event:
http.createServer((r, s) => {
console.log(r.method, r.url, r.headers);
let body = '';
r.on('data', (chunk) => {
body += chunk;
});
r.on('end', () => {
console.log(body);
s.write('OK');
s.end();
});
}).listen(42646);
Ok, I think I've found the solution. The r stream (like everything else in node.js, stupid me...) should be read in an async event-driven way:
http.createServer(function(r, s) {
console.log(r.method, r.url, r.headers);
var body = "";
r.on('readable', function() {
body += r.read();
});
r.on('end', function() {
console.log(body);
s.write("OK");
s.end();
});
}).listen(42646);
Related
Im using ytdl-core to get the direct urls for youtube videos (the long urls that start with https://r1-- etc...), and using https.get to download them. ~70% of the time when res.end is called, it returns an empty buffer instead of the data for the video.
this is the code that im currently using to handle the http request:
function largeHttpGet(url, callback, progress) {
try{
var _current = 0;
var _buffer = [];
var _req = https.get(url, function (res) {
var _total = parseInt(res.headers['content-length'], 10);
res.on('data', function (chunk) {
_buffer.push(chunk);
_current = parseFloat(_current + chunk.length);
progress(_current / _total * 100);
})
res.on('end', function () {
callback(Buffer.concat(_buffer));
//this returns an empty buffer the majority of the time
})
})
}
catch(error){
console.log("[HTTP]: " + error);
}
}
and this is what i use to get the youtube video's info:
getVidInfo(link, function(info){
back.send("return", {"key": key, "data": info} );
})
This error only occurs with the youtube video links, all other requests work consistently.
It may be due to u sing on('end'... instead of `on('finish'...)
finish event
related SO link to finish
As there may still be additional work that has not been completed yet.
Found the issue, when the request failed it was because youtube returned a redirect code (302) instead of a direct link, following the redirection lead to the actual url, which downloaded without issue.
I'm trying to pipe an image stored on Amazon S3 using node-request. However, sometimes an image doesn't exist, which is exposed by S3 as a status code 403. I'm struggling to see how I can pipe in case of success (200) but take an alternative action in case of a non-200.
Using the abort() method seemed like the way to go but getting an r.abort is not a function, even though it should be available on the request.
// context: inside a request handler, where `res` is the response to write to
const r = request(url)
.on('response', function (response) {
if (response.statusCode !== 200) {
r.abort(); //failing
// I want to stop the piping here and do whatever instead.
}
})
.pipe(res);
Thoughts?
To answer my own question: don't start piping until sure it's correct:
request(url)
.on('response', function (r) {
if (r.statusCode !== 200) {
//take other action
}
r.pipe(res);
})
var http = require('http');
var map = require('through2-map');
uc = map(function(ch) {
return ch.toString().toUpperCase();
});
server = http.createServer(function(request, response) {
request.on('data',function(chunk){
if (request.method == 'POST') {
//change the data from request to uppercase letters and
//pipe to response.
}
});
});
server.listen(8000);
I have two questions about the code above. First, I read the documentation for request, it said that request is an instance of IncomingMessage, which implements Readable Stream. However, I couldn't find .on method in the Stream documentation. So I don't know what chunk in the callback function in request.on does. Secondly, I want to do some manipulation to the data from request and pipe it to response. Should I pipe from chunk or from request? Thank you for consideration!
is chunk a stream?
nop. The stream is the flow among what the chunks of the whole data are sent.
A simple example, If you read a 1gb file, a stream will read it by chunks of 10k, each chunk will go through your stream, from the beginning to the end, with the right order.
I use a file as example, but a socket, request or whatever streams is based on that idea.
Also, whenever someone sends a request to this server would that entire thing be a chunk?
In the particular case of http requests, only the request body is a stream. It can be the posted files/data. Or the response body of the response. Headers are treated as Objects to apply on the request before the body is written on the socket.
A small example to help you with some concrete code,
var through2 = require('through2');
var Readable = require('stream').Readable;
var s1 = through2(function transform(chunk, enc, cb){
console.log("s1 chunk %s", chunk.toString())
cb(err=null, chunk.toString()+chunk.toString() )
});
var s2 = through2(function transform(chunk, enc, cb){
console.log("s2 chunk %s", chunk.toString())
cb(err=null, chunk)
});
s2.on('data', function (data) {
console.log("s2 data %s", data.toString())
})
s1.on('end', function (data) {
console.log("s1 end")
})
s2.on('end', function (data) {
console.log("s2 end")
})
var rs = new Readable;
rs.push('beep '); // this is a chunk
rs.push('boop'); // this is a chunk
rs.push(null); // this is a signal to end the stream
rs.on('end', function (data) {
console.log("rs end")
})
console.log(
".pipe always return piped stream: %s", rs.pipe(s1)===s1
)
s1.pipe(s2)
I would like to suggest you to read more :
https://github.com/substack/stream-handbook
http://maxogden.com/node-streams.html
https://github.com/maxogden/mississippi
All Streams are instances of EventEmitter (docs), that is where the .on method comes from.
Regarding the second question, you MUST pipe from the Stream object (request in this case). The "data" event emits data as a Buffer or a String (the "chunk" argument in the event listener), not a stream.
Manipulating Streams is usually done by implementing a Transform stream (docs). Though there are many NPM packages available that make this process simpler (like through2-map or the like), though in reality, they produce Transform streams.
Consider the following:
var http = require('http');
var map = require('through2-map');
// Transform Stream to uppercase
var uc = map(function(ch) {
return ch.toString().toUpperCase();
});
var server = http.createServer(function(request, response) {
// Pipe from the request to our transform stream
request
.pipe(uc)
// pipe from transfrom stream to response
.pipe(response);
});
server.listen(8000);
You can test by running curl:
$ curl -X POST -d 'foo=bar' http://localhost:8000
# logs FOO=BAR
A http request is returning me a incomplete string.
https.get(url, function(res) {
res.on('data', function(data) {
translationData = data.toString();
resolve(translationData);
})
});
I can't get more than 500 characters.
I suppose my code is vague, but what could cause this problem?
I've tried a lot of approaches but all of them failed.
I've something similar in How to display long messages in logcat, but nothing compared in nodeJS.
The response object you get from http.get is a Stream.
The 'data' event handler is called whenever a chunk of data is received. You need to handle all of the 'data' events and collect their payload until you get an 'end' event in order to get the entire response.
A simple way of doing this is using the concat-stream module.
var concat = require('concat-stream');
https.get(url, function(res) {
res.pipe(concat(function(data) {
// data is the entire response
}));
}
To learn more about streams, read substack's stream handbook
When I make a http request, I need to concatenate the response:
request.on('response', function (response) {
var body = '';
response.on('data', function (chunk) {
body += chunk;
});
...
Why was that implemented this way? Why not output the whole result?
What you're getting back is a stream, which is a very handy construct in node.js. Required reading: https://github.com/substack/stream-handbook
If you want to wait until you've received the whole response, you can do this very easily:
var concat = require('concat-stream');
request.on('response', function(response) {
response.pipe(concat(function(body) {
console.log(body);
}));
});
Node only uses a single process, no thread. This mean that if spend a lot of time doing something you can´t process other things, like for example other client requests...
For that reason when you are coding in node, you need code thinking in async way.
In this scenario, the request could be slowly, and the program will wait for this request doing nothing.
I found this:
Why is node.js asynchronous?
And this is so interesting as well:
http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop