I have got stream and I need to get stream content into string. I stream from internet using http.get. I also write stream into file, but I don't want to write file and after that open the same file and read from it...
So I need to convert stream into string
Thanks for all advices...
var http = require('http');
var string = '';
var request = http.get("http://www.google.cz", function(response) {
response.on('data', function(response){
string += response.toString();
});
response.on('end', function(string){
console.log(string);
});
});
This works for sure. I am using it.
Instead of using the second nested function, try the toString() method on the stream. This can then be piped wherever you want it to go, and you can also write a through() method that can assign it to a variable or use it directly in the one function.
var http = require('http');
var string = '';
var request = http.get('http://www.google.cz', function (err, data){
if (err) console.log(err);
string = data.toString();
//any other code you may want
});
//anything else
One last note--the http.get() method takes two parameters: the url, and a callback. This requires two parameters, and you may have been getting nothing because it was an empty error message.
Related
var http = require('http');
var map = require('through2-map');
uc = map(function(ch) {
return ch.toString().toUpperCase();
});
server = http.createServer(function(request, response) {
request.on('data',function(chunk){
if (request.method == 'POST') {
//change the data from request to uppercase letters and
//pipe to response.
}
});
});
server.listen(8000);
I have two questions about the code above. First, I read the documentation for request, it said that request is an instance of IncomingMessage, which implements Readable Stream. However, I couldn't find .on method in the Stream documentation. So I don't know what chunk in the callback function in request.on does. Secondly, I want to do some manipulation to the data from request and pipe it to response. Should I pipe from chunk or from request? Thank you for consideration!
is chunk a stream?
nop. The stream is the flow among what the chunks of the whole data are sent.
A simple example, If you read a 1gb file, a stream will read it by chunks of 10k, each chunk will go through your stream, from the beginning to the end, with the right order.
I use a file as example, but a socket, request or whatever streams is based on that idea.
Also, whenever someone sends a request to this server would that entire thing be a chunk?
In the particular case of http requests, only the request body is a stream. It can be the posted files/data. Or the response body of the response. Headers are treated as Objects to apply on the request before the body is written on the socket.
A small example to help you with some concrete code,
var through2 = require('through2');
var Readable = require('stream').Readable;
var s1 = through2(function transform(chunk, enc, cb){
console.log("s1 chunk %s", chunk.toString())
cb(err=null, chunk.toString()+chunk.toString() )
});
var s2 = through2(function transform(chunk, enc, cb){
console.log("s2 chunk %s", chunk.toString())
cb(err=null, chunk)
});
s2.on('data', function (data) {
console.log("s2 data %s", data.toString())
})
s1.on('end', function (data) {
console.log("s1 end")
})
s2.on('end', function (data) {
console.log("s2 end")
})
var rs = new Readable;
rs.push('beep '); // this is a chunk
rs.push('boop'); // this is a chunk
rs.push(null); // this is a signal to end the stream
rs.on('end', function (data) {
console.log("rs end")
})
console.log(
".pipe always return piped stream: %s", rs.pipe(s1)===s1
)
s1.pipe(s2)
I would like to suggest you to read more :
https://github.com/substack/stream-handbook
http://maxogden.com/node-streams.html
https://github.com/maxogden/mississippi
All Streams are instances of EventEmitter (docs), that is where the .on method comes from.
Regarding the second question, you MUST pipe from the Stream object (request in this case). The "data" event emits data as a Buffer or a String (the "chunk" argument in the event listener), not a stream.
Manipulating Streams is usually done by implementing a Transform stream (docs). Though there are many NPM packages available that make this process simpler (like through2-map or the like), though in reality, they produce Transform streams.
Consider the following:
var http = require('http');
var map = require('through2-map');
// Transform Stream to uppercase
var uc = map(function(ch) {
return ch.toString().toUpperCase();
});
var server = http.createServer(function(request, response) {
// Pipe from the request to our transform stream
request
.pipe(uc)
// pipe from transfrom stream to response
.pipe(response);
});
server.listen(8000);
You can test by running curl:
$ curl -X POST -d 'foo=bar' http://localhost:8000
# logs FOO=BAR
I'm using JSONStream to parse the data from server, the data can either be like {"error": "SomeError"} or {"articles":[{"id": 123}]};
My code goes like
var request = require('request');
var JSONStream = require('JSONStream');
var articleIDParser = JSONStream.parse(['articles', true, 'id']);
var errorParser = JSONStream.parse(['error']);
request({url: 'http://XXX/articles.json'})
.pipe(articleIDParser).pipe(errorParser);
errorParser.on('data', function(data) {
console.log(data);
});
articleIDParser.on('data', someFuncHere);
But unlucky, the second parser does not work even when the server returns error.
Am I wrong at pipe function or JSONStream?
Thanks in advance.
Well, I use the following way to solved the problem:
var request({url: 'http://XXX/articles.json'})
dest.pipe(articleIDParser)
dest.pipe(errorParser);
Explanation in Node.js Stream documentation.
The callback function of the 'end' event doesn't have a data parameter. Listen for the 'data' event instead. In case of piping listen for the pipe event on the destination.
var request, JSONStream, articleIDParser, errorParser;
request = require('request');
JSONStream = require('JSONStream');
articleIDParser = JSONStream.parse(['articles', true, 'id']);
errorParser = JSONStream.parse(['error']);
articleIDParser.on('pipe', function (src) {
// some code
});
errorParser.on('pipe', function (src) {
// some code
});
request({url: 'http://XXX/articles.json'}).pipe(articleIDParser).pipe(errorParser);
Note: JSONStream.getParserStream is less ambiguous, one might think you're already parsing while you're just getting the parser/writable stream. If you still have issues please give more information (code) about JSONStream. The Stream module is still marked as unstable by the way.
I have a PNG object which I created using node-png and according the docs it's a "readable and writable Stream".
I would like to convert the PNG object to base64 and send it to the client via socket.io where I'll place the string in an image src.
I've tried many things but it seems that it's not trivial to convert a stream into a string.
Note that the data is created inside Node and not from the file system.
How can I achieve this?
Here is what I did for future readers (this helped too):
png.pack();
var chunks = [];
png.on('data', function (chunk) {
chunks.push(chunk);
console.log('chunk:', chunk.length);
});
png.on('end', function () {
var result = Buffer.concat(chunks);
console.log('final result:', result.length);
io.sockets.emit('image', result.toString('base64'));
});
You do not want to convert the stream into a string but its readable chunk:
stream.on('readable', function() {
var string = stream.read().toString('base64');
// send through websocket
});
You can also do this for the complete data running through the stream:
var data = '';
stream.on('readable', function() {
data += stream.read().toString('base64');
});
stream.on('end', function() {
console.log(data);
});
Depends on if the client requires the complete png picture to be available or if it is okay to have single chunks.
However if you are interested how this could look like in an practical example (with the image being uploaded by HTML5 Drag & Drop) you can checkout messenger.
I am trying to send an http response in node to print results in the browser. The simplified source code is down below. Basically, all the variables are defined somewhere in the program, so that shouldn't be problem. When I try to run the script, I keep getting the error:
http.js:783
throw new TypeError('first argument must be a string or Buffer');
TypeError: first argument must be a string or Buffer
So can someone familiar with node.js or javascript syntax let me know what the problem is?
upload = function(req, res) {
var fileInfos = [obj, obj]; //defined as an array of objects
var counter = 0;
counter -= 1;
if (!counter) {
res.end({files: fileInfos}); //files is defined.
}
};
async.forEach(urls, downloadFile, function (err) { //all params defined.
if(err){
console.error("err");
throw err;
}
else{
http.createServer(function(req, res){
upload(req1, res); //req1 defined as an array of objects.
}).listen(3000, "127.0.0.1");
console.log('Server running at http://127.0.0.1:3000/');
}
});
This error is often caused by an attempt to call response.write with the wrong type of parameter. Looking at the documentation it suggests:
response.end([data], [encoding])#
This method signals to the server that all of the response headers and body have been sent; that server should consider this message complete. The method, response.end(), MUST be called on each response.
If data is specified, it is equivalent to calling response.write(data, encoding) followed by response.end().
Now response.write( chunk, encoding ) expects the chunk to be a string, so it seems possible that when you are calling res.end({files: fileInfos}) it is unable to write the content of that object as a string.
You can use JSON.stringify() to convert the JavaScript object to string before sending it to the client.
res.end(JSON.stringify({files: fileInfos}));
I need to allow the user of my app to download a file with Meteor. Currently what I do is when the user requests to download a file I enter into a "fileRequests" collection in Mongo a document with the file location and a timestamp of the request and return the ID of the newly created request. When the client gets the new ID it imediately goes to mydomain.com/uploads/:id. I then use something like this to intercept the request before Meteor does:
var connect = Npm.require("connect");
var Fiber = Npm.require("fibers");
var path = Npm.require('path');
var fs = Npm.require("fs");
var mime = Npm.require("mime");
__meteor_bootstrap__.app
.use(connect.query())
.use(connect.bodyParser()) //I add this for file-uploading
.use(function (req, res, next) {
Fiber(function() {
if(req.method == "GET") {
// get the id here, and stream the file using fs.createReadStream();
}
next();
}).run();
});
I check to make sure the file request was made less than 5 seconds ago, and I immediately delete the request document after I've queried it.
This works, and is secure(enough) I think. No one can make a request without being logged in and 5 seconds is a pretty small window for someone to be able to highjack the created request URL but I just don't feel right with my solution. It feels dirty!
So I attempted to use Meteor-Router to accomplish the same thing. That way I can check if they're logged in correctly without doing the 5 second open to the world trickery.
So here's the code I wrote for that:
Meteor.Router.add('/uploads/:id', function(id) {
var path = Npm.require('path');
var fs = Npm.require("fs");
var mime = Npm.require("mime");
var res = this.response;
var file = FileSystem.findOne({ _id: id });
if(typeof file !== "undefined") {
var filename = path.basename(file.filePath);
var filePath = '/var/MeteorDMS/uploads/' + filename;
var stat = fs.statSync(filePath);
res.setHeader('Content-Disposition', 'attachment; filename=' + filename);
res.setHeader('Content-Type', mime.lookup(filePath));
res.setHeader('Content-Length', stat.size);
var filestream = fs.createReadStream(filePath);
filestream.pipe(res);
return;
}
});
This looks great, fits right in with the rest of the code and is easy to read, no hacking involved, BUT! It doesn't work! The browser spins and spins and never quite knows what to do. I have ZERO error messages coming up. I can keep using the app on other tabs. I don't know what it's doing, it never stops "loading". If I restart the server, I get a 0 byte file with all the correct headers, but I don't get the data.
Any help is greatly appreciated!!
EDIT:
After digging around a bit more, I noticed that trying to turn the response object into a JSON object results in a circular structure error.
Now the interesting thing about this is that when I listen to the filestream for the "data" event, and attempt to stringify the response object I don't get that error. But if I attempt to do the same thing in my first solution(listen to "data" and stringify the response) I get the error again.
So using the Meteor-Router solution something is happening to the response object. I also noticed that on the "data" event response.finished is flagged as true.
filestream.on('data', function(data) {
fs.writeFile('/var/MeteorDMS/afterData', JSON.stringify(res));
});
The Meteor router installs a middleware to do the routing. All Connect middleware either MUST call next() (exactly once) to indicate that the response is not yet settled or MUST settle the response by calling res.end() or by piping to the response. It is not allowed to do both.
I studied the source code of the middleware (see below). We see that we can return false to tell the middleware to call next(). This means we declare that this route did not settle the response and we would like to let other middleware do their work.
Or we can return a template name, a text, an array [status, text] or an array [status, headers, text], and the middleware will settle the response on our behalf by calling res.end() using the data we returned.
However, by piping to the response, we already settled the response. The Meteor router should not call next() nor res.end().
We solved the problem by forking the Meteor router and making a small change. We replaced the else in line 87 (after if (output === false)) by:
else if (typeof(output)!="undefined") {
See the commit with sha 8d8fc23d9c in my fork.
This way return; in the route method will tell the router to do nothing. Of course you already settled the response by piping to it.
Source code of the middleware as in the commit with sha f910a090ae:
// hook up the serving
__meteor_bootstrap__.app
.use(connect.query()) // <- XXX: we can probably assume accounts did this
.use(this._config.requestParser(this._config.bodyParser))
.use(function(req, res, next) {
// need to wrap in a fiber in case they do something async
// (e.g. in the database)
if(typeof(Fiber)=="undefined") Fiber = Npm.require('fibers');
Fiber(function() {
var output = Meteor.Router.match(req, res);
if (output === false) {
return next();
} else {
// parse out the various type of response we can have
// array can be
// [content], [status, content], [status, headers, content]
if (_.isArray(output)) {
// copy the array so we aren't actually modifying it!
output = output.slice(0);
if (output.length === 3) {
var headers = output.splice(1, 1)[0];
_.each(headers, function(value, key) {
res.setHeader(key, value);
});
}
if (output.length === 2) {
res.statusCode = output.shift();
}
output = output[0];
}
if (_.isNumber(output)) {
res.statusCode = output;
output = '';
}
return res.end(output);
}
}).run();
});