I have a node service that fetches a pdf from an API and serves that pdf.
When I curl or directly open the API, I do see the correct pdf.
But when I serve it from my Node app, I get an empty pdf.
Here's the section of my code that does the pdf render.
} else if (options.type === 'pdf') {
res.writeHead(200, {'content-type' : 'application/pdf', 'content-disposition': 'attachment; filename=invoice.pdf'});
res.end(data.invoice);
I've console.log'ed data.invoice to know it's the right stuff.
typeof(data.invoice) gives string; but I've also tried res.end(new Buffer(data.invoice)); which didn't work either.
Here's the section of my code that fetches the data
var http_options = {
method : options.method
, host : Config.API.host
, path : options.path
, port : Config.API.port
, headers : options.headers
};
var req = http.request(http_options, function (response) {
var raw_response = "";
response.on('data', function (response_data) {
raw_response += response_data.toString();
});
response.on('end', function () {
if (response.statusCode !== 200) {
cb(raw_response);
} else {
cb(false, raw_response);
}
});
});
req.setTimeout(timeout, function () {
req.abort();
cb("API connection timed out");
});
req.on('error', function (error) {
cb("API error while requesting for " + options.path + '\n' + error + '\n' + "http options: " + JSON.stringify(http_options)
});
req.end();
It's quite likely that the toString() and concatenation when you're receiving the PDF are corrupting it. Try writing raw_response to a file (you can use writeFileSync() since this is just a one-time test) and doing a byte-for-byte comparison with the same PDF retrieved with curl.
Note that if the process of string conversion has corrupted it, trying to convert it back to a buffer before sending it won't help. You'll have to keep the whole thing as a buffer from start to finish.
Since you don't intend to modify or read this data in transit, I suggest just using the pipe function to pipe all the data coming in from response out to req. this question has a good sample, but here's an excerpt.
req.on('response', function (proxy_response) {
proxy_response.pipe(response);
response.writeHead(proxy_response.statusCode, proxy_response.headers);
});
Note that there's no reason to convert the chunks coming in from the response from Buffers to something else, just write them through as unmodified buffers, and stream them (this is what pipe will do for you) instead of accumulating them to get maximum efficiency (and node.js streaming hipster points).
Related
I am performing some http calls in an AWS lambda. The number of calls are ~400 per minute.
The calls are performed as in the following snippet
var req = http.get("https://www.google.com", res => {
let body = '';
res.on('data', chunk => {
body += chunk;
});
res.on('end', chunk => {
if (body.includes('html')) {
console.log('Got back healthy response');
} else {
console.log('Got an unexpected response');
}
})
});
req.on('error', e => {
console.log('Got an error response');
})
That is a simple https request. When the Lambda is invoked, it performs ~40 requests in a single execution.
My issue is that at the beginning, everything looks good and all the requests are performed correctly. After a while (that can be after ~30 min) calls start to degrade and I get back "socket hang up ECONNRESET" error.
I then tried using the request module and change the code with the following
const request = require('request');
request("https://www.google.com", function (error, response, body) {
if (!error && response.statusCode == 200 && body.includes('html')) {
console.log('Got back healthy response' );
} else {
console.log('Got an unexpected response');
console.log('Error: ' + error);
console.log('Response: ' + response);
console.log('Body: ' + body);
}
});
In this case, with the same number of requests within the same lambda, same setting I never get the ECONNRESET error.
I'm ok using the request module but I'm curious to know why this was happening with the default http implementation.
Is this caused by socket allocation that the request module handles in a more appropriate way?
I know similar questions have been asked already but I didn't find a good answer for my case.
This isn't really an answer but I cannot write comments.
The main difference I can see is the encoding. The default encoding in request is utf8, whereas in the http module it's buffer. Adding res.setEncoding('utf8'); might be helpful. It might not be faster. In the line body += chunk you just implicitly convert a Buffer to a string so it should be the same.
If adding the setEncoding won't change anything then you might want to report an issue to the nodejs team because it's the same as the example in http_http_get_url_options_callback. They should fix it or change the example.
I am facing issues when trying to process an uploaded picture and write it into a file. Usually this work very well that way:
req.on('data', function (data) {
body += data;
wstream.write(data);
});
// end process data
req.on('end', function () {
wstream.end();
});
In that case written file header looks like that:
PNG ... followed by binary data.
But in some cases written file looks like that:
--72e245e8-38eb-41e8-9118-fc0405e4837c Content-Type: multipart/form-data Content-Disposition: form-data; name=image;
filename=picture.jpg; filename*=utf-8''picture.jpg
As you can imagine, those pictures arent working well anymore until I remove that meta data as content-type etc.
But after I do so, picture is fully functional and useable.
I tried to access the request data and called toString Method to replace the "unwanted" parts, but than I entirely mess up content encoding of that output file and it becomes unuseable at all.
data = data.toString(/---.*/g), "");
Any ideas how to do the trick?
I solved my issue by help of module formidable.
var formidable = require('formidable');
var util = require('util');
form.parse(req, function(err, fields, files) {
logger.debug("Received upload: " + util.inspect({fields: fields, files: files}));
});
form.on('fileBegin', function (name, file){
file.path = "MyPathToDestinationFile";
logger.debug("File upload started for file '" + file.path + "'");
});
form.on('end', function() {
logger.debug("File upload finished for file");
// Send response to client
});
form.on('error', function(err) {
logger.debug("Failed to finish upload due to '" + err + "'");
// Send error to client
});
On my node express server, I am receiving a pdf file. I am using the below code to get the pdf contents from the request
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function() {
console.log('PDF data is '+JSON.stringify(data));
});
Now that PDF content is available on node, I need to send it as it is to a J2EE server. In order to do that, I am first saving the PDF file in the node server, reading it from the node server and then piping it to request.post (https://github.com/request/request)
var req = require('request');
fs.writeFile('abc.pdf', data, 'binary', function(err) {
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
//send our data via POST request
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload')
});
This works fine. However, I feel the part of saving the PDF file on the node server and then reading it is (before posting to the J2EE server using request module) is completely unnecessary, as I am not making any changes to the file.
Once I have the PDF contents in 'data' variable, I would like to directly post them to the J2EE server. However, I have not been able to find a way to use the request module to directly post file contents. I have seen some examples related to POST using request module but they refer to formData. In my case, I don't have formData but instead reading the file from request and directly posting it to the J2EE server.
Is there a way to achieve this and avoid the file write and read?
EDIT
Below is my complete code
function upload(request, response) {
var data = new Buffer('');
request.on('data', function (chunk) {
data = Buffer.concat([data, chunk]);
});
request.on('end', function () {
fs.writeFile('abc.pdf', data, 'binary', function(err){
if (err) {
console.log('Error ' + JSON.stringify(err) );
throw err;
}
var source = fs.createReadStream('abc.pdf');
source.pipe(req.post('http://'+j2ee_host+':'+j2ee_port+'/myj2eeapp/Upload'));
})
})
}
You can pipe directly from the data request to the servlet
var req = require('request');
function upload(request, response) {
var target = req.post('http://'+j2ee_host+':'+j2ee_port+'/myjavaapp/Upload');
request.pipe(target);
target.on('finish', function () {
console.log('All done!');
//send the response or make a completed callback here...
});
}
I am using the Twilio Node Helper Library to make a call and record it.
According to the API link, GET should return a WAV file, but in my case it just returns a json with the recording metadata.
This is what I'm writing:
twilioClient = require('twilio')(config.twilio.acct_sid, config.twilio.auth_token)
var request = twilioClient.recordings('RE01234567890123456789012345678901')
get(function (err, recording){ // <- this "recording" is JSON
It doesn't matter if I tack on a '.mp3' to the end of the SID, I always get a JSON.
Ideally I want to write something like this:
var file = fs.createWriteStream('/Users/yasemin/Desktop/rec.mp3');
twilioClient.recordings('RE01234567890123456789012345678901')
.get(function (err, recording) {
if(!err){ recording.pipe(file); }});
Thanks!
I came across this and had to develop my own code to handle this.
Here is the code I came up with below
con.on('getvmx', function(data){
comModel.find({_id: data.id}, function(err, results){
var https = require('https');
var options = {
host: 'api.twilio.com',
port: 443,
path: '/2010-04-01/Accounts/' + sid + '/Recordings/'+ results[0].sid + '.mp3',
method: 'GET',
auth: sid + ":" + auth,
agent: false
};
var req = https.request(options, function(res) {
res.setEncoding('binary');
var mp3data = '';
res.on('data', function (chunk) {
mp3data += chunk;
});
res.on('end', function(){
try{
var fileName = "/var/www/tcc/public/vm/" + results[0].sid + '.mp3';
fs.writeFile(fileName, mp3data, 'binary', function(err){
if(err){
return console.log(err);
}else{
console.log("File Saved");
con.emit('vmload', results);
}
});
}catch(err){
console.log(err.message);
}
});
});
req.end();
console.log(results);
//load all messages
//load line from reach message
});
});
TLDR: Node Helper Library doesn't have recoded file downloading capability at the moment.
This is the response from Twilio Support:
Looking at the documentation on our web portal, you are certainly
correct, downloading the .wav or .mp3 is possible via API call.
However, from what I can see looking at the Node example code here:
https://www.twilio.com/user/account/developer-tools/api-explorer/recording
And the documentation from the Twilio-Node developer here:
http://twilio.github.io/twilio-node/#recordings
It looks to me like the helper library doesn't actually support direct
downloading, just viewing the recording data. You can download the
application through an HTTP call, as shown in the original docs link
you noted on your Stackoverflow question. Let me know if you need help
with that.
In the mean time, I've reached out to the author of the library to see
if this is by design or a feature to be added to the library. It's
open source of course, so you could make a pull and add it yourself if
you like!
I am still a beginner in Node.js and I am trying to explore as much as I can.
I know that Express.js is a framework used by many people for creating websites in Node.js.
But without using Express.js, I know that it is it possible to read .html files using 'fs.readFile', and then "display" this .html file in the browser.
Is there a way to get user input (say a button click, or fill in a box) from this web page into Node.js? So far, I have not found any examples of this.
Yes, this is possible. Study how the connect bodyParser's urlencoded function works.
When a request comes in from the browser, node is going to represent this as a readable data stream. For web forms, the pattern will be:
Use the request's data and end events to buffer the chunks of data from the stream into a single string.
Parse that string appropriately given its data format. In the case of a web form, this will normally urlencoded (application/x-www-form-urlencoded) MIME type
.
var qs = require('qs'); //https://github.com/visionmedia/node-querystring
function handle(req, res) {
var buf = '';
req.setEncoding('utf8');
req.on('data', function(chunk){
//assemble the request from distinct chunks into a single string
buf += chunk
});
req.on('end', function(){
//OK, you have a usable string request body, parse it and handle it
try {
var formData = qs.parse(buf);
//Yay, it parsed. Now you have your form data
//depending on your form's html, you might have formData.email, for example
} catch (err){
//oops, respond with an error
}
});
}
Tutorial
Long story short:
http.createServer(function (req, res) {
var data = '';
req.on('data', function(chunk) {
console.log("Received body data:");
console.log(chunk);
data += chunk.toString();
});
req.on('end', function() {
console.log('Received Data: ', data);
res.end();
});
}