I am using the request library in Node.js to Google's text-to-speech API. I would like to print out the request that is being sent like in this python example.
Here is my code:
const request = require('request');
const headers = {headers: {'input': {'text':'I want to say this'}, 'voice':{ 'languageCode' : 'en-US'},'audioConfig':{'audioEncoding': 'MP3'}}}
request.post('https://texttospeech.googleapis.com/v1beta1/text:synthesize?key=API_KEY',headers, (error, res, body) => {
if (error) {
console.error(error)
return
}
console.log(`statusCode: ${res.statusCode}`)
console.log(body)
})
Simplest way to do this is to start a netcat server on any port:
$ nc -l -p 8080
and change the URL to localhost:
https://localhost:8080/v1beta1/text:synthesize?key=API_KEY
Obviously, you won't be able to see the response, but the entire raw request data will be available for you to inspect in the terminal you have netcat running
This is documented here:
There are at least three ways to debug the operation of request:
Launch the node process like NODE_DEBUG=request node script.js (lib,request,otherlib works too).
Set require('request').debug = true at any time (this does the same thing as #1).
Use the request-debug module to view request and response headers and bodies.
Related
I have a node.js GET API endpoint that calls some backend services to get data.
app.get('/request_backend_data', function(req, res) {
---------------------
}
When there is a delay getting a response back from the backend services, this endpoint(request_backend_data) is getting triggered exactly after 2 minutes. I have checked my application code, but there is no retry logic written anywhere when there is a delay.
Does node.js API endpoint gets called twice in any case(like delay or timeout)?
There might be a few reasons:
some chrome extensions might cause bugs. Those chrome extensions have been causing a lot of issues recently. run your app on a different browser. If there is no issue, that means it is chrome-specific problem.
express might be making requests for favicon.ico. In order to prevent this, use this module : https://www.npmjs.com/package/serve-favicon
add CORS policy. Your server might sending preflight requests Use this npm package: https://www.npmjs.com/package/cors
No there is no default timeouts in nodejs or something like that.
Look for issue at your frontend part:
can be javascript fetch api with 'retry' option set
can be messed up RxJS operators chain which emits events implicitly and triggers another one REST request
can be entire page reload on timeout which leads to retrieve all neccessary data from backend
can be request interceptors (in axios, angular etc) which modify something and re-send
... many potential reasons, but not in backend (nodejs) for sure
Just make simple example and invoke your nodejs 'request_backend_data' endpoint with axois or xmlhttprequest - you will see that problem is not at backend part.
Try checking the api call with the code below, which includes follwing redirects. Add headers as needed (ie, 'Authorization': 'bearer dhqsdkhqd...etc'
var https = require('follow-redirects').https;
var fs = require('fs');
var options = {
'method': 'GET',
'hostname': 'foo.com',
'path': '/request_backend_data',
'headers': {
},
'maxRedirects': 20
};
var req = https.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function (chunk) {
var body = Buffer.concat(chunks);
console.log(body.toString());
});
res.on("error", function (error) {
console.error(error);
});
});
req.end();
Paste into a file called test.js then run with node test.js.
I have a strange situation regarding http server and piping request.
From my past experience, when piping the request object of a http server to a writable stream of some sort, it does not include the headers, just the payload.
Today however, I wrote some very simple code, and from some reason, I'm spending the past 2 hours trying to figure out why it writes the headers to the file (super confusing!)
Here's my code:
server = http.createServer((req, res) => {
f = '/tmp/dest'
console.log(`writing to ${f}`)
s = fs.createWriteStream(f)
req.pipe(s)
req.on('end', () => {
res.end("done")
})
})
server.listen(port)
I test this with the following curl command:
curl -XPOST -F 'data=#test.txt' localhost:8080
And this is what I'm getting when I'm reading /tmp/dest:
--------------------------993d19e02b7578ff
Content-Disposition: form-data; name="data"; filename="test.txt"
Content-Type: text/plain
hello - this is some text
--------------------------993d19e02b7578ff--
Why am I seeing the headers here? I expected it to only write the payload
I have a code I wrote about a year ago that streams directly to a file without the headers, I don't understand what's different, but this one did the trick:
imageRouter.post('/upload', async(req, res) => {
if(!req.is("image/*")) {
let errorMessage = `the /upload destination was hit, but content-type is ${req.get("Content-Type")}`;
console.log(errorMessage);
res.status(415).send(errorMessage);
return;
}
let imageType = req.get("Content-Type").split('/')[1];
let [ err, writeStream ] = await getWritableStream({ suffix: imageType });
if (err) {
console.log("error while trying to write", err);
return res.status(500).end();
}
let imageName = writeStream.getID();
req.on('end', () => {
req.unpipe();
writeStream.close();
res.json({
imageRelativeLink: `/images/${imageName}`,
imageFullLink: `${self_hostname}/images/${imageName}`
});
});
req.pipe(writeStream);
});
What's different? Why does my code from a year ago (last block) writes without the form-data/headers? The resulting file is only an image, without text, but this time (the first block) shows http headers in the resulting file
Instead of using pipe, try using on('data') and referring to req.data to pull off the contents. This will allow the http library to process the HTTP body format and handle the "headers" (really: form part descriptors) for you.
Node Streaming Consumer API
server = http.createServer((req, res) => {
f = '/tmp/dest'
console.log(`writing to ${f}`)
s = fs.createWriteStream(f)
req.on('data', chunk) => {
s.write(chunk);
}
req.on('end', () => {
s.close();
res.end("done")
})
})
server.listen(port)
As it turns out, I had a mistake in my understanding, and therefore made a mistake in my question.
What I thought were the headers, were actually http multipart specification. This is how curl uploads a file when used with this syntax.
What I actually needed was to change the way I test my code with curl to one of the following:
cat /path/to/test/file | curl -T - localhost:8080
# or
curl -T - localhost:8080 < /path/to/test/file
# or
curl -T /path-/to/test/file localhost:8080 < /path/to/test/file
Using the -T (or --upload-file) flag, curl uploads the file (or stdin) without wrapping it in an http form.
I am using request npm module in my app, to make to create a http client, as this.
var request = require('request');
And each time, I make a request to some server, I pass the options as below:
var options = {
url: "whateverurl...",
body: { some json data for POST ... }
}
request(options, cb(e, r, body) {
// handle response here...
})
This was working fine, until I started testing with high load, and I started getting errors indicating no address available (EADDRNOTAVAIL). It looks like I am running out of ephemeral ports, as there is no pooling or keep-alive enabled.
After that, I changed it to this:
var options = {
url: "whateverurl...",
body: { some json data for POST ... },
forever: true
}
request(options, cb(e, r, body) {
// handle response here...
})
(Note the option (forever:true)
I tried looking up request module's documentation about how to set keep-alive. According to the documentation and this stackoverflow thread, I am supposed to add {forever:true} to my options.
It didn't seem to work for me, because when I checked the tcpdump, the sever was still closing the connection. So, my question is:
Am I doing something wrong here?
Should I not be setting a global option to request module, while I am "require"ing it, instead of telling it to use {forever:true}, each time I make a http request? This is confusing to me.
I've been trying to stream binary data (PDF, images, other resources) directly from a request to a remote server but have had no luck so far. To be clear, I don't want to write the document to any filesystem. The client (browser) will make a request to my node process which will subsequently make a GET request to a remote server and directly stream that data back to the client.
var request = require('request');
app.get('/message/:id', function(req, res) {
// db call for specific id, etc.
var options = {
url: 'https://example.com/document.pdf',
encoding: null
};
// First try - unsuccessful
request(options).pipe(res);
// Second try - unsuccessful
request(options, function (err, response, body) {
var binaryData = body.toString('binary');
res.header('content-type', 'application/pdf');
res.send(binaryData);
});
});
Putting both data and binaryData in a console.log show that the proper data is there but the subsequent PDF that is downloaded is corrupt. I can't figure out why.
Wow, never mind. Found out Postman (Chrome App) was hijacking the request and response somehow. The // First Try example in my code excerpt works properly in browser.
PhantomJs's webserver does not support multipart requests, so I'm trying to send a single-part request from NodeJs.
Unfortunatly the nodejs example looks to be multipart. is there any way of doing this with NodeJs?
http://nodejs.org/api/http.html#http_http_request_options_callback
edit:
in the nodejs docs it mentions:
Sending a 'Content-length' header will disable the default chunked encoding.
but unfortunatly it's still multi-part, just not multi-multipart :P
edit2: for showing code, it's a bit hard to show a distilled example, but here goes:
node.js code (it's Typescript code):
```
//send our POST body (our clientRequest)
var postBody = "hello";
var options : __node_d_ts.IRequestOptions = {
host: host,
port: port,
path: "/",
method: "POST",
headers: {
"Content-Type": "application/json",
"Content-length": postBody.length
}
};
//logger.assert(false);
var clientRequest = http.request(options,(response: http.ServerResponse) => {
//callback stuff here
});
clientRequest.on("error", (err) => {
thisObj.abort("error", "error,request error", err);
});
//clientRequest.write();
clientRequest.end(postBody);
```
when i read the results from PhantomJS, the post/postRaw fields are null.
when I use a tool like the Chrome "Advanced REST Client" extension to send a POST body, phantomjs gets it no problem.
i don't have a network sniffer, but as described here, it says phantomjs doesnt work with multipart so I think that's a good guesss: How can I send POST data to a phantomjs script
EDIT3:
indeed, here's the request phantomjs gets from my chrome extension (valid post)
//cookie, userAgent, and Origin headers removed for brevity
{"headers":{"Accept":"*/*","Accept-Encoding":"gzip,deflate,sdch","Accept-Language":"en-US,en;q=0.8,ko;q=0.6","Connection":"keep-alive","Content-Length":"5","Content-Type":"application/json","DNT":"1","Host":"localhost:41338", "httpVersion":"1.1","method":"POST","post":"hello","url":"/"}
and here's the request phantomjs gets from the nodejs code i show above:
//full request, nothing omitted!
{"headers":{"Connection":"keep-alive","Content-Type":"application/json","Content-length":"5","Host":"10.0.10.15:41338"},"httpVersion":"1.1","method":"POST","url":"/"}