My code is as follows
var http = require('http');
var host=...
var postData=({
//some fun stuff
})
var postOptions ={
host: host,
path: '/api/dostuff',
method: 'POST',
headers:{
AppID:"some stuff",
Authorization: "OAuth token",
"Content-Type":"application/json"
},
};
var req = http.request(postOptions, function(res){
var data = '';
res.on('data', function (chunk) {
data += chunk;
});
res.on('end', function () {
//sanitize data stuff here
console.log("DATA HERE: "+ data);
return data;
});
});
req.write(JSON.stringify(postData));
req.end();
It's a basic HTTP post to a C# server. The important stuff is in the headers. I send the app ID (which is ~50 characters) and the OAuth token (which can be thousands of characters). Right now, the server isn't set up to do anything with the Authorization header. It doesn't even check if its there.
My problem is that when I populate the Authorization header (or any header) with a few random characters as a test, the post succeeds. When I tried it again with a full valid Authorization token (which, to reiterate, is very long) it fails. No matter which part of the header i fill, once it gets too full it returns an error. The error I receive is "Processing of the HTTP request resulted in an exception. Please see the HTTP response returned by the 'Response' property of this exception for details". I was somewhat certain this is a server issue, but when I tried running the exact same body and headers in Postman, I got a valid response.
Does anyone have any idea what is causing this?
There's a compiled constant that's defined to be 80k for Node HTTP headers. Are you running into that? I'd recommend seeing how big the header is with your OAuth token. It shouldn't exceed 80k though, and FWIW, even a kilobyte is huge for OAuth... But regardless... Try dumping the size of the headers (in bytes).
Related
I started to write some tests for my application and I have issues to read/get response from the server. I tried many things but nothing really worked, can someone help me please ?
// /api/checkCreds
exports.checkCreds = async function(req, res){
//validation
if(!await User.checkCreds(req.body.username, req.body.password)){
var result = {error: true, data: "Incorrect"}
res.sendStatus = 401;
return res.send(JSON.stringify(result));
}
If credentials sent to the server aren't matching, return a response with "Incorrect" message back to the user.
In the test I'm trying to get data from the server to check if properties are matching the expected output.
//test.js
it("We should fail with HTTP code 401 because incorrect data is passed (username='incorrect' password='incorrect')", function(done){
supertest(app)
.post('/api/checkCreds')
.send({username: 'incorrect', password: 'incorrect'})
.expect({error: true, data: "Incorrect"})
.expect(401, done);
});
When ran, test fails because expected properties are different from the response sent by the server, which is an empty object {}.
Any help is appreciated.
You may try changing your first expect to see if you can coax supertest into showing you the actual body that it's comparing to. For example, expect('')
If that doesn't work, there's a version of expect that accepts a function. In that function, you should be able to print out what you are getting in the response body, ie. console.log(res).
It may be that there's some confusion with the JSON return type-- I haven't used that directly. You could try expecting JSON.
Finally, there's a strange paragraph in the documentation that I don't think applies, but I thought I'd mention:
One thing to note with the above statement is that superagent now sends any HTTP error (anything other than a 2XX response code) to the callback as the first argument if you do not add a status code expect (i.e. .expect(302)).
While trying to fix my issue, I noticed that in the HTTP response, Content-Type header was set to text/plain and my server was returning JSON, so that probably was the thing that confused supertest.
I think that res.send() sets the header to text/plain by default and I had to manually set the header value to application/json by using res.type('json'). At that point I was able to read the response body without an issue.
I also learned that res.json() sets the Content-Type header to application/json by default, so you don't need to do it manually like with res.send().
Working code:
// /api/checkCreds
if(!await User.checkCreds(req.body.username, req.body.password)){
var result = {error: true, data: "Incorrect"}
return res.status(401).json(result);
}
//test.js
it("We should fail with HTTP code 401 because incorrect data is passed (username='incorrect' password='incorrect')", function(done){
supertest(app)
.post('/api/checkCreds')
.set('Content-type', 'application/json')
.send({username: 'incorrect', password: 'incorrect'})
.expect(401)
.expect(function(res){
console.log(res.body);
})
.end(done);
});
Feel free to correct me if I stated something that isn't quite right.
I have a node.js GET API endpoint that calls some backend services to get data.
app.get('/request_backend_data', function(req, res) {
---------------------
}
When there is a delay getting a response back from the backend services, this endpoint(request_backend_data) is getting triggered exactly after 2 minutes. I have checked my application code, but there is no retry logic written anywhere when there is a delay.
Does node.js API endpoint gets called twice in any case(like delay or timeout)?
There might be a few reasons:
some chrome extensions might cause bugs. Those chrome extensions have been causing a lot of issues recently. run your app on a different browser. If there is no issue, that means it is chrome-specific problem.
express might be making requests for favicon.ico. In order to prevent this, use this module : https://www.npmjs.com/package/serve-favicon
add CORS policy. Your server might sending preflight requests Use this npm package: https://www.npmjs.com/package/cors
No there is no default timeouts in nodejs or something like that.
Look for issue at your frontend part:
can be javascript fetch api with 'retry' option set
can be messed up RxJS operators chain which emits events implicitly and triggers another one REST request
can be entire page reload on timeout which leads to retrieve all neccessary data from backend
can be request interceptors (in axios, angular etc) which modify something and re-send
... many potential reasons, but not in backend (nodejs) for sure
Just make simple example and invoke your nodejs 'request_backend_data' endpoint with axois or xmlhttprequest - you will see that problem is not at backend part.
Try checking the api call with the code below, which includes follwing redirects. Add headers as needed (ie, 'Authorization': 'bearer dhqsdkhqd...etc'
var https = require('follow-redirects').https;
var fs = require('fs');
var options = {
'method': 'GET',
'hostname': 'foo.com',
'path': '/request_backend_data',
'headers': {
},
'maxRedirects': 20
};
var req = https.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function (chunk) {
var body = Buffer.concat(chunks);
console.log(body.toString());
});
res.on("error", function (error) {
console.error(error);
});
});
req.end();
Paste into a file called test.js then run with node test.js.
Beginner question, I am using the fetch API in one of my Cloudflare Service Worker to send request to a third party; Everything works fine that I am able to send out the request and receive response; But now I have a need to log everything I sent and received, which I am not sure how exactly can I do so;
The main requirement:
Able to log the whole request header being sent
The problem is that the request header that I specified is not the full header being sent, like in the following example I only specified the content-type, but in real practice it will send with more properties added by the application itself, e.g. accept, x-forwarded-for, content-length, etc.
Request:
var toThirdPartyReqMethod = 'POST';
var toThirdPartyReqHeader = {'content-type': 'application/json'};
var toThirdPartyReqSetting = {
method: toThirdPartyReqMethod,
headers: toThirdPartyReqHeader,
body: toThirdPartyReqBodyArrayBuffer
};
var toThirdPartyUrl = "https://somewhere.com";
var fromThirdPartyResponse = await fetch(toThirdPartyUrl, toThirdPartyReqSetting)
var fromThirdPartyResponseBodyArrayBuffer = await fromThirdPartyResponse.arrayBuffer();
I am able to use fromThirdPartyResponse to digest the response; but I have no idea how to record the exact request I sent out.
I tried to do it as follows:
var oriReqHeaderContentType = fromThirdPartyResponse.request.headers.get("content-type");
But obviously it is not the right answer and returned with the following error:
TypeError: Cannot read property 'headers' of undefined
Anyone gets some idea of how to log the request being sent when using fetch API?
Sorry for asking beginner questions :(
You can access request headers like this
const contentType = req.get("Content-Type");
I am trying to do a GET request from node.js. This request is to a REST server which will access Hbase and return the data. The GET request contains all necessary Hbase info (table, key, column-family etc.) in it. Following is the code.
var http = require('http');
var url = {
host: '127.0.0.1',
port: 8000,
path: '/table-name/key/column-family',
headers: {
'Content-Type': 'application/octet-stream'
},
};
http.get(url, function(resp){
console.log("Status: " + resp.statusCode);
console.log("Header: " + JSON.stringify(resp.headers));
resp.setEncoding('utf8');
var completeResponse = '';
resp.on('data', function (chunk) {
completeResponse += chunk;
});
resp.on('end', function(chunk) {
console.log(completeResponse);
});
});
My problem is that the response I get is not always an octet-stream as requested. Most of the time data is in valid format with a header like the following.
{"content-length":"454","x-timestamp":"1395469504346","content-type":"application/octet-stream"}
But, say 1 out of 10 times the response is an XML string with a header like following.
{"content-type":"text/xml","content-length":"793"}
The status code is 200 in both cases and I am always requesting for an existing key. This behavior is seemingly random and not caused by any particular key.
How do I ensure that the response is always an octet-stream and not XML / JSON?
As stated in the comments, you need to set the Accept header to specify the content type you expect (accept) in response from the server. Note that you may accept more than one type of response.
The Content-Type header specify the type of what's in the body of the message. It can be set by your client in case of POST/PATCH request, or by the server in its response. On the receiving side, it is used to know how to handle the body content.
For more detail, you can refer to the comprehensive MDN content negotiation documentation
I am trying to get data from the Bing search API, and since the existing libraries seem to be based on old discontinued APIs I though I'd try myself using the request library, which appears to be the most common library for this.
My code looks like
var SKEY = "myKey...." ,
ServiceRootURL = 'https://api.datamarket.azure.com/Bing/Search/v1/Composite';
function getBingData(query, top, skip, cb) {
var params = {
Sources: "'web'",
Query: "'"+query+"'",
'$format': "JSON",
'$top': top, '$skip': skip
},
req = request.get(ServiceRootURL).auth(SKEY, SKEY, false).qs(params);
request(req, cb)
}
getBingData("bookline.hu", 50, 0, someCallbackWhichParsesTheBody)
Bing returns some JSON and I can work with it sometimes but if the response body contains a large amount of non ASCII characters JSON.parse complains that the string is malformed. I tried switching to an ATOM content type, but there was no difference, the xml was invalid. Inspecting the response body as available in the request() callback actually shows bad code.
So I tried the same request with some python code, and that appears to work fine all the time. For reference:
r = requests.get(
'https://api.datamarket.azure.com/Bing/Search/v1/Composite?Sources=%27web%27&Query=%27sexy%20cosplay%20girls%27&$format=json',
auth=HTTPBasicAuth(SKEY,SKEY))
stuffWithResponse(r.json())
I am unable to reproduce the problem with smaller responses (e.g. limiting the number of results) and unable to identify a single result which causes the issue (by stepping up the offset).
My impression is that the response gets read in chunks, transcoded somehow and reassembled back in a bad way, which means the json/atom data becomes invalid if some multibyte character gets split, which happens on larger responses but not small ones.
Being new to node, I am not sure if there is something I should be doing (setting the encoding somewhere? Bing returns UTF-8, so this doesn't seem needed).
Anyone has any idea of what is going on?
FWIW, I'm on OSX 10.8, node is v0.8.20 installed via macports, request is v2.14.0 installed via npm.
i'm not sure about the request library but the default nodejs one works well for me. It also seems a lot easier to read than your library and does indeed come back in chunks.
http://nodejs.org/api/http.html#http_http_request_options_callback
or for https (like your req) http://nodejs.org/api/https.html#https_https_request_options_callback (the same really though)
For the options a little tip: use url parse
var url = require('url');
var params = '{}'
var dataURL = url.parse(ServiceRootURL);
var post_options = {
hostname: dataURL.hostname,
port: dataURL.port || 80,
path: dataURL.path,
method: 'GET',
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': params.length
}
};
obviously params needs to be the data you want to send
I think your request authentication is incorrect. Authentication has to be provided before request.get.
See the documentation for request HTTP authentication. qs is an object that has to be passed to request options just like url and auth.
Also you are using same req for second request. You should know that request.get returns a stream for the GET of url given. Your next request using req will go wrong.
If you only need HTTPBasicAuth, this should also work
//remove req = request.get and subsequent request
request.get('http://some.server.com/', {
'auth': {
'user': 'username',
'pass': 'password',
'sendImmediately': false
}
},function (error, response, body) {
});
The callback argument gets 3 arguments. The first is an error when applicable (usually from the http.Client option not the http.ClientRequest object). The second is an http.ClientResponse object. The third is the response body String or Buffer.
The second object is the response stream. To use it you must use events 'data', 'end', 'error' and 'close'.
Be sure to use the arguments correctly.
You have to pass the option {json:true} to enable json parsing of the response