I'm using nodejs on windows 7. I'm experiencing strange behavior. The same https request that always gives the same response is most of the time not received by node whereas it is always received by a web browser.
The code is quite simple:
var url1 = 'https:<my url>'
require('request').get(url1).pipe(process.stdout)
The response is actually a chunked response, which is a big json object.
If I try this request on a browser, I always receive the anwser.
If I use the node request module (I also tried with https), the body is most of the time empty. But sometimes it is returned.
I receive status 200 and response headers are:
{ 'content-type': 'application/json',
connection: 'close',
'transfer-encoding': 'chunked',
date: 'Wed, 27 Jan 2016 21:01:59 GMT',
server: 'My Server' }
Any idea?
I found 2 solutions to fix this issue:
1) Set header 'Connection': 'keep-alive'
var opts = { url: 'my url', headers :{ 'Connection':'keep-alive'} }
request(opts).pipe(..)
2) set to true KeepAlive option in https.globalAgent
https.globalAgent.keepAlive=true
Related
I would like to use value of accept-language header for detecting language in node.js server created with using Express.
However when I try to get with headers:
console.dir(req.headers)
there is no accept-language field, other headers are present. But I can see 'Accept-Language' in chrome tab.
I am making request with JS fetch with this configuration:
fetch('/path/somepath', {
method: 'POST',
credentials: 'include',
headers: {
'Content-Type': 'application/json;charset=utf-8'
},
body: JSON.stringify(input)
})
What could be possible reasons of this behaviour and is something what I can do? I do not want to set this header explicitly in fetch because I would like it to create automatically on basis of browser settings.
I need to load page from this url: https://medalist-a805c.web.app/HeardSuggest_Medalist.html which actually loads a page from a express server. I'm able to use cors, the express middleware. But the problem is, even though i need to load this url: https://www.alwaysheard.com/embed/Medalist/brand/$2a$10$Z9ib22YZyRWWZz/Vlzju3u9eZCZM.a8oUYorNPsKGzLPKu6vM696K?uname=embed&orgin=https%3A%2F%2Fwww.alwaysheard.com eventually.
The weird problem i'm facing is it always returns the homepage rather than the url i am trying to get which is: https://www.alwaysheard.com/embed/Medalist/brand/$2a$10$Z9ib22YZyRWWZz/Vlzju3u9eZCZM.a8oUYorNPsKGzLPKu6vM696K?uname=embed&orgin=https%3A%2F%2Fwww.alwaysheard.com
Any help will be highly appreciated. Many Thanks.
This is for a expressjs server.
This is my frotend calling code:
$.ajax({
url: _url,
data: { uname: "embed", orgin: _origin },
// dataType: 'json',
crossDomain: true,
type: "GET",
mode: 'no-cors',
headers: {
'mode': 'no-cors',
// "X-TOKEN": 'xxxxx',
// 'x-Trigger': 'CORS',
// 'Access-Control-Allow-Headers': 'x-requested-with',
// "Access-Control-Allow-Origin": "*",
// 'Authorization': 'Bearer fadsfasf asdfasdf'
},
success: function (data) {
console.log("data: ", data);
$("body").html(data);
}
});
And in backend i'm using app.use(cors());
Your URL is first returning a 302 redirect to the home page and then the $.ajax() system (actually the XMLHttpRequest system underneath it) is following that redirect automatically.
You can see a discussion of that topic here:
How to prevent ajax requests to follow redirects using jQuery
In newer browsers, you can use the fetch() interface instead and it contains a redirect option that allows you to tell it not to follow redirects.
FYI, it's unclear what you're really trying to accomplish here. There is no "page" to retrieve from this URL. That returns a 302 status and a few headers, no page.
In case this isn't clear to you, your long url returns this response:
HTTP/1.1 302 Found
Date: Fri, 18 Oct 2019 17:04:05 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 23
Connection: keep-alive
Server: nginx/1.14.1
Access-Control-Allow-Origin: *
Set-Cookie: _csrf=j6JfsEb_JbmSJlvHDoyhgatS; Path=/
Set-Cookie: connect.sid=s%3A8C1PVNBcm5395dwmPJ4Xbl7BNASxPH4W.WFZ6uz3NklxaObnWvZaZXI%2BoH5VdM9dnOi2VCTw3J%2BI; Path=/; Expires=Mon, 15 Oct 2029 17:04:05 GMT; HttpOnly
Surrogate-Control: no-store
Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate
Pragma: no-cache
Expires: 0
Location: /
Vary: Accept
This content was retrieved with curl -l -v yourlongurl.
I create a http server with express.
Below is the server code:
router.get('/', function(req, res, next) {
// req.socket.setTimeout(Infinity);
res.writeHead(200, {
'Content-Type': 'text/plain; charset=utf-8', // <- Important headers
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
});
res.write('hell');
res.write('world');
res.end();
// res.write('\n\n');
// response = res;
});
when I use netcat to GET the url.
the output is like this
GET /sse HTTP/1.1
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/plain; charset=utf-8
Cache-Control: no-cache
Expires: 0
Connection: keep-alive
Keep-Alive: timeout=5, max=97
Date: Fri, 30 Jun 2017 11:50:00 GMT
Transfer-Encoding: chunked
4
hell
5
world
0
My question is why there are always a number before every res.write()? And the number seems is then length of the output of res.write.
How can I remove the number?
This is how the chunked encoding works. You don't have to declare up front how many bytes are you going to send but instead every chunk is prepended with the number of bytes that it has. For example: 4 for "hell", 5 for "world" and then finally 0 for "no more bytes to send".
You can see that you have the chunked encoding header present:
Transfer-Encoding: chunked
Now, to answer your question directly, to remove the numbers you'd have to switch off the chunked encoding. To do that you'd have to set the Content-length header to the number of bytes that you are going to send in the body of the response.
For example:
app.get('/', function(req, res, next) {
res.writeHead(200, {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Content-length': 9, // <<<--- NOTE HERE
});
res.write('hell');
res.write('world');
res.end();
});
(Of course in the real code you'd have to either calculate the length of the response or build up a string or buffer and get its length just before you set the Content-length header.)
But note that if you use curl or any other HTTP client then it will "remove" the numbers for you. It's just that with netcat you accidentally saw the underlying implementation detail of the chunked encoding but all of the real HTTP clients can handle that just fine.
Normally in HTTP you declare the length of the entire response in the headers and then send the body - in one piece, in multiple chunks, whatever - but it has to have the same length as what you declared. It means that you cannot start sending data before you know the length of everything that you want to send. With chunked encoding it's enough to declare the length of each chunk that you send but you don't have to know the length of the entire response - which could even be infinite, it's up to you. It lets you start sending as soon as you have anything to send which is very useful.
I am using nodejs and the REST API to interact with bigquery. I am using the google-oauth-jwt module for JWT signing.
I granted a service account write permission. So far I can list projects, list datasets, create a table and delete a table. But when it comes to upload a file via multipart POST, I ran into two problems:
gzipped json file doesn't work, I get an error saying "end boundary missing"
when I use uncompressed json file, I get a 401 unauthorized error
I don't think this is related to my machine's time being out of sync since other REST api calls worked as expected.
var url = 'https://www.googleapis.com/upload/bigquery/v2/projects/' + projectId + '/jobs';
var request = googleOauthJWT.requestWithJWT();
var jobResource = {
jobReference: {
projectId: projectId,
jobId: jobId
},
configuration: {
load: {
sourceFormat: 'NEWLINE_DELIMITED_JSON',
destinationTable: {
projectId: projectId,
datasetId: datasetId,
tableId: tableId
},
createDisposition: '',
writeDisposition: ''
}
}
};
request(
{
url: url,
method: 'POST',
jwt: jwtParams,
headers: {
'Content-Type': 'multipart/related'
},
qs: {
uploadType: 'multipart'
},
multipart: [
{
'Content-Type':'application/json; charset=UTF-8',
body: JSON.stringify(jobResource)
},
{
'Content-Type':'application/octet-stream',
body: fileBuffer.toString()
}
]
},
function(err, response, body) {
console.log(JSON.parse(body).selfLink);
}
);
Can anyone shine some light on this?
P.S. the documentation on bigquery REST api is not up to date on many things, wish the google guys can keep it updated
Update 1:
Here is the full HTTP request:
POST /upload/bigquery/v2/projects/239525534299/jobs?uploadType=multipart HTTP/1.1
content-type: multipart/related; boundary=71e00bd1-1c17-4892-8784-2facc6998699
authorization: Bearer ya29.AHES6ZRYyfSUpQz7xt-xwEgUfelmCvwi0RL3ztHDwC4vnBI
host: www.googleapis.com
content-length: 876
Connection: keep-alive
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/json
{"jobReference":{"projectId":"239525534299","jobId":"test-upload-2013-08-07_2300"},"configuration":{"load":{"sourceFormat":"NEWLINE_DELIMITED_JSON","destinationTable":{"projectId":"239525534299","datasetId":"performance","tableId":"test_table"},"createDisposition":"CREATE_NEVER","writeDisposition":"WRITE_APPEND"}}}
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/octet-stream
{"practiceId":2,"fanCount":5,"mvp":"Hello"}
{"practiceId":3,"fanCount":33,"mvp":"Hello"}
{"practiceId":4,"fanCount":71,"mvp":"Hello"}
{"practiceId":5,"fanCount":93,"mvp":"Hello"}
{"practiceId":6,"fanCount":92,"mvp":"Hello"}
{"practiceId":7,"fanCount":74,"mvp":"Hello"}
{"practiceId":8,"fanCount":100,"mvp":"Hello"}
{"practiceId":9,"fanCount":27,"mvp":"Hello"}
--71e00bd1-1c17-4892-8784-2facc6998699--
You are most likely sending duplicate content-type headers to the Google API.
I don't have the capability to effortlessly make a request to Google BigQuery to test, but I'd start with removing the headers property of your options object to request().
Remove this:
headers: {
'Content-Type': 'multipart/related'
},
The Node.js request module automatically detects that you have passed in a multipart array, and it adds the appropriate content-type header. If you provide your own content-type header, you most likely end up with a "duplicate" one, which does not contain the multipart boundary.
If you modify your code slightly to print out the actual headers sent:
var req = request({...}, function(..) {...});
console.log(req.headers);
You should see something like this for your original code above (I'm using the Node REPL):
> req.headers
{ 'Content-Type': 'multipart/related',
'content-type': 'multipart/related; boundary=af5ed508-5655-48e4-b43c-ae5be91b5ae9',
'content-length': 271 }
And the following if you remove the explicit headers option:
> req.headers
{ 'content-type': 'multipart/related; boundary=49d2371f-1baf-4526-b140-0d4d3f80bb75',
'content-length': 271 }
Some servers don't deal well with multiple headers having the same name. Hopefully this solves the end boundary missing error from the API!
I figured this out myself. This is one of those silly mistakes that would have you stuck for the whole day and at the end when you found the solution you would really knock on your own head.
I got the 401 by typing the selfLink URL in the browser. Of course it's not authorized.
I am trying to properly handle Accept headers in RESTful API in node.js/restify by using WrongAcceptError as follows.
var restify = require('restify')
; server = restify.createServer()
// Write some content as JSON together with appropriate HTTP headers.
function respond(status,response,contentType,content)
{ var json = JSON.stringify(content)
; response.writeHead(status,
{ 'Content-Type': contentType
, 'Content-Encoding': 'UTF-8'
, 'Content-Length': Buffer.byteLength(json,'utf-8')
})
; response.write(json)
; response.end()
}
server.get('/api',function(request,response,next)
{ var contentType = "application/vnd.me.org.api+json"
; var properContentType = request.accepts(contentType)
; if (properContentType!=contentType)
{ return next(new restify.WrongAcceptError("Only provides "+contentType)) }
respond(200,response,contentType,
{ "uri": "http://me.org/api"
, "users": "/users"
, "teams": "/teams"
})
; return next()
});
server.listen(8080, function(){});
which works fine if the client provides the right Accept header, or no header as seen here:
$ curl -is http://localhost:8080/api
HTTP/1.1 200 OK
Content-Type: application/vnd.me.org.api+json
Content-Encoding: UTF-8
Content-Length: 61
Date: Tue, 02 Apr 2013 10:19:45 GMT
Connection: keep-alive
{"uri":"http://me.org/api","users":"/users","teams":"/teams"}
The problem is that if the client do indeed provide a wrong Accept header, the server will not send the error message:
$ curl -is http://localhost:8080/api -H 'Accept: application/vnd.me.org.users+json'
HTTP/1.1 500 Internal Server Error
Date: Tue, 02 Apr 2013 10:27:23 GMT
Connection: keep-alive
Transfer-Encoding: chunked
because the client is not assumed to understand the error message, which is in JSON, as
seen by this:
$ curl -is http://localhost:8080/api -H 'Accept: application/json'
HTTP/1.1 406 Not Acceptable
Content-Type: application/json
Content-Length: 80
Date: Tue, 02 Apr 2013 10:30:28 GMT
Connection: keep-alive
{"code":"WrongAccept","message":"Only provides application/vnd.me.org.api+json"}
My question is therefore, how do I force restify to send back the right error status code and body, or am I doing things wrong?
The problem is actually that you're returning a JSON object with a content-type (application/vnd.me.org.api+json) that Restify doesn't know (and therefore, creates an error no formatter found).
You need to tell Restify how your responses should be formatted:
server = restify.createServer({
formatters : {
'*/*' : function(req, res, body) { // 'catch-all' formatter
if (body instanceof Error) { // see text
body = JSON.stringify({
code : body.body.code,
message : body.body.message
});
};
return body;
}
}
});
The body instanceof Error is also required, because it has to be converted to JSON before it can be sent back to the client.
The */* construction creates a 'catch-all' formatter, which is used for all mime-types that Restify can't handle itself (that list is application/javascript, application/json, text/plain and application/octet-stream). I can imagine that for certain cases the catch-all formatter could pose issues, but that depends on your exact setup.