In NodeJS and express I am handling a batch request to a OData API. Sometimes the OData service returns invalid results in the response which is denoted by a status 500 (could be more than one, since this is a OData batch response), even though the request header status code returns 200 because the connection to the API itself was successful.
I am new to NodeJS and am not sure how / the best way to detect these status 500 in the response body so that I can retry the request (reason is these 500 errors are usually timeouts and on the next retry goes away).
Do I need a package to be able to retrieve or intercept res.body? I don't see it available in express.
Request and response example
Request header
Request URL: https://odata/$batch
Request Method: POST
Status Code: 202 Accepted
Remote Address: xxxx
Referrer Policy: no-referrer-when-downgrade
Response
--batch_1564714041997_0
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 500 OK
Content-Type: text/plain;charset=utf-8
Internal server error
--batch_1564714041997_0
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 500 OK
Content-Type: text/plain;charset=utf-8
Internal server error
--batch_1564714041997_0--
I would use a regular expression and a capturing group for the HTTP status code.
var regExForHTTPStatusCodeDetection = RegExp('HTTP\/1\.\\d (\\d{3})','g');
var requestNumber = 1;
while ((regExMatches = regExForHTTPStatusCodeDetection.exec(res.body)) !== null) {
// Get status code from first capturing group
var httpStatusCode = regExMatches[1];
// Output for above example:
// "Request 1 has returned HTTP 500."
// "Request 2 has returned HTTP 500."
console.log(`Request ${requestNumber} has returned HTTP ${}.`);
requestNumber++;
}
Related
I have a GET route in express that should return a binary png image stored in mongodb. However, when I enter the url into chrome to test, the image is downloaded but the request never completes. From the Network tab in Chrome DevTools the request is just stuck in the 'pending' state. I'm only getting this problem with binary data it seems. I have plenty of other json GET requests that work just fine with send().
I am using the send() function like this:
exports.getProjectPng = (req, res) => {
Project.findById(req.params.projectId).select('project.png')
.then(project => {
res.send(project.png.buffer);
});
If I simply replace send() with end() the request completes as expected. Also, perhaps significantly, the png image is actually rendered within the browser rather than downloading as a file.
So why does end() work but send() doesn't?
If you point curl at an express server and see what the response looks like for both methods it is quite interesting. The main difference is what when we call send, the Content-Type header is populated, which is consistent with the Express docs:
When the parameter is a Buffer object, the method sets the Content-Type response header field to “application/octet-stream”, unless previously defined
It's worthwhile noting that res.send() actually calls res.end() internally at the end of the call, so the different behaviour is likely down to something that res.send does in addition to res.end.
It might be worth populating the Content-Type Header in your example to "image/png" before sending.
e.g.
res.set('Content-Type', 'image/png');
For .end():
* Connected to localhost (::1) port 8081 (#0)
> GET /downloadpng_end HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Date: Mon, 25 Jun 2018 13:14:58 GMT
< Connection: keep-alive
< Content-Length: 69040
<
And for send():
* Connected to localhost (::1) port 8081 (#0)
> GET /downloadpng_send HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/octet-stream
< Content-Length: 69040
< ETag: W/"10db0-KwFSGG5Ib/DQNZChAbluTiKSP0o"
< Date: Mon, 25 Jun 2018 13:15:25 GMT
< Connection: keep-alive
<
Nodejs does not handle straight binary data very well.So , thats what buffer is used to handle binary data.
End() Method
end − This event is fired when there is no more data to read. while send has no guarantee whether its completed or not. You can read more about Buffer on offical docs here
I am making a Node.js/Express backend for a mobile app that makes requests to an API using the RESTful approach, where I can use the JSON data returned by this API for users to use in my mobile app.
My confusion lies in the differences that go on 'under to hood' and what automatically get handled when using specific headers that are intended for browsers and a server VS when making curl request through the terminal, that you'll run more than once, sending and responding with different headers
Headers/The Rules That Apply I Make Requests
I must use the conditional GETs convention where:
1) All responses return an HTTP Cache-Control header. It’s content indicates how long a cached response can be used to reduce unnecessary API requests.
2) In addition to that, each response returns an HTTP ETag header. It’s content is to be used in subsequent requests to the same resource in an HTTP If-None-Match header. The API will then return a status code 304 Not Modified if the cached information is still valid.
Clients accessing the This API MUST use this techniques, also known as conditional GET.
Using Curl Requests
Making Initial Request:
A) Returns the data I requested. B) Returns an Etag to use for future requests C) Returns a Cache-Control header to know how long the data is cache-able, like so:
First Request:
$ curl -v "https://api.example.com/data/3" -X GET \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-u token:secret
Responce:
< HTTP/1.1 200 OK
< Date: Wed, 01 Sep 2017 22:22:22 GMT
< ETag: "xxxxxxxxxxxxxxxxxxxxxxx" <-----Got Etag
< Last-Modified: Wed, 09 Sep 2015 11:11:11 GMT
< Content-Type: application/json; charset=utf-8
< Cache-Control: max-age=3600, private <-----and Cache-Control is set
{"data":{"id":1, "...": "..."}} to 3600 seconds (1 hour).
Here I understand the Cache-Control header specifies that the data I just got is good for 3,600 seconds, i.e. 1 hour. So I can use the returned data-information for the next hour without having to request the data from the API again. After that time period, another request can be made to the API. This time I include the returned xxx..... ETag value I got one hour ago (because thats how long the Cache-Control was set for), and use it in the If-None-Match header if I make another request like so:
Making Another Curl Request:
Future Request:
$ curl -v "https://api.example.com/data/3" -X GET \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "If-None-Match: xxxxxxxxxxxxxxxxxxxxxxx" <--Added here
-u token:secret
Future Responce:
< HTTP/1.1 304 Not Modified <--Got 304 because the
< Date: Wed, 23 Sep 2015 20:24:20 GMT
< ETag: "xxxxxxxxxxxxxxxxxxxxxxx" <-- `Etag` returned is the same
< Last-Modified: Wed, 09 Sep 2015 11:45:39 GMT (so I know it hasn't changed)
< Cache-Control: max-age=3600, private
< Connection: close
So this leads to me having a few questions:
Making None-Curl Requests (via a Node.js server):
I am using the pretty straight forward Node.js request module to make my request like so:
var request = require('request');
var options = {
url: 'https://api.example.com/data/3',
headers: { <------ Setting up Headers
'Content-Type': 'application/json', Note: no "If-None-Match"
'Accept': 'application/json'
'Authorization: Basic QQWERQWERWQER='
}
};
function callback(error, response, body) {
if (!error && response.statusCode == 200) {
var data = body <------ Got 200 response and loaded
} return data into data varible
}
request(options, callback);
So how is my Express server communicating with this API when making requests?
So after I make my first request on my Express server, it (my server) can then determine if it needs to fetch that content from the network or from cache based on the Cache-Control header and ETag. So how is this being done?
Do I need to programmatically/write code to do that every hour? So will I have to write code to grab the etag like this
if (!error && response.statusCode == 200) {
var data = body;
var etag = response.headers.etag <------ Grab this
}
and create a second set of headers like this
headers: {
'Content-Type': 'application/json',
'Accept': 'application/json'
'Authorization: Basic QQWERQWERWQER='
'if-none-match': etag <------ Adding it here
}
like I had to do when did when I did my second curl? (I did this and it will hold up on every requests not returning any status code, if this is how its done could it just be I did something wrong on my end?)
or does the initial response where Cache-Control headers set to max-age=3600, private and the Etag (set to xxxxxxxx in my example) automatically set the cache on my Express server and tells it to act accordingly with the it wants this resource where it check it again in an hour without having to do anything to tell it to?
Right now whenever I request a resource in the mobile app that will need a something (lets say the resource in my example https://api.example.com/data/3 ) from this API it triggers my Express server to fetch it on mobile app users behalf and return it. However when I request the same resource more than once, the API returns a 200 response every time. Where as it should be responding with a 304 for every request after the first. So what's going on here/ what am I doing/ understanding wrong? why would I not get 304 response?
When I am trying to update the a property (AvaliableQuantity) on an object and then update it via ODATA (using the oData v4 Client Code Generator). The problem is when I send the request (Fiddler output below) I get an error saying that "Only AvailableQuantity may be updated." Is there a way of specifying which of the properties can be updated (maybe via attributes?)
PATCH https://*****************/v1/DCQuantities(ProductID=33578709,DistributionCenterID=17) HTTP/1.1
OData-Version: 4.0
OData-MaxVersion: 4.0
Content-Type: application/json;odata.metadata=minimal
Accept: application/json;odata.metadata=minimal
Accept-Charset: UTF-8
User-Agent: Microsoft ADO.NET Data Services
Authorization: Bearer ************************
Host: **************
Content-Length: 153
Expect: 100-continue
{"#odata.type":"#ChannelAdvisor.RestApi.V1.Models.DCQuantity","AvailableQuantity":30,"DistributionCenterID":17,"ProductID":33578709,"ProfileID":22001149}
Below is the raw response from Fiddler
HTTP/1.1 400 Bad Request
Content-Length: 88
Content-Type: application/json; odata.metadata=minimal; charset=utf-8
Server: Microsoft-IIS/7.5
OData-Version: 4.0
X-ServerName: **********
Date: Wed, 16 Dec 2015 15:50:52 GMT
{
"error":{
"code":"","message":"Only AvailableQuantity may be updated."
}
}
I have been able to update properties on other objects so I know that the authentication (OAUTH2) and the basic classes are working.
Does anybody have any ideas on what I could try please?
Many thanks
Jonathan
OK, I have realised what I was doing wrong from this post (http://blogs.msdn.com/b/odatateam/archive/2014/04/10/client-property-tracking-for-patch.aspx). I was using code like this to get the object before changing it:
var product = target.Products.Expand("DCQuantities").Where(p => p.Sku == "JRSTEST1").First();
product.First().DCQuantities[0].AvailableQuantity = 42;
target.UpdateObject(product.DCQuantities[0]);
var result = target.SaveChanges();
What I needed to do was assign the result of the query to a DataServiceCollection<>() class which would then look after tracking what was actually changed and manage the whole update for me.
var product = new DataServiceCollection<Product>(target.Products.Expand("DCQuantities").Where(p => p.Sku == "JRSTEST1"));
product.First().DCQuantities[0].AvailableQuantity = 42;
var result = target.SaveChanges();
I have a simple API upload, it is used to accept upload file from client.
var flg=true;
app.post('/test', function(req, res){
flg=!flg;
var returnJson='{';
if(flg){
req.form.on('part', function (part) {
if(part){
part.resume();
}
returnJson=returnJson+',\"status\":\"0\"}';
res.send(returnJson);
});
}else{
console.log('close');
returnJson=returnJson+',\"status\":\"1\"}';
res.header('Connection', 'close');
res.send(413, returnJson);
}
});
I'd like to test this API with Jmeter. "status":"0" means success. "status":"1" means failure. I write Jmeter script like this:
http://i.imgur.com/vEUJKc8.jpg
Jmeter only displays all the samplers which response contains "status":"0". it seems Jmeter exclude failure sampler response which comes from else section.
http://imgur.com/bkFSpK2
How can I see all samplers which includes all success and failure samplers in Jmeter?
successful Sampler result is :
Thread Name: API 1-1
Sample Start: 2013-12-18 11:46:08 PST
Load time: 7
Latency: 6
Size in bytes: 178
Headers size in bytes: 163
Body size in bytes: 15
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 15
Date: Wed, 18 Dec 2013 19:46:08 GMT
Connection: keep-alive
HTTPSampleResult fields:
ContentType: text/html; charset=utf-8
DataEncoding: utf-8
Any suggestion?
I don't like this stanza:
ContentType: text/html;
Correct ContentType for JSON will be application/json
You can try using HTTP Header Manager to set Content-Type header of your request to be application/json and see what happens.
Also there is a JSON plugin which provides JSON Path Extractor and JSON Path Assertion (Select “Extras with libs set” from the download list).
I have built a API using node.js and express.
But i need to be able to proxy some requests on a specific route to a external server and show the response from the external server to the clint doing the request.
But i also need to forward the basic auth that the client is send along with the request.
I have tried using the request module like:
app.get('/c/users/', function(req,res) {
//modify the url in any way you want
var newurl = 'https://www.external.com'
request(newurl).pipe(res),
})
But it seems to not send the basic auth header because i get "403 Forbidden" back form the external server(www.external.com)
The request im making is looking like:
GET http://example.se:4000/c/users/ HTTP/1.1
Accept-Encoding: gzip,deflate
X-version: 1
Authorization: Basic bmR4ZHpzNWZweWFpdjdxfG1vcmV1c2*******=
Accept: application/json
Host: example.se:4000
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
And it works if i do the exact same request but against www.external.com directly so there is some issue when doing the proxy in node.
The request module is totally unaware of anything that you don't pass explicitly to it. To set the request headers and copy the response headers as well do the following:
// copy headers, except host header
var headers = {}
for (var key in req.headers) {
if (req.headers.hasOwnProperty(key)) {
headers[key] = req.get(key)
}
}
headers['host'] = 'final-host'
var newurl = 'http://final-host/...'
request.get({url:newurl, headers: headers }, function (error, response, body) {
// debug response headers
console.log(response.headers)
// debug response body
console.log(body)
// copy response headers
for (var key in response.headers) {
if (response.headers.hasOwnProperty(key)) {
res.setHeader(key, response.headers[key])
}
}
res.send(response.statusCode, body)
})
Try explicitly passing in the auth details like so
request(newurl).auth(username, password).pipe(res);
https://github.com/mikeal/request#http-authentication