I am experiencing occasional retries for a POST request, when there is no response from server due to timeout. All modern browsers have retry logic for idempotent requests (GET, HEAD, etc) but I am unable to reason out why it happens for a POST request.
I am testing this case using a simple node.js server with 3 routes and chrome browser .
/ : gives a html page with jquery and code snippets to fire ajax requests
/hi : gives a text response 'hello'
/sleep : request will timeout without any response
By default, node.js http server times out a request after 2 minutes.
retry.js
var http = require('http');
var server = http.createServer();
server.on('request', function(req, res) {
console.log(new Date() + ' ' + req.method + ' request on ' + req.url);
if (req.url === '/sleep') {
console.log('!!! sleeping');
} else if (req.url === '/') {
html = "$.post('/hi', {'for':'server'}, function() { console.log(arguments) } ).error(function() { console.log(arguments) })";
html += "<br><br>";
html += "$.post('/sleep', {'for':'infinite'}, function() { console.log(arguments) } ).error(function() { console.log(arguments) })";
html += '<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>';
res.writeHead(200, {'Content-Type': 'text/html'});
res.end(html);
} else {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('hello');
}
});
server.listen(2020);
console.log('server listening on port 2020');
run it
$ node retry.js
server listening on port 2020
1
Load this page in browser http://localhost:2020
Fri Mar 01 2013 12:21:59 GMT+0530 (IST) GET request on /
Fri Mar 01 2013 12:21:59 GMT+0530 (IST) GET request on /favicon.ico
2
From Dev console, fire an ajax POST request to /hi using jquery
$.post('/hi', {'for':'server'}, function() { console.log(arguments) } ).error(function() { console.log(arguments) })
Fri Mar 01 2013 12:22:05 GMT+0530 (IST) POST request on /hi
3
Fire a POST request to /sleep, results in a retry after 2 mins and errors out after 4 mins.
$.post('/sleep', {'for':'infinite'}, function() { console.log(arguments) } ).error(function() { console.log(arguments) })
server logs shows 2 requests
Fri Mar 01 2013 12:22:21 GMT+0530 (IST) POST request on /sleep
!!! sleeping
Fri Mar 01 2013 12:24:21 GMT+0530 (IST) POST request on /sleep
!!! sleeping
Firing it again, errors out in 2 mins without any retry.
Fri Mar 01 2013 12:30:01 GMT+0530 (IST) POST request on /sleep
!!! sleeping ?
It's not getting retried until we fire a request to /hi (or any other url) that results in a response. And retry happens for just one subsequent request to /sleep.
In browser, the network tab shows the pattern like
/hi - success
/sleep - cancelled - 4 mins (retry happens)
/sleep - cancelled - 2 mins (no retry)
/sleep - cancelled - 2 mins (no retry)
/hi - success
/sleep - cancelled - 4 mins (retry happens)
/sleep - cancelled - 2 mins (no retry)
/sleep - cancelled - 2 mins (no retry)
Question
Though we need to design our web app to tolerate these extra requests (either by browsers or any other intermediaries), this inconsistent browser retries looks weird. I had observed this behaviour in chrome (v16 & v24) & firefox.
Can someone help me to understand the browser retry logic behind timed out non-idempotent requests ?
Other relevant stackoverflow questions
What happens when no response is received for a request? I'm seeing retries
Browsers retry requests (including POSTs) when the connection is closed before receiving a response from the server. This is defined in the HTTP 1.1 Spec Section 8.2.4.
Related
I am setting up a AWS Lambda function to connect to my DynamoDB. To access it I'm also setting up an API Gateway.
The lambda seems to work when I test it. Because of this I believe the issue to be in the API gateway setup.
For the lambda I have configured the following test event:
I have configured a test event which looks like this:
{
"httpMethod": "GET"
}
This test event gives me the following response:
Response:
{
"statusCode": "200",
"body": "{\"Items\":[{\"id\":1,\"brand\":\"Test brand\",\"title\":\"Test product\"}],\"Count\":1,\"ScannedCount\":1}",
"headers": {
"Content-Type": "application/json"
}
}
For the API Gateway I have tried with the following test:
I have tried automatically creating the API Gateway in the lambda management console. Recreating the lambda and the API Gateway.
Lambda function:
console.log('Loading function');
const doc = require('dynamodb-doc');
const dynamo = new doc.DynamoDB();
/**
* Demonstrates a simple HTTP endpoint using API Gateway. You have full
* access to the request and response payload, including headers and
* status code.
*
* To scan a DynamoDB table, make a GET request with the TableName as a
* query string parameter. To put, update, or delete an item, make a POST,
* PUT, or DELETE request respectively, passing in the payload to the
* DynamoDB API as a JSON body.
*/
exports.handler = (event, context, callback) => {
//console.log('Received event:', JSON.stringify(event, null, 2));
const done = (err, res) => callback(null, {
statusCode: err ? '400' : '200',
body: err ? err.message : JSON.stringify(res),
headers: {
'Content-Type': 'application/json',
},
});
switch (event.httpMethod) {
case 'DELETE':
dynamo.deleteItem(JSON.parse(event.body), done);
break;
case 'GET':
dynamo.scan({ "TableName": "productdb" }, done);
//dynamo.scan({"TableName":"productdb"})
break;
case 'POST':
dynamo.putItem(JSON.parse(event.body), done);
break;
case 'PUT':
dynamo.updateItem(JSON.parse(event.body), done);
break;
default:
done(new Error(`Unsupported method "${event.httpMethod}"`));
}
};
API Gateway logs from the test:
Execution log for request 885e5505-2212-11e9-aee0-7f024016f574
Sun Jan 27 09:04:20 UTC 2019 : Starting execution for request: 885e5505-2212-11e9-aee0-7f024016f574
Sun Jan 27 09:04:20 UTC 2019 : HTTP Method: GET, Resource Path: /
Sun Jan 27 09:04:20 UTC 2019 : Method request path: {}
Sun Jan 27 09:04:20 UTC 2019 : Method request query string: {}
Sun Jan 27 09:04:20 UTC 2019 : Method request headers: {}
Sun Jan 27 09:04:20 UTC 2019 : Method request body before transformations:
Sun Jan 27 09:04:20 UTC 2019 : Endpoint request URI: https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:eu-central-1:304886708348:function:dynamoDBService/invocations
Sun Jan 27 09:04:20 UTC 2019 : Endpoint request headers: {x-amzn-lambda-integration-tag=885e5505-2212-11e9-aee0-7f024016f574, Authorization=*****************************************************************************************************************************************************************************************************************************************************************************************500617, X-Amz-Date=20190127T090420Z, x-amzn-apigateway-api-id=lqhm3agxxf, X-Amz-Source-Arn=arn:aws:execute-api:eu-central-1:304886708348:lqhm3agxxf/test-invoke-stage/GET/, Accept=application/json, User-Agent=AmazonAPIGateway_lqhm3agxxf, X-Amz-Security-Token=FQoGZXIvYXdzEOH//////////wEaDFvawdYGjH/+gSI14yK9AzQFZtlDghAr2NUHIhLGWmeJkKL8sUP3L6fu0h5PtFPN7wA7hgfWMtUNHCWyGykG0g5Zs81zKx5bUGMLCMK2zuVwD4WMgBRmkx40bZYehHdeS8czOxRTbQIqwP1lfZ0d74l4MqG4g8XpigkcLACLEn6buaq37rO4WYOo+J8ecFeSpti+u+V8OON4idxxXEHiYGJEc23OwjVvf3GTr1EUscB+Lsp/nw58oCWQArUA6LLSwcnGYXYcmnPav2Xs8mJgvqnVowxxYre0N8Gca8D9XBN2Y93/qnVTsOI5nWHSUQOnwaoXSZzgBAXKrUV1S5X+UH3zQI9p [TRUNCATED]
Sun Jan 27 09:04:20 UTC 2019 : Endpoint request body after transformations:
Sun Jan 27 09:04:20 UTC 2019 : Sending request to https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:eu-central-1:304886708348:function:dynamoDBService/invocations
Sun Jan 27 09:04:20 UTC 2019 : Received response. Integration latency: 17 ms
Sun Jan 27 09:04:20 UTC 2019 : Endpoint response body before transformations: {"statusCode":"400","body":"Unsupported method \"undefined\"","headers":{"Content-Type":"application/json"}}
Sun Jan 27 09:04:20 UTC 2019 : Endpoint response headers: {Date=Sun, 27 Jan 2019 09:04:20 GMT, Content-Type=application/json, Content-Length=108, Connection=keep-alive, x-amzn-RequestId=6c00229e-caa1-4d37-aeaa-7c1cbd0ddd71, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5c4d7414-e52b0fba267596b50fdbb102;sampled=0}
Sun Jan 27 09:04:20 UTC 2019 : Method response body after transformations: {"statusCode":"400","body":"Unsupported method \"undefined\"","headers":{"Content-Type":"application/json"}}
Sun Jan 27 09:04:20 UTC 2019 : Method response headers: {X-Amzn-Trace-Id=Root=1-5c4d7414-e52b0fba267596b50fdbb102;Sampled=0, Access-Control-Allow-Origin=*, Content-Type=application/json}
Sun Jan 27 09:04:20 UTC 2019 : Successfully completed execution
Sun Jan 27 09:04:20 UTC 2019 : Method completed with status: 200
I expect the result to be the same as the test event in the lambda returns.
When you create an API method, you need to select the option "Use Lambda Proxy integration" in order for the httpMethod field, along with other information from the API Gateway, to be accessible in the event object in your Lambda function.
From the docs:
You can set up a Lambda proxy integration for any API method. But a
Lambda proxy integration is more potent when it is configured for an
API method involving a generic proxy resource. The generic proxy
resource can be denoted by a special templated path variable of
{proxy+}, the catch-all ANY method placeholder, or both. The client
can pass the input to the backend Lambda function in the incoming
request as request parameters or applicable payload. The request
parameters include headers, URL path variables, query string
parameters, and the applicable payload. The integrated Lambda function
verifies all of the input sources before processing the request and
responding to the client with meaningful error messages if any of the
required input is missing.
You can find the "Use Lambda Proxy integration" option here, on the "Create Method" screen in your API Gateway instance:
Edit: For reference, you can tell that the API Gateway method is not using the Lambda proxy integration because under "Integration Request" the type is "LAMBDA", but when using the Lambda proxy integration the type is "LAMBDA_PROXY".
For anyone who couldn't get the above solutions to work, I had to change the switch statement for the http method that the Lambda function was originally built with
from: switch (event.httpMethod) {
to: switch (event.requestContext.http.method) {
I think V1.0 uses event.httpMethod? but I wasn't able to change the version in the API Gateway (message: "API Gateway managed resources are view only"). So I had to change the switch statement to match the event object.
I fixed the issue for HTTP API, by changing the payload format to V1.0 (under the integration details for the route).
I'm using nodejs on windows 7. I'm experiencing strange behavior. The same https request that always gives the same response is most of the time not received by node whereas it is always received by a web browser.
The code is quite simple:
var url1 = 'https:<my url>'
require('request').get(url1).pipe(process.stdout)
The response is actually a chunked response, which is a big json object.
If I try this request on a browser, I always receive the anwser.
If I use the node request module (I also tried with https), the body is most of the time empty. But sometimes it is returned.
I receive status 200 and response headers are:
{ 'content-type': 'application/json',
connection: 'close',
'transfer-encoding': 'chunked',
date: 'Wed, 27 Jan 2016 21:01:59 GMT',
server: 'My Server' }
Any idea?
I found 2 solutions to fix this issue:
1) Set header 'Connection': 'keep-alive'
var opts = { url: 'my url', headers :{ 'Connection':'keep-alive'} }
request(opts).pipe(..)
2) set to true KeepAlive option in https.globalAgent
https.globalAgent.keepAlive=true
I'd like to send a response code of 401 if the requesting user is not authenticated, but I'd also like to redirect when the request was an HTML request. I've been finding that Express 4 doesn't allow this:
res.status(401).redirect('/login')
Does anyone know of a way to handle this? This might not be a limitation of Express, since I'm asking to essentially pass two headers, but I don't see why that should be the case. I should be able to pass a "not authenticated" response and redirect the user all in one go.
There are some subtle diferences with the methods for sending back a new location header.
With redirect:
app.get('/foobar', function (req, res) {
res.redirect(401, '/foo');
});
// Responds with
HTTP/1.1 401 Unauthorized
X-Powered-By: Express
Location: /foo
Vary: Accept
Content-Type: text/plain; charset=utf-8
Content-Length: 33
Date: Tue, 07 Apr 2015 01:25:17 GMT
Connection: keep-alive
Unauthorized. Redirecting to /foo
With status and location:
app.get('/foobar', function (req, res) {
res.status(401).location('/foo').end();
});
// Responds with
HTTP/1.1 401 Unauthorized
X-Powered-By: Express
Location: /foo
Date: Tue, 07 Apr 2015 01:30:45 GMT
Connection: keep-alive
Transfer-Encoding: chunked
With the original (incorrect) approach using redirect:
app.get('/foobar', function (req, res) {
res.status(401).redirect('/foo')();
});
// Responds with
HTTP/1.1 302 Moved Temporarily
X-Powered-By: Express
Location: /foo
Vary: Accept
Content-Type: text/plain; charset=utf-8
Content-Length: 38
Date: Tue, 07 Apr 2015 01:26:38 GMT
Connection: keep-alive
Moved Temporarily. Redirecting to /foo
So it looks like redirect will abandon any previous status codes and send the default value (unless specified inside the method call). This makes sense due to the use of middleware within Express. If you had some global middleware doing pre-checks on all requests (like checking for the correct accepts headers, etc.) they wouldn't know to redirect a request. However the authentication middleware would and thus it would know to override any previous settings to set them correctly.
UPDATE: As stated in the comments below that even though Express can send a 4XX status code with a Location header does not mean it is an acceptable response for a request client to understand according to the specs. In fact most will ignore the Location header unless the status code is a 3XX value.
You can certainly send a Location: /login header alongside with your 401 page, however, this is ill-advised, and most browsers will not follow it, as per rfc2616.
One way to do overcome this, is to serve <meta http-equiv="refresh" content="0; url=/login"> alongside with your 401 page:
res.set('Content-Type', 'text/html');
res.status(401).send('<!DOCTYPE html><html><head><meta http-equiv="refresh" content="0; url=/login"></head></html>');
I fell on the same issue and decided to use the session to handle this kind of job.
I didn't want to have an intermediate view...
With the below code, I can redirect to the homepage, which will be rendered with 401 unauthorized code.
app.get('patternForbiddenRoute', (req, res, next) => {
// previousCode
if (notForbidden === true) {
return res.render("a_view");
}
req.session.httpCode = 401;
res.redirect('patternHomeRoute');
});
app.get('patternHomeRoute', (req, res, next) => {
res.render("my_home_view", {}, (error, html) => {
// ... handle error code ...
const httpCode = req.session.httpCode;
if (httpCode !== undefined) {
delete req.session.httpCode;
res.status(httpCode);
}
res.send(html);
}));
});
I am not entirely sure if this is a bad thing or not, but i was monitoring (logging) what incoming http request i get with this piece of code (note that running an NodeJS application on the scalable OpenShift platform):
function onRequest(request, response)
{
var date = new Date();
console.log(date.toUTCString() + " A request was made with url: " + request.url + " and header: " + JSON.stringify(request.headers));
// continue handling the request
}
The results i get are the following (every 2 seconds):
Fri, 07 Mar 2014 09:43:59 GMT A request was made with url: / and header: {}
Fri, 07 Mar 2014 09:44:01 GMT A request was made with url: / and header: {}
Fri, 07 Mar 2014 09:44:03 GMT A request was made with url: / and header: {}
So i am wondering if this is normal behaviour for a scalable NodeJS app (with a MongoDB database gear attached) in openshift, or is this something that could cause problems?
Sincerely,
Hylke Bron
If you are running a scaled application, then that is haproxy making sure that your application is up so that it can forward requests to it. You can change haproxy settings in your haproxy/haproxy.cfg file on your main gear.
I am using this method to inform OPENSHIFT that the application is alive
app.head("/", function(req, res, next){
res.status(200).end();
});
Since I didn't want to mess with haproxy.cfg, I did this and it seems to work. In your main app, add a middleware function to abort on the ping
function ignoreHeartbeat(except) {
except = except || 0;
var count = except;
return function(req, res, next) {
if (req.headers["x-forwarded-for"])
return next();
if (except > 0) {
if (--count <= 0) {
count = except;
return next();
}
}
res.end();
}
}
app.use(ignoreHeartbeat(1000));
Place that code before the call to setup the logger (Express 3 example shown below)
app.use(express.logger(...))
This logs out every 1000th ping. Set except to 0 or -1 to ignore all the pings.
I have a simple API upload, it is used to accept upload file from client.
var flg=true;
app.post('/test', function(req, res){
flg=!flg;
var returnJson='{';
if(flg){
req.form.on('part', function (part) {
if(part){
part.resume();
}
returnJson=returnJson+',\"status\":\"0\"}';
res.send(returnJson);
});
}else{
console.log('close');
returnJson=returnJson+',\"status\":\"1\"}';
res.header('Connection', 'close');
res.send(413, returnJson);
}
});
I'd like to test this API with Jmeter. "status":"0" means success. "status":"1" means failure. I write Jmeter script like this:
http://i.imgur.com/vEUJKc8.jpg
Jmeter only displays all the samplers which response contains "status":"0". it seems Jmeter exclude failure sampler response which comes from else section.
http://imgur.com/bkFSpK2
How can I see all samplers which includes all success and failure samplers in Jmeter?
successful Sampler result is :
Thread Name: API 1-1
Sample Start: 2013-12-18 11:46:08 PST
Load time: 7
Latency: 6
Size in bytes: 178
Headers size in bytes: 163
Body size in bytes: 15
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 15
Date: Wed, 18 Dec 2013 19:46:08 GMT
Connection: keep-alive
HTTPSampleResult fields:
ContentType: text/html; charset=utf-8
DataEncoding: utf-8
Any suggestion?
I don't like this stanza:
ContentType: text/html;
Correct ContentType for JSON will be application/json
You can try using HTTP Header Manager to set Content-Type header of your request to be application/json and see what happens.
Also there is a JSON plugin which provides JSON Path Extractor and JSON Path Assertion (Select “Extras with libs set” from the download list).