I'm using the node js request module to send emails via sendgrid. I am getting the error ETIMEDOUT. I intend on using node-retry npm module to retry the sending, but how can I detect what the error code is? Does the sendgrid API return the error code somehow? Also when I do detect the error code, is it just a matter of waiting X seconds to send the email again? If so how do I determine what X is?
_makeAPIRequest (httpMethod, url, body) {
var defer = Q.defer();
var options = {
method: httpMethod,
url: this.SENDGRID_API_URL + url,
headers: {
'content-type': 'application/json',
authorization: 'Bearer ' + SENDGRID_API_KEY
},
body: body,
json: true
};
request(options, function (error, response, body) {
if (error) {
console.dir(error);
return defer.reject(error);
}
defer.resolve(body);
});
return defer.promise;
}
ETIMEDOUT is an OS error message. It indicates a failed attempt, at the TCP/IP level, to connect to a remote host, probably the one mentioned in SENDGRID_API_URL.
The default value for that is https://api.sendgrid.com/v3/. For some reason, possibly an outbound firewall or some sort of network configuration trouble, your nodejs program cannot reach that URL, and waits for a response. You should check your value of that URL.
If this is intermittent (doesn't happen all the time) you probably can wait a few seconds and try again.
If it starts happening after you've sent a bunch of emails, you may be hitting a limit at sendgrid. Pace out your sending of emails; try putting a half-second delay between them.
Related
here is my code that makes an Http Get request to an API end point from one of the services running on Amazon Fargate service. The API is powered by Amazon API gateway and Lambda. Also this is a private api used with in the VPC and I also have setup the apigateway VPC end point to facilitate the same. I have received this error only once. All the subsequent calls made to the API were successful.
My suspicion is that the lambda was not warm and that resulted a timeout. I am going to try setting a timeout for the axios code. any suggestions welcome
async getItems(): Promise < any > {
try {
let url = `https://vpce-[id].execute-api.ap-southeast-2.vpce.amazonaws.com/prod/items`
const response = await axios.get(url, {
headers: {
'Authorization': `Bearer ${token}`,
'x-apigw-api-id': `[api-id]`
}
});
return response.data;
} catch(error) {
console.log(error);
throw error;
}
}
Turns out my lambda is timing out after the 30 seconds configured time. I could increase the lambda timeout, but the configurable timeout for API gateway is 30 seconds.
It has only happened once and i believe that it's because lambda cold start. As a workaround, I am taking the retry approach. The API request will be retried 3 times.
My node server has a strange behaviour when it comes to a GET endpoint that resplies with a big JSON (30-35MB).
I am not using any npm package. Just the core API.
The unexpected behaviour only happens when querying the server from the Internet and it behaves fine if it is queried from the local network.
The problem is that the server stops writing to the response after it writes the first 1260 bytes of the content body. It does not close the connection nor throw an error. Insomnia (the REST client I use for testing) just states that it received a 1260B chunk. If I query the same endpoint from a local machine it says that it received more and bigger chunks (a few KB each).
I don't even think the problem is caused by node but since I am on a clean raspberry pi (installed raspbian and then just node v13.0.1) and the only process I use is node.js I don't know how to find the source of the problem, there is no load balancer or web server to blame. Also the public IP seems OK, every other endpoint is working fine (they reply with less than 1260B per request)
The code for that endpoint looks like this
const text = url.parse(req.url, true).query.text;
if (text.length > 4) {
let results = await models.fullTextSearch(text);
results = await results.map(async result=>{
result.Data = await models.FindData(result.ProductID, 30);
return result;
});
results = await Promise.all(results);
results = JSON.stringify(results);
res.writeHead(200, {'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Access-Control-Allow-Origin': '*', 'Cache-Control': 'max-age=600'});
res.write(results);
res.end();
break;
}
res.writeHead(403, {'Content-Type': 'text/plain', 'Access-Control-Allow-Origin': '*'});
res.write("You made an invalid request!");
break;
Here are a number of things to do in order to debug this:
Add console.log(results.length) to make sure the length of the data is what you expect it to be.
Add a callback to res.end(function() { console.log('finished sending response')}) to see if the http library thinks it is done sending the response.
Check the return value from res.write(). If it is false (indicating that not all data has yet been sent), add a handler for the drain event and see if it gets called.
Try increasing the sending timeout with res.setTimeout() in case it's just taking too long to send all the data.
Hello i'm using sheets v4 in my nodejs application everything works good for the majority of users but for some users using the service they always get INTERNAL status error
what's the meaning of this status and how can i fix it.
here's the error message:
{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}
here's my code:
getSpreadsheetValues(tokens, ssID, SheetName, range, callback){
request({
url: 'https://sheets.googleapis.com/v4/spreadsheets/'+ssID+'/values/'+SheetName+'!'+range+'?access_token='+tokens.access_token,
method: 'GET'
}, function(error, response, body){
var body = JSON.parse(body)
callback(body.error, body.error ? "":body);
});
}
The API has a default quota of 100 reads/day. After that it throws a meaningless error message such as Internal Error Encountered. I suddenly encountered these errors, even though I didn't change any of my code [github].
You can put a 100 second delay every 100 requests.
I am using request npm module in my app, to make to create a http client, as this.
var request = require('request');
And each time, I make a request to some server, I pass the options as below:
var options = {
url: "whateverurl...",
body: { some json data for POST ... }
}
request(options, cb(e, r, body) {
// handle response here...
})
This was working fine, until I started testing with high load, and I started getting errors indicating no address available (EADDRNOTAVAIL). It looks like I am running out of ephemeral ports, as there is no pooling or keep-alive enabled.
After that, I changed it to this:
var options = {
url: "whateverurl...",
body: { some json data for POST ... },
forever: true
}
request(options, cb(e, r, body) {
// handle response here...
})
(Note the option (forever:true)
I tried looking up request module's documentation about how to set keep-alive. According to the documentation and this stackoverflow thread, I am supposed to add {forever:true} to my options.
It didn't seem to work for me, because when I checked the tcpdump, the sever was still closing the connection. So, my question is:
Am I doing something wrong here?
Should I not be setting a global option to request module, while I am "require"ing it, instead of telling it to use {forever:true}, each time I make a http request? This is confusing to me.
Trying to understand how to work properly with :
1. Express
2. request
3. middleware
It's a follow up question from here where the discussion wad fruitful and helpfull (thanks #BlazeSahlzen , you are great!) but I realize that I tried at one point to put too much issues (although they are all related) into the same question.
So, this one is a focused question... I hope :-)
Case: I want to build POST() that recives parameter via path (/:param1),
uses it to request() #1 an external API,
gets the result from the external API,
Uses the result to do somwething and send ANOTHER request() #2 to a 2nd external API,
get's the outcome of the 2nd APi request(),
decide if the POST is statusCode = 200 with message="ok" or statusCode = something_else and message = "problem"
and res.send() it properly.
for that, here is my pseudo code -
var middle_1 = function(req, res, next) {
param1 = req.params.param1; //trying to access the param1 from the path, not sure it will work in middleware
req.middle_1_output = {
statusCode: 404,
message: "param1"
}
var options = {
method: 'PUT',
url: `EXTERNAL_API_1`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function(error, response, body) {
if (error) throw new Error(error);
// CODE THAT DO SOMETHING AND GET INFORMATION
req.request_1_output.statusCode = 200;
req.request_1_output.message = "hello world";
next(); // not sure what happens here - will it "jump" outside of the middle_1() or go to the next request() down the code??
});
var options = {
method: 'PUT',
url: `EXTERNAL_API_2`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function(error, response, body) {
if (error) throw new Error(error);
//Can I use here the req.request_1_output.message ???
//How can I use here ALSO some of the EXTERNAL_API_1 outcome????
// Some more CODE THAT DO SOMETHING AND GET INFORMATION
req.request_2_output.statusCode = 201;
req.request_2_output.message = "hello world";
next(); // not sure what happens here
});
}
//This middleware is only used to send success response
var response_success = function(req, res) {
sum_statusCode = req.request_1_output.statusCode + req.request_2_output.statusCode;
if (req.request_2_output.message == req.request_1_output.message) {
meassge = "hello world";
} else {
message = "goodbye world!";
}
res.json({
"statusCode": sum_statusCode,
"message": message
});
}
app.post('/test', middle_1, response_success);
I am not sure how to connect the different requests (request #1 and request #2) in this case - should they all become middleware? how should I write it? (connect => make them run one only after the other is done.)
How can I get also infomation from the request #1 outcome and use it in the request #2 ?
look at my code at response_success() -> will this work? can I access like this data from req that originated within the request #1 and request #2?
How am I suppose to access inside the response_success() data which is the OUTCOME of the request #1 and request #2?
// EDITED - question #5 and #6 are a late edition of mine but should be a stand alone questions. I leave them here but I will be opening a new thread just for them.
Let's say my middle_1 needs to get information as an outcome from the request_1 , calculate something, and move it forward to a middle_2... how do I take the request_1 information into something that can be transffered into a middle_2? I think I am suppose to create a property inside "req" , something like req.middle_1_outcome = DATA , but I am not sure how to "get the DATA" from the request_1 outcome...
How do I "monitor and wait" for request_1 to be done before my middle_1 moves forward to calculate things? is there a requestSync() funciton for Synced requests?
Thanks in advance to all the helpers :-)
A given middleware function should call next() only once when it is done with all its processing.
A given request handler should only submit one response per request. Anything more will be considered an error by Express.
I am not sure how to connect the different requests (request #1 and
request #2) in this case - should they all become middleware? how
should I write it? (connect => make them run one only after the other
is done.)
If your two request() calls can run in parallel (the second one does not depend upon the first results), then you can run them both and then monitor when they are both done, collect the results, do what you need to do with the request and then once and only once call next().
If they must be run in sequence (use the results from the first in the second), then you can nest them.
How can I get also information from the request #1 outcome and use it
in the request #2 ?
There are a variety of ways to solve that issue. If the requests are being run in sequence, then the usual way is to just put request #2 inside the completion callback for request #1 where the results from #1 are known.
Look at my code at response_success() -> will this work? can I access like this data from req that originated within the request #1 and request #2?
You can't quite do it like that because you can't call next() multiple times from the same middleware.
How am I suppose to access inside the response_success() data which is the OUTCOME of the request #1 and request #2?
If you nest the two operations and run request #2 from inside the completion of
request #1, then inside the completion for request #2, you can access both results. There is no need to a completely separate request handler to process the results. That just makes more complication that is necessary.
If you need to serialize your two requests because you want to use the result from the first request in the second request, then you can use this structure where you nest the second request inside the completion of the first one:
function middle_1(req, res, next) {
var param1 = req.params.param1; //trying to access the param1 from the path, not sure it will work in middleware
var options = {
method: 'PUT',
url: `EXTERNAL_API_1`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function (error, response, body) {
if (error) return next(error);
// CODE THAT DO SOMETHING AND GET INFORMATION
var options = {
method: 'PUT',
url: `EXTERNAL_API_2`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
// this second request is nested inside the completion callback
// of the first request. This allows it to use the results from
// from the first request when sending the second request.
request(options, function (error2, response2, body2) {
if (error2) return next(error2);
// right here, you can access all the results from both requests as
// they are all in scope
// response, body from the first request and
// response2, body2 from the second request
// When you are done with all your processing, then you
// can send the response here
res.json(....);
});
});
}
app.post('/test', middle_1);
Note several things about the structure of this code:
To use the results of the first request in the second one, just nest the two.
When nesting like this, the results from both requests will be available in the completion callback for request #2 as long as you give the arguments unique names so they don't accidentally hide parent scoped variables of the same name.
It does you no good to throw from an async callback, since there's no way for your code to ever catch an exception throw from a plain async callback. The request will likely just sit there forever until it eventually times out if you throw. You need to actually handle the error. Since I didn't know what error handling you wanted, I called next(err) to at least give you default error handling.
I would not suggest using multiple middleware functions that are really just one operation. You may as well just put the one operation in one request handler function as I've shown.