I'm making an HTTP request using https://github.com/request/request and I want to receive JSON. The response will be seriously large, so I want to use a stream approach to process the response. However, the API I'm calling returns 'text/plain' for status >= 400, which means that my JSONStream will bork. Code:
req = request.get({url: data_url});
req.pipe(require('JSONStream').parse([true]))
.pipe(require('stream-batch')({maxItems: 1000}))
.on('data', callback)
.on('end', done);
Error:
Invalid JSON (Unexpected "I" at position 0 in state STOP)
("A" as in "Internal server error".) It seems request does not emit 'error' events for requests that completes so adding req.on('error', (err) => ...) does not trigger. I could add
req.on('response', function(res) {
if(res.statusCode >= 400) { ... }
})
But then I seem not to be able to get at the error message in the body.
How can I get at the error message, log meaningfully and break processing?
Since the argument passed to the response event is a readable stream as well, you can create the pipeline inside its handler:
req.on('response', function(res) {
if (res.statusCode >= 400) {
let chunks = [];
res.on('data', c => chunks.push(c))
.on('end', () => {
console.log('error', Buffer.concat(chunks).toString());
});
} else {
res.pipe(require('JSONStream').parse([true]))
.pipe(require('stream-batch')({maxItems: 1000}))
.on('data', callback)
.on('end', done);
}
})
Related
I'm having a very strange error where my http requests aren't sending at all. Once I send the request it just stops executing. I'm ingesting a CSV file through a ReadStream and sending requests inside the emitter listening blocks. If I try to send the request outside the emitters it can send fine, but I need to POST data from the file so it has to be sent within them (unless there's a way to export the data I don't know about.
Code:
const buffer = multipart.parse(event, true).fileName.content;
const file = stream.Readable.from(buffer);
let events;
file.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
events = parseEvents(event, results);
console.log("sending request");
request({method: 'POST', url: 'https://httpbin.org/anything', json: {hello: 'world'}}, function (err, response, body) {
console.log(err);
console.log(response);
console.log(body);
});
console.log("finished request");
})
.on('error', error => {
console.log(error);
});
Before you say, I've also tried all kinds of requests. Using got, Axios, and request I've done awaits and tried to process it that way. I actually can get the promise but if I await it nothing happens. It's also not stuck in an infinite loop or anything because when put in a for loop it tries it every time and just always returns nothing.
For more info: The console gets the "sending request" log and then "finished request" and that's it. If I go the promise route, it doesn't even log the "finished request".
Per the documentation, url is not an option for http.request(), nor is json:
https://nodejs.org/dist/latest-v16.x/docs/api/http.html#httprequestoptions-callback
This seems like an async issue. After you fire your request, your code executes the next line rather than waits for the response and thats why you see the logs for sending and finished. Not sure how you implemented the promise solution before, and without seeing the rest of your code it would be hard to debug but I assume something like this should work.
return new Promise((resolve, reject) => {
file.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
events = parseEvents(event, results);
console.log("sending request");
request({method: 'POST', url: 'https://httpbin.org/anything', json: {hello: 'world'}}, function (err, response, body) {
console.log(err);
console.log(response);
console.log(body);
resolve(body);
});
console.log("finished request");
})
.on('error', error => {
reject(error);
console.log(error);
})
});
Then wherever you are returning the promise to you can call .then(() => { //do stuff })
I'm making a cryptocurrency dashboard for a project, and I'm completely new to Node and Express. This is what I have currently
app.get('/search', function(req,res){
res.writeHead(200, {'Content-Type': 'text/html'});
res.write(req.url);
data = []
var options = {
"method": "GET",
"hostname": "rest.coinapi.io",
"path": "/v1/assets",
"headers": {'X-CoinAPI-Key': 'MY_API_KEY_HERE'}
};
const request = https.request(options, (data, response) => {
response.on('data', d => {
data.push(d);
})
});
console.log(data);
request.end();
res.end();
})
The idea is on my front end, I have a button that when clicked, will make a request to the CoinAPI api, getting all reported assets and current values. I'm not quite sure how I'm supposed to send that response data back to my frontend as a response. So, I tried to pull the response data out of the JSON that gets returned by the https.request line. I have a data array data = [] as you can see at the top of my code.
I originally had my request set up like:
const request = https.request(options, response => {
but when I would try to push d onto data, I console logged and the data array was empty. This makes sense, the data array is out of scope of the request function, so data doesn't get updated. But when I tried to pass data into the function, I errored out.
Basically I want to be able to send the JSON data back to my front end after making the request to CoinAPI. If I do process.stdout.write(d) in my https.request callback, I do actually see the coinapi data coming back. I just don't know how to send it to the front end as part of my response.
Issues:
The use of (data, response) is incorrect. The first and only argument is response so it should be (response).
The on('data') event receives buffered data. Concatenating it to a final string is the standard usage, not appending an array.
You're missing an on('end') which you should use to output the final data or do any processing.
Using res.write you're sending a text/html content type and some content which you don't want if the goal is to output JSON which the frontend can parse and use.
Missing an error handler for the API call.
Complete updated code:
app.get('/search', function(req,res){
let str = '';
const options = {
"method": "GET",
"hostname": "rest.coinapi.io",
"path": "/v1/assets",
"headers": {'X-CoinAPI-Key': 'MY_API_KEY_HERE'}
};
const request = https.request(options, (response) => {
response.on('data', d => {
str += d;
});
response.on('end', () => {
try {
let obj = JSON.parse(str);
// do any manipulation here
res.json(obj);
} catch(e){
console.log(e);
res.status(500).json({ message: 'Something went wrong - parse error' });
}
});
});
request.end();
request.on('error', (e) => {
console.log(e);
res.status(500).json({ message: 'Something went wrong - req error' });
});
});
I added a JSON.parse() to show how you'd handle that if you wanted to do some manipulation of the data before sending it to the frontend. If you simply want to return the exact response of the coin API then use an end event like:
response.on('end', () => {
res.json(str);
});
To send JSON data back to the client as response all you need to do is :
return res.json(data);
Its as simple as this. :)
I am performing some http calls in an AWS lambda. The number of calls are ~400 per minute.
The calls are performed as in the following snippet
var req = http.get("https://www.google.com", res => {
let body = '';
res.on('data', chunk => {
body += chunk;
});
res.on('end', chunk => {
if (body.includes('html')) {
console.log('Got back healthy response');
} else {
console.log('Got an unexpected response');
}
})
});
req.on('error', e => {
console.log('Got an error response');
})
That is a simple https request. When the Lambda is invoked, it performs ~40 requests in a single execution.
My issue is that at the beginning, everything looks good and all the requests are performed correctly. After a while (that can be after ~30 min) calls start to degrade and I get back "socket hang up ECONNRESET" error.
I then tried using the request module and change the code with the following
const request = require('request');
request("https://www.google.com", function (error, response, body) {
if (!error && response.statusCode == 200 && body.includes('html')) {
console.log('Got back healthy response' );
} else {
console.log('Got an unexpected response');
console.log('Error: ' + error);
console.log('Response: ' + response);
console.log('Body: ' + body);
}
});
In this case, with the same number of requests within the same lambda, same setting I never get the ECONNRESET error.
I'm ok using the request module but I'm curious to know why this was happening with the default http implementation.
Is this caused by socket allocation that the request module handles in a more appropriate way?
I know similar questions have been asked already but I didn't find a good answer for my case.
This isn't really an answer but I cannot write comments.
The main difference I can see is the encoding. The default encoding in request is utf8, whereas in the http module it's buffer. Adding res.setEncoding('utf8'); might be helpful. It might not be faster. In the line body += chunk you just implicitly convert a Buffer to a string so it should be the same.
If adding the setEncoding won't change anything then you might want to report an issue to the nodejs team because it's the same as the example in http_http_get_url_options_callback. They should fix it or change the example.
Looking to log an error and reject a request if a node request stream returns any statusCode other than 200 range, but I want to get the response from the request if available even if it fails because it sometimes has useful information on failure cause. So my question is if I put my check for statusCode inside the end event, is there a possibility it will never be called and the program will hang? Is there a better way to get the response body and reject on statusCode?
const request = http.get(url, (response) => {
const body = [];
response.on('data', (chunk) => body.push(chunk));
response.on('end', () => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new ErrorClass(response.statusCode, body.join('')));
} else {
resolve(body.join(''))
}
});
});
The response passed to the callback is a Readable Stream.
The Node.js doc says this about the end event:
The 'end' event is emitted when there is no more data to be consumed
from the stream.
Note: The 'end' event will not be emitted unless the data is
completely consumed. This can be accomplished by switching the stream
into flowing mode, or by calling stream.read() repeatedly until all
data has been consumed.
It says this about the data event:
Attaching a 'data' event listener to a stream that has not been
explicitly paused will switch the stream into flowing mode. Data will
then be passed as soon as it is available.
And it says this about the error event that can be emitted by the request object returned by http.request (of which http.get is a special case):
If any error is encountered during the request (be that with DNS
resolution, TCP level errors, or actual HTTP parse errors) an 'error'
event is emitted on the returned request object. As with all 'error'
events, if no listeners are registered the error will be thrown.
Since your code attaches a data event listener and consumes all chunks, the end data listener will be called after there is no more data to be consumed -- unless the error event is emitted (usually due to some underlying issue that prevents completion of response processing). Therefore, you code should also listen for the error event. For example:
const request = http.get(url, (response) => {
const body = [];
response.on('data', (chunk) => body.push(chunk));
response.on('end', () => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new ErrorClass(response.statusCode, body.join('')));
} else {
resolve(body.join(''))
}
});
});
request.on('error', (e) => {
reject(e);
});
Calling the Riot-Api Im receiving incomplete JSON on a https GET-request.
After debugging, I realized that depending how much I wait (breakpoint) pre-executing the
https on'data' callback Im actually receiving the complete JSON object.
(Average API response time for me is 200-300ms)
let getOptions = function(url) {
return {
host: 'na.api.pvp.net',
port: 443,
path: `${url}?api_key=${apiKey}`,
method: 'GET'
};
}
exports.Call = function(url, callback) {
let response = {};
let req = https.request(getOptions(url), function(res) {
response.statusCode = res.statusCode;
res.on('data', function(data) {
response.json = JSON.parse(data);
callback(response);
});
});
req.on('error', function(err) {
response.err = err;
callback(response);
});
req.end();
};
Running the code without breakpoints or only breaking a short time I run either into error:
JSON.parse(data): Unexpected Token in JSON at position ...
or
JSON.parse(data): Unexptected end of JSON Input.
As I expect the 'data' callback to be executed only after the request is complete im confused about how to fix it (without artificially delaying it ofc.).
http.request returns a stream – its not a simple callback that contains the whole response.
You will have to buffer and concatenate everything if you want to parse the whole response.
I would strongly recomment to use a helper library like got or request