I am trying to parse a response from an API which is a streamed response: but the structure looks different from any other response.
A sample can be found in python.
let body = []
response.on('data', function(chunk) {
body.push(chunk)
})
.on('end', function() {
const bff = Buffer.concat(body).toString("utf-8")
console.log( bff)
})
it returns like:
�U
:content-typeapplication/octet-stream
:event-typeRecordsJane,(949) 123-45567,Chicago,Developer
:content-typtext/xml
:event-typeStats<?xml version="1.0" encoding="UTF-8"?><Stats><BytesScanned>339</Byt:message-typeeventocessed>339</BytesProcessed><BytesReturned>39</BytesReturned></Stats>�h�8(�Ƅ�
:event-typeEndϗӒ
what is the correct way to parse such a response?
Use Bytes parsing because it is a nested message.
The spec to parse can be found here
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTSelectObjectAppendix.html
Related
I'm making a GET request to an API that responds with Transfer-Encoding: chunked.
It's an https request, I've attached on('data') listener to it which then adds the response to an array.
There are around 5 chunks incoming each of them being a Buffer depending on the response. When I concat the array and try converting it to a String I get the weirdly decoded reply which looks to be encoded. I think it might be due to the request being HTTPS but I'm not sure, below's my code for decoding it
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
try{
const formattedData = JSON.parse(buffer.toString('utf8'));
resolveRequest(formattedData);
}catch(error){
setTimeout(makeRequest, 5000);
}
});
Any help would be appreciated.
Check your res.headers['content-encoding'], the response may be encoded by gzip.
So you need to unzip the response buffer like this:
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
zlib.gunzip(Buffer.concat(respData), function (err, decoded) {
if (err) throw err;
const formattedData = decoded.toString()
resolveRequest(formattedData)
});
});
app.get("/dashboard", function(req, res) {
const country = "Singapore";
// I used a constant in desperation
const url =
"https://api.covid19api.com/live/country/" + country;
https.get(url, function(response) {
response.on("data", function(data) {
const caseData = JSON.parse(data);
// Error stems from here
res.render("cases", { caseData: caseData });
});
});
});
You're not waiting for the entire message before trying to parse it.
The data event is invoked for each chunk of data from the HTTP response. The response may not necessarily fit in a single chunk.
You'll have to wait for the end event before trying to parse the JSON... or preferably use a higher-level library such as node-fetch for a more comfortable promise-based API.
I'm making a cryptocurrency dashboard for a project, and I'm completely new to Node and Express. This is what I have currently
app.get('/search', function(req,res){
res.writeHead(200, {'Content-Type': 'text/html'});
res.write(req.url);
data = []
var options = {
"method": "GET",
"hostname": "rest.coinapi.io",
"path": "/v1/assets",
"headers": {'X-CoinAPI-Key': 'MY_API_KEY_HERE'}
};
const request = https.request(options, (data, response) => {
response.on('data', d => {
data.push(d);
})
});
console.log(data);
request.end();
res.end();
})
The idea is on my front end, I have a button that when clicked, will make a request to the CoinAPI api, getting all reported assets and current values. I'm not quite sure how I'm supposed to send that response data back to my frontend as a response. So, I tried to pull the response data out of the JSON that gets returned by the https.request line. I have a data array data = [] as you can see at the top of my code.
I originally had my request set up like:
const request = https.request(options, response => {
but when I would try to push d onto data, I console logged and the data array was empty. This makes sense, the data array is out of scope of the request function, so data doesn't get updated. But when I tried to pass data into the function, I errored out.
Basically I want to be able to send the JSON data back to my front end after making the request to CoinAPI. If I do process.stdout.write(d) in my https.request callback, I do actually see the coinapi data coming back. I just don't know how to send it to the front end as part of my response.
Issues:
The use of (data, response) is incorrect. The first and only argument is response so it should be (response).
The on('data') event receives buffered data. Concatenating it to a final string is the standard usage, not appending an array.
You're missing an on('end') which you should use to output the final data or do any processing.
Using res.write you're sending a text/html content type and some content which you don't want if the goal is to output JSON which the frontend can parse and use.
Missing an error handler for the API call.
Complete updated code:
app.get('/search', function(req,res){
let str = '';
const options = {
"method": "GET",
"hostname": "rest.coinapi.io",
"path": "/v1/assets",
"headers": {'X-CoinAPI-Key': 'MY_API_KEY_HERE'}
};
const request = https.request(options, (response) => {
response.on('data', d => {
str += d;
});
response.on('end', () => {
try {
let obj = JSON.parse(str);
// do any manipulation here
res.json(obj);
} catch(e){
console.log(e);
res.status(500).json({ message: 'Something went wrong - parse error' });
}
});
});
request.end();
request.on('error', (e) => {
console.log(e);
res.status(500).json({ message: 'Something went wrong - req error' });
});
});
I added a JSON.parse() to show how you'd handle that if you wanted to do some manipulation of the data before sending it to the frontend. If you simply want to return the exact response of the coin API then use an end event like:
response.on('end', () => {
res.json(str);
});
To send JSON data back to the client as response all you need to do is :
return res.json(data);
Its as simple as this. :)
I'm trying to access the body parameters from the request using only the http internal module from NodeJS. I've looked up everywhere and I can't seem to find how to do that other than using express or some other library. I know that with express you do that by simply using req.body.myparameter. Can I do it only with http?
You will then need to handle data Buffer manually:
if (req.method === 'POST') {
let body = '';
req.on('data', chunk => { //start reading data
body += chunk.toString(); // convert Buffer to string
});
req.on('end', () => {
console.log(body); //logging submitted data
res.end('ok');
});
}
I am still a beginner in Node.js and I am trying to explore as much as I can.
I know that Express.js is a framework used by many people for creating websites in Node.js.
But without using Express.js, I know that it is it possible to read .html files using 'fs.readFile', and then "display" this .html file in the browser.
Is there a way to get user input (say a button click, or fill in a box) from this web page into Node.js? So far, I have not found any examples of this.
Yes, this is possible. Study how the connect bodyParser's urlencoded function works.
When a request comes in from the browser, node is going to represent this as a readable data stream. For web forms, the pattern will be:
Use the request's data and end events to buffer the chunks of data from the stream into a single string.
Parse that string appropriately given its data format. In the case of a web form, this will normally urlencoded (application/x-www-form-urlencoded) MIME type
.
var qs = require('qs'); //https://github.com/visionmedia/node-querystring
function handle(req, res) {
var buf = '';
req.setEncoding('utf8');
req.on('data', function(chunk){
//assemble the request from distinct chunks into a single string
buf += chunk
});
req.on('end', function(){
//OK, you have a usable string request body, parse it and handle it
try {
var formData = qs.parse(buf);
//Yay, it parsed. Now you have your form data
//depending on your form's html, you might have formData.email, for example
} catch (err){
//oops, respond with an error
}
});
}
Tutorial
Long story short:
http.createServer(function (req, res) {
var data = '';
req.on('data', function(chunk) {
console.log("Received body data:");
console.log(chunk);
data += chunk.toString();
});
req.on('end', function() {
console.log('Received Data: ', data);
res.end();
});
}