Express JS - Network connection was lost - node.js

I am trying to set up an express js server that will be hosting a mongodb database. Everything is pretty standard: I have some routes open that will take in data from the client and then store that in the database.
Here is my query string:
let url = "http://xxx.xxx.xx.xxx:3000/update/data=" + JSON.stringify(params);
What I have noticed is that if params doesn't contain much information, it works fine. However, if params is contains a lot of information, then the client throws this error:
Failed to load resource: The network connection was lost.
Http failure response for (unknown url): 0 Unknown Error
(This same error is happening in both Safari and Chrome.)
For example, if params is as below:
{
"accountId": "12345678910",
"data": [
1, 2, 3, 4
]
}
then there is no issue. However, if params.data is a huge array with a ton of information in it instead of just [1, 2, 3, 4], then the error is thrown.
Also, my express server never even seems to receive the request. No logs; nothing. What I would expect to happen is just a normal response and result, however it seems like the client is just giving up on sending something large. Perhaps it has something to do with sending it as a big string?

You put your data on your URL. But, URLs have limited length.
You need to use POST and put your data in the HTTP request body.
You haven't shown us how you use that URL, so it's hard to make suggestions about altering your code. Using the http request operation is the way to go. Something like this might work...
const payload = JSON.stringify(params);
const url = 'http://xxx.xxx.xx.xxx:3000/update/';
const options = {
method: 'POST', // <--- tell it to POST
headers: {
'Content-Type': 'application/json', // <--- tell it you're posting JSON
'Content-Length': payload.length; // <--- tell it how much data you're posting.
}
};
const req = http.request(url, options, (res) => {
/* handle stuff coming back from request here */
console.log(`STATUS: ${res.statusCode}`);
console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
res.setEncoding('utf8');
let chunks=[];
res.on('data', (chunk) => {
chunks.push(chunk);
console.log(`BODY: ${chunk}`);
});
res.on('end', () => {
const resultingData = chunks.join();
console.log('No more data in response.');
});
});
req.on('error', (e) => {
console.error(`problem with request: ${e.message}`);
});
// write data to request body
req.write(payload);
req.end();

Related

How to make an https version of a Unirest example

I would like to use the https library in node.js to send a request to this api:
https://rapidapi.com/dimas/api/NasaAPI?endpoint=apiendpoint_b4e69440-f966-11e7-809f-87f99bda0814getPictureOfTheDay
The given example on the RapidAPI website uses Unirest, and I would like to only use the https library. I've tried to write it like this:
const https = require('https');
var link = "https://NasaAPIdimasV1.p.rapidapi.com/getPictureOfTheDay";
var options = {host: "https://NasaAPIdimasV1.p.rapidapi.com/getPictureOfTheDay",
path: "/", headers: {"X-RapidAPI-Key": "---MY KEY(Yes, I've replaced it)---", "Content-Type": "application/x-www-form-urlencoded"}}
https.get(link, options, (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end', () => {
console.log(data);
});
}).on("error", (err) => {
console.log("https error 4: " + err.message);
});
But that returns the following response:
{"message":"Endpoint\/ does not exist"}
Thanks for any help
There are several mistakes.
First, you essentially pass URL in https twice - first as link param, second as combination of host and path properties for options param.
Second, your host is actually the full path - but it shouldn't be. In the end, looks like the library got confused and sent request to https://NasaAPIdimasV1.p.rapidapi.com/ instead.
Finally, this particular API requires using 'POST', not 'GET' method. That's actually mentioned in the documentation. That's why you have 'endpoint does not exist' error even on correctly formed request.
One possible approach is dropping link altogether, sending URL as part of options:
var options = {
host: 'NasaAPIdimasV1.p.rapidapi.com',
method: 'POST',
path: '/getPictureOfTheDay',
headers: {/* the same */}
};
https.request(options, (resp) => { /* the same */ }).end();

Sending Form Data with the native node module

for my current project I have to send form-data from my lambda function to an api endpoint. The api endpoint essentially expects two images (that it compares with one another) and a key. As mentioned before, I somehow seem unable to send the correct form-data to the api endpoint. I checked out postman, and it seems to have worked alright, but something doesn't seem to work in my function. I presume it must be related the form-data string that I'm sending. Below you can find a shortened version of the function (I excluded the two image files), but somehow I'm getting an error back telling me that the api cannot read the key property:
const http = require('http');
const https = require('https');
const httpPromise = (protocol, params, postData) => {
return new Promise((resolve, reject) => {
const requestModule = protocol === 'http' ? http : https;
const req = requestModule.request(params, res => {
// grab request status
const statusCode = res.statusCode;
if(statusCode < 200 || statusCode > 299) {
throw new Error('Request Failed with Status Code:', statusCode);
}
let body = '';
// continuosly update data with incoming data
res.setEncoding('utf8');
res.on('data', data => body += data);
// once all data was received
res.on('end', () => resolve(body));
})
// write data to a post request
if(typeof(params.method) === 'string' && params.method === 'POST' && postData) {
req.write(postData)
}
// bind to the error event
req.on('error', err => reject(err));
// end the request
req.end();
})
}
const controller = async () => {
const apiKey = "00000000";
const options = {
hostname: '***"
port: 80,
path: '***'
method: 'POST',
headers: {"content-type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW"}
}
const postData = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"key\"\r\n\r\00000000\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
let result = await httpPromise('http', options, postData)
console.log(result);
}
yeah, so somehow it just doesn't seem to recognise the key in the postData string. I have tried various different combinations but just can't seem to get this to work.
The default http and https libraries are kind of wordy and annoying.
Would recommend using the request library instead. Read more here
In which case, to make the request, you can simply write it as :
var request = require('request');
var formData = {
// Pass a simple key-value pair
my_field: 'my_value',
}
request.post({url:'http://service.com/upload', formData: formData}, (err, response, body) => {
// Handle response here
});
Alright, so for anyone who might also face the same issue, it took me a little but figured out what the issue was. I didn't set the Content-Length header, which then in turn meant that node automatically added the Transfer-Encoding Header and set its value to chunk. This broke the receiving api and resulted in the issue. Setting the Content-Length header to the correct length and setting the Transfer-Encoding Header to an empty string solved my issue here (but I think one could also simply omit the transfer-encoding header once you defined the Content-Length Header).

Post call from Nodejs

I am trying to get an authentication token from an API.
the request is supposed to look like
POST /oauth2/token HTTP/1.1
Host: mysageone.ca.sageone.com
client_id=4b64axxxxxxxxxx00710&
client_secret=iNumzTxxxxxxxxxxhVHstrqWesH8tm9&
code=12a0f9c12cxxxxxxxxxxxxxxx92a48cc1f237ead&
grant_type=authorization_code&
redirect_uri=https://myapp.com/auth/callback
My current code keeps giving me status 400. I have tried to modify the headers but it doesn't work. i have also tried to make the required parameters part of the path using ?.
const http = require('http');
var options = {
hostname: 'app.sageone.com',
path: '/oauth2/token',
method: 'POST',
headers: {
"client_id":"xxxxx",
"client_secret":"xxxxx",
"code":"xxxxxx",
"grant_type":"authorization_code",
"redirect_uri":"https://some link"
}
};
console.log('in users file point 2');
var req1 = http.request(options, (res1) => {
console.log('statusCode:', res1.statusCode);
console.log('headers:', res1.headers);
console.log('message',res1.statusMessage);
res1.on('data', (d) => {
res.json(d);
});
});
req1.on('error', (e) => {
console.error('error starts here',e);
});
req1.end();
});
Looks to me like your problem is not with Node.js, but with your use of Sage One's api. This is the relevant documentation that might solve your problem.
From a quick glance it looks like you want to send a GET not a POST, and you should send those parameters in the URL. Here is the example URL they give:
https://www.sageone.com/oauth2/auth?response_type=code&client_id=4b64axxxxxxxxxx00710&redirect_uri=https://myapp.com/auth/callback


&scope=full_access
I've never used Sage One before, but that would match my experience with other OAuth APIs.

Streaming a Continous HTTP Response through Node

I am trying to act as a proxy between a client and an IP Camera using a NodeJS server. When I request a real-time stream from the camera, it responds with
HTTP/1.0 200 OK
Content-Type: Application/octet-stream
followed by a continuous stream of data. If I open the camera stream in Chrome it initiates a never ending download and curling it also initiates a continuous response.
Node appears to be buffering the response from the camera and parsing it through its HTTP parser each time. This works fine the first time as it has the correct headers but upon the second buffer of data it errors with
HPE_INVALID_HEADER_TOKEN
Can someone please help explain why this is happening? It's a continuous stream of data, why is it trying to parse the HTTP headers on the second buffer? I am not sure whether there is an option I am missing or my camera is not following the HTTP specification properly.
Edit: Example Code
const options = {
family: 4,
headers: {
Authorization: 'Basic ' + base64EncodedAuth,
},
host: '192.168.1.131',
method: 'GET',
path: '/cgi-bin/realmonitor.cgi?action=getStream&channel=1&subtype=0',
port: 80,
protocol:'http:',
};
const req = http.request(options, (res) => {
console.log(`STATUS: ${res.statusCode}`);
console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
res.on('data', (chunk) => {
console.log(`BODY: ${chunk}`);
});
res.on('end', () => {
console.log('No more data in response.');
});
});
req.on('error', (e) => {
console.log(`problem with request: ${e.message}`);
});
req.end();
The only callback that is hit is the 'error' one.
I further examined the curl log from the camera and noticed that everything was being marked as:
<= Recv header
It is never sending the separate CRLF required by the HTTP specification to signal that all the headers have been sent. That is why the parser was trying to parse it as a header and, quite rightly, throwing an error.

408 Timeout in NodeJS app requesting Github API

Following the documentation of the Github API to create an authorization for a NodeJS app.
I have the following code:
var _options = {
headers: {
'User-Agent': app.get('ORGANISATION')
},
hostname: 'api.github.com'
};
var oauth2Authorize = function () {
var path = '/authorizations?scopes=repo';
path += '&client_id='+ app.get('GITHUB_CLIENT_ID');
path += '&client_secret='+ app.get('GITHUB_CLIENT_SECRET');
path += '&note=ReviewerAssistant';
_options.path = path;
_options.method = 'POST';
var request = https.request(_options, function (response) {
var data = "";
response.on('data', function (chunk) {
data += chunk;
});
response.on('end', function () {
console.log(data);
});
});
request.on('error', function (error) {
console.log('Problem with request: '+ error);
});
};
And all I get is:
408 Request Time-out
Your browser didn't send a complete request in time.
Doing a GET request works though.
http.request() doesn't immediately send the request:
With http.request() one must always call req.end() to signify that you're done with the request - even if there is no data being written to the request body.
It opens the underlying connection to the server, but leaves the request incomplete so that a body/message can be sent with it:
var request = http.request({ method: 'POST', ... });
request.write('data\n');
request.write('data\n');
request.end();
And, regardless of whether there's anything to write() or not, you must call end() to complete the request and send it in its entirety. Without that, the server will eventually force the open connection to close. In this case, with a 408 response.

Resources