So I have an rss feed url, that has around 200 items. However, request module is only able to fetch at random 15 - 20 items, and it ends in the middle of receiving the response without giving any error.
I tried with regular curl, and I am receiving all the data, any reason for this descrepancy?
const { request } = require('https');
const url = //some url;
const req = request(url, (res) => {
console.log(res.statusCode)
let data = "";
res.on('data', (d) => {
data += d;
})
res.on('end', () => {
console.log('data ---', data);
})
})
req.end();
I tried with request npm package, and it's still the same
Related
I am trying to write a function that creates an HTTPS request.
This is all part of an expressjs project with Typescript.
I can get the HTTPS request working and getting a response - but the response is encoded with GZIP. I am trying my best to follow the documentation. But no luck, the response stays zipped.
This is my code
private getData = (host, pathname): Promise<string> => {
return new Promise((resolve, reject) => {
const options = {
hostname: host,
path: pathname,
gzip: true,
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
const req = https.request(options, (res) => {
if (res.statusCode < 200 || res.statusCode >= 300) {
console.log('error!')
return reject(new Error('statusCode=' + res.statusCode));
}
let body = '';
res.on('data', function (chunk) {
body += chunk;
});
res.on('end', function () {
try {
console.log('res.headers', res.headers)
console.log('res.headers', body)
body = JSON.parse.toString();
resolve(body);
} catch (e) {
reject(e);
}
resolve(body);
});
req.on('error', (e) => {
reject(e.message);
});
// send the request
});
req.end();
});
}
The console log statement shows this
So obviously the JSONparsing is failing. What am I missing?
Thanks heaps for the help
https.request() does not natively support gzip compression. So, you will have to either add support for it yourself as shown here or use an https request library such as got() that already supports gzip (and also already supports promises too).
import got from 'got';
private getData = (host, pathname): Promise<string> => {
const options = {
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
return got(`https://${host}${pathname}`, options).json();
}
FYI, it's not clear in your question what type of response you are expecting. You attempt to use:
body = JSON.parse.toString();
which doesn't make any sense because JSON.parse.toString() is trying to get string version of the JSON.parse function without actually even calling that function - very odd. Perhaps you meant JSON.parse(body)? But, your typescript makes it look like you're expecting a promise that resolves to a string so that wouldn't usually be the result of JSON parsing.
If you are expecting a gzipped JSON response, then you would use the:
return got(`https://${host}${pathname}`, options).json();
I show above. If you are expecting just a string, not JSON, then you would change that line to:
return got(`https://${host}${pathname}`, options).text();
Got a 403 signature error , when using the below fetch function:
function elasticsearchFetch(AWS, elasticsearchDomain, endpointPath, options = {}, region = process.env.AWS_REGION) {
return new Promise((resolve, reject) => {
const { body, method = 'GET' } = options;
const endpoint = new AWS.Endpoint(elasticsearchDomain);
const request = new AWS.HttpRequest(endpoint, region);
request.method = method;
request.path += endpointPath;
request.headers.host = elasticsearchDomain;
if (body) {
request.body = body;
request.headers['Content-Type'] = 'application/json';
request.headers['Content-Length'] = request.body.length;
}
const credentials = new AWS.EnvironmentCredentials('AWS');
const signer = new AWS.Signers.V4(request, 'es');
signer.addAuthorization(credentials, new Date());
const client = new AWS.HttpClient();
client.handleRequest(request, null, (res) => {
let chunks = '';
res.on('data', (chunk) => {
chunks += chunk;
});
res.on('end', () => {
if (res.statusCode !== 201) console.log('Got these options STATUSCODE', JSON.stringify(options, false, 2));
return resolve({ statusCode: res.statusCode, body: chunks });
});
}, (error) => {
console.log('Got these options ERROR', JSON.stringify(options, false, 2));
return reject(error);
});
});
}
This is the options used for the request in above function :
{
"method": "POST",
"body": "{\"prefix\":\"image_233/ArtService/articles-0/GB/ART-60297885/\",\"id\":\"ART-60297885\",\"retailUnit\":\"GB\",\"commercial\":{\"name\":{\"en-GB\":\"FÖRBÄTTRA\"}},\"schemaType\":\"product\",\"productType\":\"ART\"}"
}
and got this error :
{
"statusCode": 403,
"body": "{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}"
}
This is the endpoint : 233/_doc/
I believe your Content-Length header is incorrect, causing the signature mismatch.
Your payload includes the string FÖRBÄTTRA, which has two double-byte characters.
You're setting the Content-Length to request.body.length, which comes to 186.
While this is the number of characters in the body, it is not the number of bytes in the body (188).
To calculate the Content-Length, use Buffer.byteLength(request.body). For a POST request like this, you can even remove that line of code altogether, and the request will succeed.
// Content-Length is only needed for DELETE requests that include a request
// body, but including it for all requests doesn't seem to hurt anything.
request.headers['Content-Length'] = Buffer.byteLength(request.body);
Source: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html#es-request-signing-node
By the way, why not use elasticsearch client for nodejs to communicate with elasticsearch rather than writing your own logic. You can consider using http-aws-es which does the request signing part for you. The code will look like
const { Client } = require("elasticsearch");
const esConnectionClass = require("http-aws-es");
const elasticsearchConfig = {
host: "somePath",
connectionClass: esConnectionClass
};
const nativeClient = new Client(elasticsearchConfig);
const result = await nativeClient.search({});
Basically I want to test 500+ live website URLs and get the statusCode of them using protractor and getting help from node's http module.
But I'm not getting anything in response and all console statements related to the http request are ignored in my terminal.
I've tried using some external libraries such as axios and protractor-http-client but all are giving me same result.
I'm using TypeScript.
import request from 'request' // tried this one earlier instead of http
import https from 'https';
async testFunction () {
const req = await https.get('https://www.google.com');
let body = '';
req.on('data',(d) => {
body += d;
});
req.on('end',(resp) => {
console.log(resp.statusCode); // Not printing in terminal at all
body = JSON.parse(body);
});
req.on('response', (resp) => { // also tried information instead of response
console.log(resp.statusCode); // Not printing in terminal at all
});
console.log(body); // printing as empty
}
}
I would expect the response status code be 200.
I'm new to both protractor and node. So any kind of help and suggestion would be appreciated.
const http = require('http');
const res = http.request({
host: 'google.de',
method: 'get',
path: '/'
}, (response) => {
let dataStr = [];
const receiveData = chunk => {
const b = Buffer.from(chunk);
dataStr.push(b);
};
response.on('end', () => {
console.log('status Code:', response.statusCode);
console.log('response:', Buffer.concat(dataStr).toString());
});
response.on('error', (e) => console.error(e));
response.on('data', receiveData);
});
res.end();
//route to search (POST http://localhost:8080/api/search)
apiRoutes.post('/search', function(req, res) {
console.log('search');
var query = req.params;
console.log(query);
options = {
protocol : "https:/",
host: "https://api.themoviedb.org",
path: "/3/search/movie?api_key=35f7a26be584f96e6b93e68dc3b2eabd&language=en-US&page=1&include_adult=false&query="+query,
};
var req = https.request(options, function(res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function () {
var body = Buffer.concat(chunks);
console.log(body.toString());
});
});
req.write("{}");
req.end();
})
DOES ANYONE KNOW WHERE THE PROBLEM IS?
I'm trying to do a request to do a research to the api the movie db and get the result back
There are some problems with the code. I have tested it and made it to work.
let options = {
host: "api.themoviedb.org",
path: "/3/search/movie?api_key=35f7a26be584f96e6b93e68dc3b2eabd&language=en-US&page=1&include_adult=false&query="+query.data.replace(' ','%20'),
};
first of all since you are using https module you don't need to specify the protocol nor you need to put it in the url. That's how your options variable should be.
Second you are appending the entire query object to the url which is {} instead you should append a string which will be in one of the key of your query object in my case its query.data
Third if there are spaces in the string Eg: Home Alone you to maintain space and avoid the error we replace the string with %20 which is a escaping character.
Forth Try giving a unique name for https request variable and its response variable in the callback function or it will override the route's req res variables cause your code to not work. Notice how I have used route's res function to send the data back and end the response
Also I am getting the data in req.body and you are using req.params however there are no params defined in your routes. Try going through the documentation for more information
Here is the complete code
apiRoutes.post('/search',function (req, res) {
https = require('https');
var query = req.body;
console.log(query.data);
let options = {
host: "api.themoviedb.org",
path: "/3/search/movie?api_key=35f7a26be584f96e6b93e68dc3b2eabd&language=en-US&page=1&include_adult=false&query="+query.data.replace(' ','%20'),
};
var request = https.request(options, function(response) {
var chunks = [];
response.on("data", function (chunk) {
chunks.push(chunk);
});
response.on("end", function () {
var body = Buffer.concat(chunks);
console.log(body.toString());
res.send(body);
res.end()
});
});
request.end();
});
Hope it helps.
I'm trying to make a chain of promises functions which use HTTP requests in NodeJS with Kraken framework.
My code could work in 90% of cases, but if the distant requested server takes time to respond, the code will return an error with undefined values. So I think Q is a good solution to prevent that.
Here's the situation :
We access to a URL with a "code" parameter -> the route controller takes this param to use it in a HTTP POST request -> the response (a token) is stored in a variable and used in an other HTTP GET request -> the response (multiple JSON objects) is stored in variable too -> all variables are stored in a MongoDB.
If functions are not used in this order, of course it fails.
var Q = require('q');
module.exports = function (router) {
router.get('/', function (req, res) {
var codein = req.param('code');
if(codein){
console.log('Provided code: ' + codein+'\n');
getAccessToken(codein).then(function(token){
console.log('Provided AccessToken: ' + token + '\n');
getUsername(token).then(function(userdata){
console.log('Provided Username: ' + JSON.parse(userdata).username + '\n');
storeData(userdata).then(function(msg){
console.log(msg);
res.redirect('/dashboard/' + JSON.parse(userdata).username);
});
});
});
}
else{
console.log('Access Denied, redirecting...');
res.redirect('/');
}
});
};
This method works, but actually didn't resolve the problem, because sometimes variable are undefined again. I think it's my request functions which aren't well made...
Here's an example of the first function with POST request :
var getAccessToken = function(cod){
var def = Q.defer();
var data = querystring.stringify({
client_id:"1234567890",
client_secret:"******",
grant_type:"authorization_code",
redirect_uri:"http://localhost:8000/r/callback",
code:cod
});
var options = {
host: 'domain.server.com',
port: 443,
path: '/api/oauth2/token',
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': Buffer.byteLength(data)
}
};
var response = "";
var req = https.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
response += chunk;
});
res.on('end', function(){
var json = JSON.parse(response);
var acto = json.access_token;
def.resolve(acto);
});
});
req.write(data);
req.end();
return def.promise;
};
In this case the acto variable can be undefined... So am I using Q in a wrong way ?
EDIT
To understand my problem, let me show you what can I have in my output console (really rare but happens) :
Provided code: 12345678910
Provided Username: user543210
Instead of :
Provided code: 12345678910
Provided AccessToken: 9876543210
Provided Username: user
I think you need to account for 2 scenarios
Where the Twitch API takes time to respond.
The Twitch response cannot be parsed
The code
res.on('end', function(){
var json = JSON.parse(response);
var acto = json.access_token;
def.resolve(acto);
});
Should be modified as:
try {
var json = JSON.parse(response);
var acto = json.access_token;
//check if acto is undefined
if (acto === undefined) {
def.reject('Some error message');
} else {
def.resolve(acto);
}
} catch (error) {
//since the JSON could not be parse
def.reject(error);
}