Random Fetch Error when I restart my program - node.js

I'm looping through this array of names and using each name to change the end point in the URI. The data from the end points is then stored in a Promise.all() array. I then run the array through a for loop and outputs the data into an EJS file. This works sometimes. However sometimes the console logs the error at the bottom of this post. When I increase the amount of endpoints i'm requesting this error happens everytime. The error appears to be very random and the error stops at random endpoints. I'm assuming i'm maybe sending to many requests? I eventually want to loop through around 5000 endpoints in this code but i'm struggling just loop through 20. Any help is much appreciated.
body: new URLSearchParams({ grant_type: 'client_credentials' }),
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Basic ' + Buffer.from(appId + ':' + appSecret).toString('base64')
}
})
.then(response => response.json())
.then(token2 => secondtoken(token2))
function secondtoken(token2){
let accesstoken = token2.access_token;
const promises = [];
for(var i = 0; i < 20;i++){
promises.push(fetch('https://us.api.blizzard.com/profile/wow/character/'+ slug[i]+'/'+ lowercasenames[i] +'?namespace=profile-us&locale=en_US', {
method: 'GET',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${accesstoken}`
}
})
.then(response => response.json()))
}
Promise.all(promises)
.then(data => { profiledata(data)})
};
function profiledata(data){
const name = []
for(var i = 0; i < data.length;i++){
name.push(data[i].name)
};
console.log(name)
res.render('index', {
leaderboard: diomerda,
rank: rank,
rating: rating,
slug: slug,
faction: faction,
statsplayed: statsplayed,
statswon: statswon,
statslost: statslost,
race: racename,
name: name,
})
}}});```
The error is below:
D:\Coding\Form 1\node_modules\node-fetch\lib\index.js:1461
reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
^
FetchError: request to https://us.api.blizzard.com/profile/wow/character/area-52/toonah?namespace=profile-us&locale=en_US failed, reason: read ECONNRESET
at ClientRequest.<anonymous> (D:\Coding\Form 1\node_modules\node-fetch\lib\index.js:1461:11)
at ClientRequest.emit (node:events:520:28)
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:520:28)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET'
}

Related

Use google calendar API and Getting 401 error message with Axios get

When developing the Alexa skill. I'm trying to use Axios get to get the whole day's events information from google calendar api. When I redo the account linking, it works well. I thought the asscessToken could only last for 2 hours, and I get the 401 error message because the token is expired.
Here is my Axios code
const getEvents = async (url, accessToken) => {
try {
const config = {
headers: {'Authorization': 'Bearer ' + accessToken}
};
return await Axios.get(url, config);
} catch (error) {
console.log('Error getting events');
console.error(error);
}
}
Code for my url and the asscesstoken
const url = EVENTS_URL + "?orderBy=updated&timeMin=" + timeMin + "&timeMax="
+ timeMax;
const accessToken =
handlerInput.requestEnvelope.context.System.user.accessToken;
Error Message:
ERROR { Error: Request failed with status code 401
at createError (/var/task/node_modules/axios/lib/core/createError.js:16:15)
at settle (/var/task/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/var/task/node_modules/axios/lib/adapters/http.js:260:11)
at IncomingMessage.emit (events.js:203:15)
at IncomingMessage.EventEmitter.emit (domain.js:448:20)
at endReadableNT (_stream_readable.js:1145:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
method: 'get',
headers:
{ Accept: 'application/json, text/plain, */*',
Authorization: 'Bearer undefined',
'User-Agent': 'axios/0.21.1' },

Securing Cloud Functions using IAM + NodeJS application running on AppEngine

I need some help with example if possible of code changes to be done on my NodeJS server code running on AppEngine in order to allow it to securely access my cloud fucntions.
I created a cloud function and I have a public URL for it. I then went ahead and removed allUser access in the Permissions' tab of the function. Under Service account I have App Engine Default Service account` selected.
My server on AppEngine was calling the public URL of the CF and when `allUser' permission was there, everything was hunky dory. But when it was removed, I started getting 403 error.
I need help with code changes on NodeJS side to be able to invoke the cloud fucntion again please.
My CF and App engine are in the same project and in the same region.
My server code is as follows using an https library to make the post request.
const checkingData = JSON.stringify({
'check' : 123
})
const checkingOptions = {
hostname: MY_CLOUD_PUBLIC_URL,
port: 443,
timeout: 5000,
path: MY_CLOUD_URL_PATH,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': checkingData.length,
}
}
const checkRequest = https.request(checkingOptions, checkD => {
if (checkD.statusCode==200) { // OK?
checkD.on('data', d => {
// Do something useful with the data
})
})
checkRequest.write(checkingData);
checkRequest.end();
I have been trying to find an answer for this for days now and started having migranes because of this. Any help with example code would be much appreciated. Many thanks!
EDIT
I can see the token now! (Yayyy!)
I changed my checkingOptions to work with request-promise in the following way:
const checkingOptions = {
uri: 'https://'+process.env.CLOUD_URL+process.env.CHECK,
port: 443,
timeout: 5000,
body:checkingData,
json: true,
// path: process.env.CHECK,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': checkingData.length,
}
}
I am getting the following errors:
Unhandled rejection StatusCodeError: 401 - "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>401 Unauthorized</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Unauthorized</h1>\n<h2>Your client does not have permission to the requested URL <code>/check</code>.</h2>\n<h2></h2>\n</body></html>\n"
2021-02-27 19:38:01 default[20210227t192944] at new StatusCodeError (/workspace/node_modules/request-promise-core/lib/errors.js:32:15)
2021-02-27 19:38:01 default[20210227t192944] at Request.plumbing.callback (/workspace/node_modules/request-promise-core/lib/plumbing.js:104:33)
2021-02-27 19:38:01 default[20210227t192944] at Request.RP$callback [as _callback] (/workspace/node_modules/request-promise-core/lib/plumbing.js:46:31)
2021-02-27 19:38:01 default[20210227t192944] at Request.self.callback (/workspace/node_modules/request/request.js:185:22)
2021-02-27 19:38:01 default[20210227t192944] at Request.emit (events.js:314:20)
2021-02-27 19:38:01 default[20210227t192944] at Request.<anonymous> (/workspace/node_modules/request/request.js:1154:10)
2021-02-27 19:38:01 default[20210227t192944] at Request.emit (events.js:314:20)
2021-02-27 19:38:01 default[20210227t192944] at IncomingMessage.<anonymous> (/workspace/node_modules/request/request.js:1076:12)
2021-02-27 19:38:01 default[20210227t192944] at IncomingMessage.emit (events.js:326:22)
2021-02-27 19:38:01 default[20210227t192944] at endReadableNT (_stream_readable.js:1241:12)
Do I need a middleware in my cloud function as well? My cloud function looks like the follow:
exports.check = functions.https.onRequest((req, res) => {
console.log('----CHECKING----');
res.status(200).send('Hello from checking');
})
IAM ROLES:
CLOUD FUNCTION > PERMISSIONS TAB
You have a great example in the documentation. If you want I adapt it to your code, it could look like to this
// Make sure to `npm install --save request-promise` or add the dependency to your package.json
const request = require('request-promise');
// Set up metadata server request
// See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#request_signature
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
const tokenRequestOptions = {
uri: metadataServerTokenURL + MY_CLOUD_PUBLIC_URL,
headers: {
'Metadata-Flavor': 'Google'
}
};
const checkingData = JSON.stringify({
'check' : 123
})
const checkingOptions = {
uri: MY_CLOUD_PUBLIC_URL, //!!!changed here!!!
port: 443,
timeout: 5000,
path: MY_CLOUD_URL_PATH,
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': checkingData.length,
}
}
request(tokenRequestOptions).then((token) => {
request(checkingOptions).auth(null, null, true, token).then((response) => { //!!!changed here!!!
checkD.on('data', d => {
// Do something useful with the data
})
})
.catch((error) => {
res.status(400).send(error);
}); => {
})
checkRequest.write(checkingData);
checkRequest.end();
Don't forget to add the role role/cloudfunctions.invoker to the App Engine default service account, either at the project level or at the Cloud Functions level.
EDIT 1
Correct. request-promise has been deprecated for year. What alternative? I absolutely don't know because NodeJS hurts me (as I said in the comment).
As a lazy man, I found an alternative. I found the request-promise in the Cloud Run documentation. But you have to know that Cloud Run and Cloud Functions are very closed (they share the same underlying platform). I take my chance with Cloud Functions authentication documentation and bingo! There is an example with node-fetch
const fetch = require('node-fetch');
const MY_CLOUD_PUBLIC_URL = '....'
const metadataServerTokenURL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=';
// Fetch the token
const tokenResponse = await fetch(metadataServerTokenURL + MY_CLOUD_PUBLIC_URL, {
headers: {
'Metadata-Flavor': 'Google',
},
});
const token = await tokenResponse.text();
console.log(token)
const checkingData = JSON.stringify({
'check' : 123
})
// Provide the token in the request to the receiving function
try {
const functionResponse = await fetch(MY_CLOUD_PUBLIC_URL, {
method: 'post',
body: checkingData,
headers: {
'Content-Type': 'application/json',
Authorization: `bearer ${token}`},
});
console.log(await functionResponse.text());
} catch (err) {
console.error(err);
}
Add your check functions and it should work now!

Reading big .txt and checking if string contains on response body

I have this code:
var fs = require('fs');
var readline = require('readline');
var request = require("request");
var filename = process.argv[2];
readline.createInterface({
input: fs.createReadStream(filename),
terminal: false
}).on('line', function(line) {
sep = line.split("|");
id = sep[0];
name = sep[1];
var options = { method: 'POST',
url: 'https://api.com/mobile_authentications',
proxy: '...',
headers:
{ Host: 'api.com',
'Content-Type': 'application/json',
'Accept-Language': 'pt-BR',
'X-Client-Info': 'Timezone%3D-Infinity%26device_id%3D578e4ac016b69248%26utc_offset%3D-0300%26app_id%3D7092%26resolution%3D720x1280%26connectivity%3DWIFI%26site_id%3DMLB%26density%3D320%26user_id%3Dguest' },
body: '{"authentication_request":{"client_id":"'+id+'|'+name+'"}}' };
request(options, function (error, response, body) {
if (error) throw new Error(error);
if(body.includes('ATTEMPS'))
{
console.log("CAPTCHA");
}
//console.log(body);
});
});
First question: How can I check if body response contains "ATTEMPS" string?
Another problem: When the ids.txt file has more than 100 lines the script returns me this error:
if (error) throw new Error(error);
^
Error: Error: tunneling socket could not be established, statusCode=502
at Request._callback (C:\Users\PC\Desktop\top\convert.js:65:20)
at self.callback (C:\Users\PC\Desktop\top\node_modules\request\request.js:185:22)
at Request.emit (events.js:182:13)
at Request.onRequestError (C:\Users\PC\Desktop\top\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.onConnect (C:\Users\PC\Desktop\top\node_modules\tunnel-agent\index.js:168:23)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at Socket.socketOnData (_http_client.js:475:11)
at Socket.emit (events.js:182:13)
That is, the script is not able to work with large files?
Thanks.

Error: socket hang up in Http Request made in Node JS using request-promise causing for loop to restart

I am trying to make an Http Request using request-promise inside a for loop. But it seems if a Http Request takes long, request-promise closes the connection.
This behavior is ok but what I am not able to grasp is the for loop starts from 0 again after the error is printed.
Below is the code
const rp = require('request-promise');
async function stepIterator(processingSteps, documentId) {
var finalResult = null;
for (var step = 0, len = processingSteps.length; step < len; step++) {
if (step === 0 || step === 1 || step == 2 || step == 3) {
try {
console.log('Calling step ', step);
let url = 'http://internal-server:8080/process';
let collection = getCollection(documentId);
let splitText = getSPlit(documentId);
let outputFormat = 'xmi';
let documentObject = await callServer(url, collection, splitText, outputFormat);
finalResult = documentObject;
} catch (error) {
console.log("Error");
}
}
}
return finalResult;
}
async function callServer(url, collection, splitText, outputFormat) {
var options = {
method: 'POST',
uri: url,
headers: {
'Content-Type': 'multipart/form-data',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive '
},
formData: {
collection: collection,
text: splitText,
output: outputFormat
}
};
return rp(options)
}
The complete error trace is as follows
{ RequestError: Error: socket hang up
at new RequestError (D:\New_Projects\new-data-access-layer\node_modules\request-promise-core\lib\errors.js:14:15)
at Request.plumbing.callback (D:\New_Projects\new-data-access-layer\node_modules\request-promise-core\lib\plumbing.js:87:29)
at Request.RP$callback [as _callback] (D:\New_Projects\new-data-access-layer\node_modules\request-promise-core\lib\plumbing.js:46:31)
at self.callback (D:\New_Projects\new-data-access-layer\node_modules\request\request.js:185:22)
at Request.emit (events.js:182:13)
at Request.onRequestError (D:\New_Projects\new-data-access-layer\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:182:13)
at Socket.socketOnEnd (_http_client.js:425:9)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1094:12)
at process._tickCallback (internal/process/next_tick.js:63:19) name: 'RequestError', message: 'Error: socket hang up', cause:
{ Error: socket hang up
at createHangUpError (_http_client.js:322:15)
at Socket.socketOnEnd (_http_client.js:425:23)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1094:12)
at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }, error: { Error: socket hang up
at createHangUpError (_http_client.js:322:15)
at Socket.socketOnEnd (_http_client.js:425:23)
at Socket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1094:12)
at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }, options: { method: 'POST',
uri: 'http://internal-server:8080/process',
json: true,
headers: { Connection: 'keep-alive ' },
body:
{ docSplitId: [Array],
_id: 5c579d84812acb17ec74ac39,
contentType: 'application/pdf',
location:
'C:\\Users\\newuser\\AppData\\Local\\Temp\\2\\report.pdf',
docModelVersion: '1',
visualMetaDataId: null,
categoryId: '5c52a72f6df294140c0535bc',
deductedInfo: null,
status: 'New',
isDeleted: false,
metadata: [Object],
detailedStatus: [Array] },
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false }, response: undefined }
Obviously the socket is hanging! Don't bother with http, it is a little complex. Use node unirest and it closes the data stream.
var unirest = require('unirest');
var req = unirest('POST', 'localhost:3200/store/artifact/metamodel')
.attach('file', '/home/arsene/DB.ecore')
.field('description', 'We are trying to save the metamodel')
.field('project', '6256d72a81c4b80ccfc1768b')
.end(function (res) {
if (res.error) throw new Error(res.error);
console.log(res.raw_body);
});
Hope this helps!

If I disable NODE_TLS_REJECT_UNAUTHORIZED, my request is still denied

I am disabling Node from rejecting self signed certificates and making a request.
const { USER, PW } = process.env;
const b64 = new Buffer(`${VP_API_USER}:${VP_API_PW}`).toString("base64");
const Authorization = `Basic ${b64}`;
const doFind = async url => {
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0;
const results = await fetch(url, { headers: { Authorization } })
.then(r => (r.ok ? r.json() : Promise.reject(r)))
.catch(err => {
return Promise.reject(err);
});
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 1;
return results;
};
I am still being rejected.
{ FetchError: request to https://<url>:55544/contracts failed, reason: connect ECONNREFUSED <url>:55544
at ClientRequest.<anonymous> (/Users/mjhamm75/Developer/sedd-monorepo/node_modules/node-fetch/index.js:133:11)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
name: 'FetchError',
message: 'request to https://<url>:55544/contracts failed, reason: connect ECONNREFUSED <url>:55544',
type: 'system',
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED' }
What am I doing wrong?
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 1;
line should go inside the callback (your then or catch before the return. because a promise gets resolved in the callback, but your line
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 1;
is written outside of it, even though it appears after the statement, it runs immediately without waiting for the callback. so, your tls is effectively never disabled.
I hope this helps:)
Previous answer looks incorrect - await postpones execution of next line until promise will be resolved.
According to the documentation the NODE_TLS_REJECT_UNAUTHORIZED value should be string '0' to disable TLS validation.
This is how I would approach it, if I had to reset the env var afterwards.
Using .finally() the statement will execute regardless of the outcome of the fetch.
const doFind = async url => {
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0;
const results = await fetch(url, { headers: { Authorization } })
.then(r => (r.ok ? r.json() : Promise.reject(r)))
.catch(err => {
return Promise.reject(err);
})
.finally(() => {
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 1;
});
return results;
};

Resources