Unable to send request from request callback - node.js

I cant execute a second request inside the request callback.
request({
url: url,
headers: {
'auth-token': token
},
method: 'POST',
json: true,
body: data
}, (err, req, body) => {
if (err) {
console.log(err);
} else {
// Prosses data;
// This is the second request.
request({
url: url2,
headers; {
'auth-token': token
},
method: 'POST',
json: true,
body: data2
}, (err, req, body) => {
if (err) {
console.log(err);
return;
}
//Process data.
})
}
})
The problem is that the second request is not executing.
I am using nodemon to start the express server, but on the nodemon only the first request is receive on the express.
But the strange thing is that when I tried to call the method on the second time without closing the electron app, the second request is executed. And I can see it on the nodemon that the second request is executed first.
The output of the nodemon is like this.
POST /path/to/url 200 6.181 ms - 641 //-> this is the first execution. then there is nothing follows.
// Then I call the method again using the app. And the result is this.
POST /path/to/second/url 200 9.645 ms - 21
POST /path/to/url 200 21.628 - 641
It look like the /path/to/second/url is staying on somewhere nowhere and just send to the server if the method is called for the second time.
Please help, thanks.
Update: Added more info.
I have a folder could routes all the .js file is save there.
then I am loading it using this on the my app.js
let rs = fs.readdirSync(path.join(process.cwd(), '/routes/'));
rs.forEach((file) => {
if (file.indexOf('.js') !== -1) {
let fn = '/' + file.replace(/\.[^/.]+$/, '');
let pt = path.join(__dirname, './routes', fn);
let req = require(pt);
app.use(fn, req);
}
});
Then on the routes files I have something similar like this.
router.post('url', (req, res) => {
// here calling the controller.
// mostly just single line passing the request and result variable to the controller.
});
Actually that requests is called inside the ipc callback. I have a menuitems and on the click() event I just used the browserWindow.getFocusedWindow().webContents.send('ipc-name') then this will be triggered.
ipc.on('ipc-name', () => {
// The request is called here.
}

This does not solve the OP problem as the problem exists in Linux env but acts as an workaround.Instead of request module we have to make use of ClientRequest API in electron which is Event based and only makes the task much more difficult.But doesn't suffer from the issue faced in callback inside callback.
Example Code:
function triggerCall() {
const request = net.request(`${root}/routes/get1`);
request.on('response', (response) => {
response.on('data', (chunk) => {
console.log(`BODY: ${chunk}`)
})
response.on('end', () => {
console.log('req 1 end');
const request1 = net.request(`${root}/routes/get2`);
request1.on('response', (response) => {
response.on('data', (chunk) => {
console.log(`BODY: ${chunk}`)
})
response.on('end', () => {
console.log('req 2 end');
})
})
request1.end();
})
})
request.end();
};

Related

Node, how to send result back to client from nested HTTP request?

I'm using ReactJS to run my front-end and using Express for my back-end. I want to make a get request to my back-end using the "/paas" path to get a listing of all of my pods that are running inside my namespace in Rancher(Kubernetes).
The back-end then needs to be able to make an https request to my Rancher API endpoint and return the result to the front-end. I can make the successful call to Rancher API and see the data print to the screen on my back-end but I get lost when trying to send this data to the front-end and console log it out inside the browser.
Due to "pre-flight" errors, I can't just make a direct call to the Rancher endpoint inside of my App.js file. More info on this here. So I need to go the custom back-end route. I any case, it seems like this should be pretty straightforward. Any guidance would be appreciated.
App.js:
import React, { useEffect } from "react"
import axios from "axios"
function App() {
useEffect(() => {
const fecthPods = async () => {
try {
const response = await axios.get(`http://localhost:3001/paas`)
console.log(response.data)
} catch (err) {
if (err.response) {
// Not in the 200 response range
console.log(err.response.data)
console.log(err.response.status)
console.log(err.response.headers)
} else {
console.log(`Error: ${err.message}`)
}
}
}
fecthPods()
},[])
return (
<div>
Hello World!
</div>
);
}
export default App;
Back-end server.js:
import express from "express"
import cors from "cors"
import https from "https"
import bodyParser from "body-parser";
const app = express()
app.use(cors())
app.use("/data", (req, res) => {
res.json({ name: "Minion", favFood: "pizza"})
})
app.get("/paas", bodyParser.json(), (req, res) => {
const options = {
hostname: "k8.fqdn.com",
port: 443,
path: "/k8s/clusters/c-wwfc/v1/pods/mynamespace",
method: "GET",
headers: {
Authorization: "Bearer token:12345"
}
}
const request = https.get(options, (res) => {
let responseBody = ""
res.setEncoding("UTF-8")
res.on("data", (chunk) => {
console.log("---chunk", chunk.length);
responseBody += chunk;
});
res.on("end", () => {
let json = JSON.parse(responseBody)
// console.log(responseBody)
console.log("Response finished");
res.json({data: responseBody})
});
});
request.end()
res.json({ status: "complete", data: request.data})
})
app.listen(3001)
console.log("backend up on 3001")
I see a couple of errors on your backend code.
First, you are naming the res variable for the express middleware and also for the response received by the https module. In this way, you lose the possibility to access to the express response object in the on.('end') callback.
Secondly, you are triyng to respond to the client multiple times (inside the on.('end') callback and also directly inside the express middleware with the instruction res.json({ status: "complete", data: request.data}). Also, consider that the code you wrote is repliyng to the client before the call to the k8s cluster is made. And the response will always be a JSON with this data: { "status": "complete", "data": undefined}.
To fix all, try with this code (I will try to comment all edits):
app.get("/paas", bodyParser.json(), (req, res) => {
const options = {
hostname: "k8.fqdn.com",
port: 443,
path: "/k8s/clusters/c-wwfc/v1/pods/mynamespace",
method: "GET",
headers: {
Authorization: "Bearer token:12345"
}
}
const k8sRequest = https.get(options, (k8sResponse ) => { // as you can see I renamed request and res to k8sRequest and k8sResponse, to avoid loosing the scope on req and res express middleware variables
let responseBody = ""
res.setEncoding("UTF-8")
k8sResponse.on("data", (chunk) => { // here use k8sResponse to collect chunks
console.log("---chunk", chunk.length);
responseBody += chunk;
});
k8sResponse.on("end", () => { // here use k8sResponse again
let json = JSON.parse(responseBody)
// console.log(responseBody)
console.log("Response finished");
res.json({ status: "complete", data: responseBody}) // here use the express res variable, to reply to the client.
});
});
k8sRequest.end() // here use the k8sRequest variable to make the https call to the k8s cluster
// here I deleted the res.json instruction
})
The above code should just works. Anyway, I suggest you using axios also with your backend service. You are already using it with React, so you know how to use it. The syntax is minimal and easier and you can use the async/await approach.
Axios solution:
import axios from "axios"
app.get("/paas", bodyParser.json(), async (req, res) => {
try {
const url = 'https://k8.fqdn.com/k8s/clusters/c-wwfc/v1/pods/mynamespace'
const k8sResponse = await axios.get(url, headers: {
Authorization: "Bearer token:12345"
})
res.json({ status: "complete", data: k8sResponse.data })
} catch (e) {
res.json({status: "error", data: e.response.data})
}
})
You should wrap your axios call inside a try/catch block to properly handle errors like you are doing with your React implementation. Error handling should be also implemented if you still want you the native node.js https module

I need help making an API request to an external API in nodejs express server (Angular Universal)

I have an Angular Universal app. I am trying to access an external API, but doing it directly through the HttpClient generates a Cors error. In development, I successfully used a proxy to make the call. I am trying to implement a proxy in production by creating a route on my express server that will swap in the appropriate external API route. I am having trouble seeing anything online that can help with this particular situation. I seem to have set up the route ok. I'm getting a 200 ok error but no data is being sent. Can anyone help?
server.ts
app.route('/api/book').get((req, res) => {
https.get('https://api2.isbndb.com/book/' + req, (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
res.send(res.json(data)
);
});
}).on("error", (err) => {
console.log("Error: " + err.message);
});
});
You can simply enable cors from your server side like this.
var express = require('express')
var cors = require('cors')
var app = express()
app.use(cors())
For better understanding or to configure cors() url through.
can see here.
As by default it will call the options method to check the permission for the user to access that end points.
or you can use below example from client side api call,
let data = { name: 'Peter Parker', age: 34 };
const results = await fetch(
'http://localhost:3000/api/v1/results',
{
method: "post",
mode: "cors",
headers: {
"Content-Type": "application/json",
"Access-Control-Allow-Origin": "*"
},
body: JSON.stringify(data)
}
)
.then(res => res.json())
.then(res => {
return res;
})
.catch(error => {
error.response = {
status: 0,
statusText:
"Cannot connect. Please make sure you are connected to internet."
};
throw error;
});

Awaiting http request in AWS Lambda / Pulumi

I have an AWS Lambda function which triggers https request to Google API. I want the function to be awaitable, so that it does not end immediately, but only after getting response from Google API.
Yes, I know I pay for the execution, but this will not be called often, so it is fine.
The problem is that the http request does not seem to fire correctly. The callback is never executed.
I have made sure that the async/await works as expected by using setTimeout in a Promise. So the issue is somewhere in the https.request.
Also note that I am using Pulumi to deploy to AWS, so there might be some hidden problem in there. I just can't figure out where.
The relevant code:
AWS Lambda which calls the Google API
import config from '../../config';
import { IUserInfo } from '../../interfaces';
const https = require('https');
function sendHttpsRequest(options: any): Promise<any> {
console.log(`sending request to ${options.host}`);
console.log(`Options are ${JSON.stringify(options)}`);
return new Promise(function (resolve, reject) {
console.log(` request to ${options.host} has been sent A`);
let body = new Array<Buffer>();
const request = https.request(options, function (res: any) {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
if (res.statusCode != 200) {
reject(res.statusCode);
}
res.on('data', (data: any) => {
console.log(`body length is ${body.length}`);
console.log('data arrived', data);
body.push(data);
console.log('pushed to array');
console.log(data.toString());
});
});
request.on('end', () => {
console.error('Request ended');
// at this point, `body` has the entire request body stored in it as a string
let result = Buffer.concat(body).toString();
resolve(result);
});
request.on('error', async (err: Error) => {
console.error('Errooooorrrr', err.stack);
console.error('Errooooorrrr request failed');
reject(err);
});
request.end();
console.log(` request to ${options.host} has been sent B`);
});
}
/**
* AWS Lambda to create new Google account in TopMonks domain
*/
export default async function googleLambdaImplementation(userInfo: IUserInfo) {
const payload = JSON.stringify({
"primaryEmail": userInfo.topmonksEmail,
"name": {
"givenName": userInfo.firstName,
"familyName": userInfo.lastName
},
"password": config.defaultPassword,
"changePasswordAtNextLogin": true
});
const resultResponse: Response = {
statusCode: 200,
body: 'Default response. This should not come back to users'
}
console.log('Calling google api via post request');
try {
const options = {
host: 'www.googleapis.com',
path: '/admin/directory/v1/users',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': payload.length.toString()
},
form: payload
}
const responseFromGoogle = await sendHttpsRequest(options);
console.log('responseFromGoogle', JSON.stringify(responseFromGoogle));
}
catch (err) {
console.log('Calling google api failed with error', err);
resultResponse.statusCode = 503;
resultResponse.body = `Error creating new Google Account for ${userInfo.topmonksEmail}.`;
return resultResponse;
}
console.log('request to google sent');
return resultResponse;
}
The problem is that the http request does not seem to fire correctly. The callback is never executed.
I believe this part of the issue is related to some combination of (a) potentially not actually sending the https request and (b) not using the correct callback signature for https.request. See the documentation at https://nodejs.org/api/https.html#https_https_request_options_callback for details on both of these.
Use node-fetch package
The following example works for me using node-fetch:
import * as aws from "#pulumi/aws";
import fetch from "node-fetch";
const api = new aws.apigateway.x.API("api", {
routes: [{
method: "GET", path: "/", eventHandler: async (ev) => {
const resp = await fetch("https://www.google.com");
const body = await resp.text();
return {
statusCode: resp.status,
body: body,
}
},
}],
})
export const url = api.url;
Pulumi complains, it something like "Can not serialize native function" or something like that. The problematic part is that node-fetch relies on Symbol.iterator
As noted in the comments, some of the conditions that can lead to this are documented at https://pulumi.io/reference/serializing-functions.html. However, I don't see any clear reason why this code would hit any of those limitations. There may be details of how this is used outside the context of the snippet shared above which lead to this.

Make CRUD request with Nodejs

I want to make ajax calls with nodeJS to an endpoint. I have done many "get" requests by doing
http.get('url', (res) => ....)
Now , I want to make "post, put and delete" request
For example, I would to like to do
http.post('url', 'body', (res) => ...)
http.put....
http.delete ...
When I did that http.post request, I obtain "TypeError: "listener" argument must be a function".
And this is my complete request
http.post(environment.url , req.body, (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
return res.status(200).send(JSON.parse(data))
});
}).on("error", (err) => {
res.status(404).send(err)
});
It is possible ? Many thanks
For some reason NodeJS don't seem to implement the other HTTP methods as their own methods in the http module. You'll have to use http.request and set the request method in the option.
In its most simple form, you should be able to write
var req = http.request({method: 'POST', path: url}, (res) => ...);
req.write(body);
req.end();
Source:
https://nodejs.org/api/http.html#http_http_request_options_callback

Why is this HTTP request not working on AWS Lambda?

I'm getting started with AWS Lambda and I'm trying to request an external service from my handler function. According to this answer, HTTP requests should work just fine, and I haven't found any documentation that says otherwise. (In fact, people have posted code that use the Twilio API to send SMS.)
My handler code is:
var http = require('http');
exports.handler = function(event, context) {
console.log('start request to ' + event.url)
http.get(event.url, function(res) {
console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
console.log("Got error: " + e.message);
});
console.log('end request to ' + event.url)
context.done(null);
}
and I see the following 4 lines in my CloudWatch logs:
2015-02-11 07:38:06 UTC START RequestId: eb19c89d-b1c0-11e4-bceb-d310b88d37e2
2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 start request to http://www.google.com
2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 end request to http://www.google.com
2015-02-11 07:38:06 UTC END RequestId: eb19c89d-b1c0-11e4-bceb-d310b88d37e2
I'd expect another line in there:
2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 Got response: 302
but that's missing. If I'm using the essential part without the handler wrapper in node on my local machine, the code works as expected.
The inputfile.txt I'm using is for the invoke-async call is this:
{
"url":"http://www.google.com"
}
It seems like the part of the handler code that does the request is skipped entirely. I started out with the request lib and fell back to using plain http to create a minimal example. I've also tried to request a URL of a service I control to check the logs and there's no requests coming in.
I'm totally stumped. Is there any reason Node and/or AWS Lambda would not execute the HTTP request?
Of course, I was misunderstanding the problem. As AWS themselves put it:
For those encountering nodejs for the first time in Lambda, a common
error is forgetting that callbacks execute asynchronously and calling
context.done() in the original handler when you really meant to wait
for another callback (such as an S3.PUT operation) to complete, forcing
the function to terminate with its work incomplete.
I was calling context.done way before any callbacks for the request fired, causing the termination of my function ahead of time.
The working code is this:
var http = require('http');
exports.handler = function(event, context) {
console.log('start request to ' + event.url)
http.get(event.url, function(res) {
console.log("Got response: " + res.statusCode);
context.succeed();
}).on('error', function(e) {
console.log("Got error: " + e.message);
context.done(null, 'FAILURE');
});
console.log('end request to ' + event.url);
}
Update: starting 2017 AWS has deprecated the old Nodejs 0.10 and only the newer 4.3 run-time is now available (old functions should be updated). This runtime introduced some changes to the handler function. The new handler has now 3 parameters.
function(event, context, callback)
Although you will still find the succeed, done and fail on the context parameter, AWS suggest to use the callback function instead or null is returned by default.
callback(new Error('failure')) // to return error
callback(null, 'success msg') // to return ok
Complete documentation can be found at http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html
Simple Working Example of Http request using node.
const http = require('https')
exports.handler = async (event) => {
return httprequest().then((data) => {
const response = {
statusCode: 200,
body: JSON.stringify(data),
};
return response;
});
};
function httprequest() {
return new Promise((resolve, reject) => {
const options = {
host: 'jsonplaceholder.typicode.com',
path: '/todos',
port: 443,
method: 'GET'
};
const req = http.request(options, (res) => {
if (res.statusCode < 200 || res.statusCode >= 300) {
return reject(new Error('statusCode=' + res.statusCode));
}
var body = [];
res.on('data', function(chunk) {
body.push(chunk);
});
res.on('end', function() {
try {
body = JSON.parse(Buffer.concat(body).toString());
} catch(e) {
reject(e);
}
resolve(body);
});
});
req.on('error', (e) => {
reject(e.message);
});
// send the request
req.end();
});
}
Yeah, awendt answer is perfect. I'll just show my working code... I had the context.succeed('Blah'); line right after the reqPost.end(); line. Moving it to where I show below solved everything.
console.log('GW1');
var https = require('https');
exports.handler = function(event, context) {
var body='';
var jsonObject = JSON.stringify(event);
// the post options
var optionspost = {
host: 'the_host',
path: '/the_path',
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
};
var reqPost = https.request(optionspost, function(res) {
console.log("statusCode: ", res.statusCode);
res.on('data', function (chunk) {
body += chunk;
});
context.succeed('Blah');
});
reqPost.write(jsonObject);
reqPost.end();
};
I faced this issue on Node 10.X version.
below is my working code.
const https = require('https');
exports.handler = (event,context,callback) => {
let body='';
let jsonObject = JSON.stringify(event);
// the post options
var optionspost = {
host: 'example.com',
path: '/api/mypath',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'blah blah',
}
};
let reqPost = https.request(optionspost, function(res) {
console.log("statusCode: ", res.statusCode);
res.on('data', function (chunk) {
body += chunk;
});
res.on('end', function () {
console.log("Result", body.toString());
context.succeed("Sucess")
});
res.on('error', function () {
console.log("Result Error", body.toString());
context.done(null, 'FAILURE');
});
});
reqPost.write(jsonObject);
reqPost.end();
};
Modern Async/Await Example
You need to prevent the lambda from finishing before the https request has completed. It makes code with multiple requests easier to read as well.
const https = require('https');
// Helper that turns https.request into a promise
function httpsRequest(options) {
return new Promise((resolve, reject) => {
const req = https.request(options, (res) => {
if (res.statusCode < 200 || res.statusCode >= 300) {
return reject(new Error('statusCode=' + res.statusCode));
}
var body = [];
res.on('data', function(chunk) {
body.push(chunk);
});
res.on('end', function() {
try {
body = JSON.parse(Buffer.concat(body).toString());
} catch(e) {
reject(e);
}
resolve(body);
});
});
req.on('error', (e) => {
reject(e.message);
});
req.end();
});
}
// Lambda starts executing here
exports.handler = async event => {
// --- GET example request
var options = {
method: 'GET',
hostname: 'postman-echo.com',
path: encodeURI('/get?foo1=bar1'),
headers: {
},
};
try {
const getBody = await httpsRequest(options);
// The console.log below will not run until the GET request above finishes
console.log('GET completed successfully! Response body:', getBody);
} catch (err) {
console.error('GET request failed, error:', err);
}
// --- POST example request
var options = {
method: 'POST',
hostname: 'postman-echo.com',
path: encodeURI('/hi/there?hand=wave'),
headers: {
},
};
try {
const postBody = await httpsRequest(options);
// The console.log below will not run until the POST request above finishes
console.log('POST response body:', postBody);
} catch (err) {
console.error('POST request failed, error:', err);
}
};
I had the very same problem and then I realized that programming in NodeJS is actually different than Python or Java as its based on JavaScript. I'll try to use simple concepts as there may be a few new folks that would be interested or may come to this question.
Let's look at the following code :
var http = require('http'); // (1)
exports.handler = function(event, context) {
console.log('start request to ' + event.url)
http.get(event.url, // (2)
function(res) { //(3)
console.log("Got response: " + res.statusCode);
context.succeed();
}).on('error', function(e) {
console.log("Got error: " + e.message);
context.done(null, 'FAILURE');
});
console.log('end request to ' + event.url); //(4)
}
Whenever you make a call to a method in http package (1) , it is created as event and this event gets it separate event. The 'get' function (2) is actually the starting point of this separate event.
Now, the function at (3) will be executing in a separate event, and your code will continue it executing path and will straight jump to (4) and finish it off, because there is nothing more to do.
But the event fired at (2) is still executing somewhere and it will take its own sweet time to finish. Pretty bizarre, right ?. Well, No it is not. This is how NodeJS works and its pretty important you wrap your head around this concept. This is the place where JavaScript Promises come to help.
You can read more about JavaScript Promises here. In a nutshell, you would need a JavaScript Promise to keep the execution of code inline and will not spawn new / extra threads.
Most of the common NodeJS packages have a Promised version of their API available, but there are other approaches like BlueBirdJS that address the similar problem.
The code that you had written above can be loosely re-written as follows.
'use strict';
console.log('Loading function');
var rp = require('request-promise');
exports.handler = (event, context, callback) => {
var options = {
uri: 'https://httpbin.org/ip',
method: 'POST',
body: {
},
json: true
};
rp(options).then(function (parsedBody) {
console.log(parsedBody);
})
.catch(function (err) {
// POST failed...
console.log(err);
});
context.done(null);
};
Please note that the above code will not work directly if you will import it in AWS Lambda. For Lambda, you will need to package the modules with the code base too.
I've found lots of posts across the web on the various ways to do the request, but none that actually show how to process the response synchronously on AWS Lambda.
Here's a Node 6.10.3 lambda function that uses an https request, collects and returns the full body of the response, and passes control to an unlisted function processBody with the results. I believe http and https are interchangable in this code.
I'm using the async utility module, which is easier to understand for newbies. You'll need to push that to your AWS Stack to use it (I recommend the serverless framework).
Note that the data comes back in chunks, which are gathered in a global variable, and finally the callback is called when the data has ended.
'use strict';
const async = require('async');
const https = require('https');
module.exports.handler = function (event, context, callback) {
let body = "";
let countChunks = 0;
async.waterfall([
requestDataFromFeed,
// processBody,
], (err, result) => {
if (err) {
console.log(err);
callback(err);
}
else {
const message = "Success";
console.log(result.body);
callback(null, message);
}
});
function requestDataFromFeed(callback) {
const url = 'https://put-your-feed-here.com';
console.log(`Sending GET request to ${url}`);
https.get(url, (response) => {
console.log('statusCode:', response.statusCode);
response.on('data', (chunk) => {
countChunks++;
body += chunk;
});
response.on('end', () => {
const result = {
countChunks: countChunks,
body: body
};
callback(null, result);
});
}).on('error', (err) => {
console.log(err);
callback(err);
});
}
};
Use promises with resolve reject. It worked for me!
Add above code in API gateway under GET-Integration Request> mapping section.
Yes, there's in fact many reasons why you can access AWS Lambda like and HTTP Endpoint.
The architecture of AWS Lambda
It's a microservice. Running inside EC2 with Amazon Linux AMI (Version 3.14.26–24.46.amzn1.x86_64) and runs with Node.js. The memory can be beetwen 128mb and 1gb. When the data source triggers the event, the details are passed to a Lambda function as parameter's.
What happen?
AWS Lambda run's inside a container, and the code is directly uploaded to this container with packages or modules. For example, we NEVER can do SSH for the linux machine running your lambda function. The only things that we can monitor are the logs, with CloudWatchLogs and the exception that came from the runtime.
AWS take care of launch and terminate the containers for us, and just run the code. So, even that you use require('http'), it's not going to work, because the place where this code runs, wasn't made for this.

Resources