Google Cloud Storage NodeJS getFilesStream Async - node.js

I have a Google Function that never returns; it just hits the timeout limit as a Google Function. It works fine locally within under 60 seconds. Not sure what the issue might be. Code is below:
/**
* Responds to any HTTP request.
*
* #param {!express:Request} req HTTP request context.
* #param {!express:Response} res HTTP response context.
*/
const {Storage} = require('#google-cloud/storage');
exports.main = async (req, res) => {
const storage = new Storage({projectId: 'our-project'});
const store = storage.bucket('our-bucket');
const incomplete = {
LT04: [],
LT05: [],
LE07: [],
LC08: []
};
store.getFilesStream({prefix : 'prefixToMatch', autoPaginate : true })
.on('error', (err) => {
return console.error(err.toString())
})
.on('data', (file) => {
// Find small/bad files
if (file.metadata.size === 162) {
const split = file.name.split('/');
const prefix = split[2].substr(0, 4);
incomplete[prefix].push(file.name);
}
})
.on('end', () => {
return JSON.stringify(incomplete, false, ' ');
});
};

Your code it seems ok. But you need to take into account some additional details about this.
Does your Cloud function's memory is enough for this? I think that you could increase the memory allocated of your CF.
Are you sure that this is due to a timeout issue? If you have not seen the logs you can do it going to the Error reporting section.
In case that you have already confirm this, another option could be to change the timeout duration.

I think the issue was that I needed to send a "res.send" instead a Promise.resolve. As well, I needed to remove the async before the function.
Thanks for the quick response with guidelines, error was easier than that apparently.

Related

Why can't I make express route synchronous

I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.

Saving req to global variable in nodejs server

What are the consequences of making request global (or singleton), so that it is accessible all over the server and does not have to be passed in each function call? For example:
index.js:
const http = require('http');
const { saveReq } = require('./shared');
const {
doSomethingWithReqPassingItAsParameter,
doSomethingWithReqPassingItAsGlobal
} = require('./lib');
const requestListener = function (req, res) {
// approach 1
doSomethingWithReqPassingItAsParameter(req);
// approach 2
saveReq(req);
doSomethingWithReqPassingItAsGlobal();
res.writeHead(200);
res.end('Hello, World!');
}
const server = http.createServer(requestListener);
server.listen(8080);
lib.js:
const { loadReq } = require('./shared');
const doSomethingWithReqPassingItAsParameter = (req) => {
console.log('req as parameter', req.url);
};
const doSomethingWithReqPassingItAsGlobal = () => {
console.log('req as global', loadReq().url);
};
module.exports = {
doSomethingWithReqPassingItAsParameter,
doSomethingWithReqPassingItAsGlobal,
};
shared.js
var request;
const saveReq = (r) => request = r;
const loadReq = () => request;
module.exports = {
saveReq,
loadReq,
}
This is very convenient for large projects with many levels of function calls, but how parallel requests will be handled? I know that nodejs is single-threaded, does it mean than each http request will be run from end to finish separately or they can overlap, thus using a global request object would make a mess?
The consequences are that your server will only work for one request at a time and as soon as you have more than one request in flight at the same time, data will be mixed up between requests dealing to bugs, crashes, security issues and incorrect results.
Simply put, you cannot program a server that way. Pass the req object or data from it to any functions that need it. That keeps the appropriate req object associated with the right execution to avoid all the problems of trying to store a req object in some sort of global location where multiple requests in flight at the same time will/can conflict.
There is a relatively new thing in nodejs called "async local storage" that could perhaps be used for this. You can read a little about it here, though it's my personal opinion that it's still better to pass your request data to the functions that want to use it rather than the async local storage for this.

Using https for REST requests from inside expressjs applicaiton

From inside my expressJS application I have to verify that a cookie token is valid with a back-end server. So the relevant code involved in this is as follows:
app.get('*', (req, res, next) => {
console.log('GET: ' + req.path);
// ...
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
// build request
const restReq = https.request(authServerOptions, result => {
console.log('back-end response' + result.statusCode);
result.on('data', data => {
next(); // token is good now proceed.
});
result.on('error', error => {
res.redirect('somewhere'); // token is bad or timeout
});
});
restReq.write(token);
restReq.end();
}
So the main get function sets the REST request in motion and then just returns without calling next() or anything.
Questions:
Is this the right code for doing this? What happens if the callbacks are never called?
Is the application blocked from processing other requests until the back-end server returns or times out?
If so is there some way of freeing up the thread to process more requests?
Thanks in advance for any help. I haven't found many examples for this code pattern so if there is one a link would be appreciated.
Yes, I think the general idea of your implementation is correct.
I would also suggest, as done in the comments, to use a client such as axios to handle the request in a less verbose and more comprehensive manner, which would leave your code looking something like this:
const axios = require('axios');
app.get('*', (req, res, next) => {
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
axios.post(url, payload, opts)
.then(response => next())
.catch(error => {
console.error(error);
res.redirect('somewhere');
});
});
A bit more to the point, but functionally almost equivalent to your implementation. The one thing you are missing is the onerror callback for your request object, which currently may fail and never return a response as you correctly suspected. You should add:
restReq.on('error', error => {
console.error(error);
res.redirect('somewhere');
});
On the same vein, it would probably be more fitting to call next on result end, instead of doing so while reading response data:
result.on('end', () => {
next();
});
Then you'd be covered to guarantee that a callback would be invoked.
Neither implementation blocks the processing of future requests, as the call to the token validation service is done asynchronously in both cases.

Passing params from a Google Cloud Function GET Request to BigQuery

I have successfully deployed a google cloud function that takes in parameters from a POST request. I am now trying to change it to take in parameters from a GET request since the parameters don't contain any private data.
It seems I am correctly getting the passed in parameters, but when I then try to pass them to bigQuery it tells me I have missing parameters for my query. I know my code is correct because if I hard code the values of the parameters it works correctly e.g.:
bigQuery.createQueryJob({
query,
params: {
"make": "acura",
"model": "mdx",
"modelYear": 2005
}
}).then...
I also know I am getting the parameters correctly because if I change my cloud function to just return the passed in query string params, it correctly returns them (commented out lines below). If I change the cloud function to use req.body instead of req.query and make it a POST request, it also works fine.
I'm at a loss as to why "params" is not being passed to createQueryJob correctly. Any help would be much appreciated. Here is the code (I had to remove the actual query for privacy reasons):
package.json:
{
"name": "sample-http",
"version": "0.0.1",
"dependencies": {
"#google-cloud/bigquery": "^2.0.6"
}
}
index.js:
const { BigQuery } = require("#google-cloud/bigquery");
/**
* Responds to any HTTP request.
*
* #param {!express:Request} req HTTP request context.
* #param {!express:Response} res HTTP response context.
*/
exports.getRecallDataByVehicleInfo = (req, res) => {
res.set('Access-Control-Allow-Origin', "*");
res.set('Access-Control-Allow-Methods', 'GET, POST');
res.setHeader(
"Access-Control-Allow-Headers",
"X-Requested-With,content-type"
);
const params = req.query;
// res.status(200).send("make is - " + params.make + ", model is - " + params.model + ", model year is -" + params.modelYear);
// return;
const bigQuery = new BigQuery();
const query = `myQuery
where Make = #make
and Model = #model
and ModelYear = #modelYear`
bigQuery.createQueryJob({
query,
params
}).then(results => {
const job = results[0];
return job.getQueryResults({
autoPaginate: false,
timeoutMs: 1000000
},
callback());
});
const callback = () => (err, rows) => {
if (err) {
res.status(401).send(JSON.stringify(err));
}
else {
res.status(200).send(rows);
}
};
}
You don't use the right method. Have a look to the createQueryJob definition
There is no param, it's for creating a query as job. If you have a look to the official (bad) example, you have to use query method. Here its definition
Note: Why the example is bad?
In the official example, the param provided to the query method is named option. In the documentation of the query method, the first (and mandatory) param is named query, and you can optionally add an option param. So, the naming is confusing.
Thank-you #guillaume blaquiere for the help! I changed my code to the following and it appears to be working now:
const { BigQuery } = require("#google-cloud/bigquery");
/**
* Responds to any HTTP request.
*
* #param {!express:Request} req HTTP request context.
* #param {!express:Response} res HTTP response context.
*/
exports.getRecallDataByVehicleInfoTest = (req, res) => {
res.set('Access-Control-Allow-Origin', "*");
res.set('Access-Control-Allow-Methods', 'GET, POST');
res.setHeader(
"Access-Control-Allow-Headers",
"X-Requested-With,content-type"
);
const params = req.query;
const query = `query
where lower(Make) = #make
and lower(Model) = #model
and CAST(ModelYear as String) = #modelYear`
const queryObj = {query, params};
const options = {
autoPaginate: false,
timeoutMs: 1000000
}
const bigQuery = new BigQuery();
bigQuery.query(queryObj, options, function(err, rows){
if (err) {
res.status(401).send(JSON.stringify(err));
}
else {
res.status(200).send(rows);
}
});
}
UPDATE
The issue wasn't me using an old package or me using createQueryJob instead of query. The reason a POST would work and why hard coded params would work was because they would properly send over modelYear as a number. But when you GET the params from the queryString modelYear comes in as a string (obviously) which blows up the bigQuery sql comparison in the whereclause.
The UI for testing cloud functions assumes a POST request and so it would tell me that I wasn't passing in the make param whereas the actual call in the browser would just error out without a helpful error message because I wasn't catching and returning the error properly because I just did a .then and not a .catch on my createQueryJob. So I didn't know what the actual issue was.
Bottom line - both createQueryJob and query can accept params despite the documentation only showing it for query and there is no difference between GET and POST for cloud functions in terms of how they handle passing params into these functions.
Here is the code that shows createQueryJob also handles params:
https://github.com/googleapis/nodejs-bigquery/blob/master/src/bigquery.ts#L1139

How to get data from a RESTful API with dialogflow

i am struggling a bit with my google assistant action. Right now i am using Dialogflow and Firebase for my webhook. In my code i would like to get data from an API, for example this one: API. I am coding with Node.js by the way. Since Node is asynchronous i do not know how to get the Data. When i try to make an Callback it doesnt work e.g.:
app.intent(GetData, (conv) => {
var test= "error";
apicaller.callApi(answer =>
{
test = answer.people[0].name
go()
})
function go ()
{
conv.ask(`No matter what people tell you, words and ideas change the world ${test}`)
}
For some reason this works when i test it in an other application. With Dialogflow it does not work
I have also tried to use asynch for the function app.intent and tried it with await but this did not work too.
Do you have any idea how i could fix this?
Thank you in advance and best regards
Luca
You need to return Promise like
function dialogflowHanlderWithRequest(agent) {
return new Promise((resolve, reject) => {
request.get(options, (error, response, body) => {
JSON.parse(body)
// processing code
agent.add(...)
resolve();
});
});
};
See following for details:
Dialogflow NodeJs Fulfillment V2 - webhook method call ends before completing callback
If this works in another application then I believe you're getting an error because you're attempting to access an external resource while using Firebases free Spark plan, which limits you to Google services only. You will need to upgrade to the pay as you go Blaze plan to perform Outbound networking tasks.
Due to asynchronicity, the function go() is gonna be called after the callback of callapi been executed.
Although you said that you have tried to use async, I suggest the following changes that are more likely to work in your scenario:
app.intent(GetData, async (conv) => {
var test= "error";
apicaller.callApi(async answer =>
{
test = answer.people[0].name
await go()
})
async function go ()
{
conv.ask(`No matter what people tell you, words and ideas change the world ${test}`)
}
First follow the procedure given in their Github repository
https://github.com/googleapis/nodejs-dialogflow
Here you can find a working module already given in the README file.
You have to make keyFilename object to store in SessionsClient object creation (go at the end of post to know how to genearate credentials file that is keyFileName). Below the working module.
const express = require("express");
const bodyParser = require("body-parser");
const app = express();
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json());
//////////////////////////////////////////////////////////////////
const dialogflow = require("dialogflow");
const uuid = require("uuid");
/**
* Send a query to the dialogflow agent, and return the query result.
* #param {string} projectId The project to be used
*/
async function runSample(projectId = "<your project Id>") {
// A unique identifier for the given session
const sessionId = uuid.v4();
// Create a new session
const sessionClient = new dialogflow.SessionsClient({
keyFilename: "<The_path_for_your_credentials_file>"
});
const sessionPath = sessionClient.sessionPath(projectId, sessionId);
// The text query request.
const request = {
session: sessionPath,
queryInput: {
text: {
// The query to send to the dialogflow agent
text: "new page",
// The language used by the client (en-US)
languageCode: "en-US"
}
}
};
// Send request and log result
const responses = await sessionClient.detectIntent(request);
console.log("Detected intent");
console.log(responses);
const result = responses[0].queryResult;
console.log(` Query: ${result.queryText}`);
console.log(` Response: ${result.fulfillmentText}`);
if (result.intent) {
console.log(` Intent: ${result.intent.displayName}`);
} else {
console.log(` No intent matched.`);
}
}
runSample();
Here you can obtain the keyFileName file that is the credentials file using google cloud platform using
https://cloud.google.com/docs/authentication/getting-started
For complete tutorial (Hindi language) watch youtube video:
https://www.youtube.com/watch?v=aAtISTrb9n4&list=LL8ZkoggMm9re6KMO1IhXfkQ

Resources