I'm guessing this is related to not understanding promises and execution order, but I'm currently stumped why this Firebase Function (repackaged Google Cloud Functions) code runs recursively.
Currently the function executes once successfully (fetches data, writes database entry, writes file in storage), and then repeats every 15-30 seconds until it reaches the '402' error state. It is intended to only execute once.
Any help would be appreciated.
exports.add = functions.https.onRequest((req, res) => {
cors(req, res, () => {
if (req.query.idToken) {
// there's a query param
var idToken = req.query.idToken;
admin.auth().verifyIdToken(idToken)
.then(function(decodedToken) {
var uid = decodedToken.uid;
var userRef = database.ref('users/' + uid);
var feedCountRef = database.ref('users/' + uid).child('feeds');
var plansRef = database.ref('plans')
userRef.once('value', function(snapshot){
var feedsCount = snapshot.val().feeds;
var currentPlan = snapshot.val().membership;
var planRef = database.ref('plans/' + currentPlan);
planRef.once('value', function(snapshot) {
console.log(snapshot.val());
var allowedFeeds = snapshot.val().feeds;
if(feedsCount < allowedFeeds) {
fetchFeed(req.body.feedSource, function(feedData) {
var defaultFeedName = 'Untitled';
var defaultUpdateFrequency = 'Weekly';
var feedsdatabaseRef = database.ref('feeds/' + uid);
var newFeedDatabaseRef = feedsdatabaseRef.push();
var feedKey = newFeedDatabaseRef.key;
writeFeedStorage(feedKey, feedData, function(response) {
console.log(response);
newFeedDatabaseRef.set({
// write data
})
});
feedCountRef.transaction(function(feeds){
return (feeds || 0) + 1;
});
return;
});
} else {
console.log('over quota');
res.status(402).send({error: 'You are at the maximum number of feeds your plan allows.'});
}
});
})
}).catch(function(error) {
res.status(401);
});
} else {
res.status(401);
}
})
})
From your code snippet, a potential reason that it would be running repeatedly is that you are not returning an ok status if things worked out correctly, e.g.
res.status(200).send('ok');
According to the Firebase documentation, this is something you should be doing for HTTP Functions.
Related
This one has had me spinning my tires for about 2 days now, I'm ready to reach out for help :)
I have a google firebase functions app, running as a middle-ware to an angular SPA. Hoping to avoid some of the pay-by-use cost of Azure SQL, I wanted to implement a caching option for the most common queries.
I thought I knew redis, I've worked with it before. There's a simple enough example on the repo: https://www.npmjs.com/package//redis
Everything works fine, if it is top level.
But the way my application is built, i need the ability to set a cache value from within the .then of a Promise, and when I try to do that, all operation just stops, with no identifiable error logging, or even response from redis. Even in Azure insights, i'm not getting much feed-back, only that the 'set' operation isn't being counted in metrics.
So, just to clarify, this works:
// "cache", the object, set globally
export const testCache = functions.https.onRequest(
async (req: any, res) => {
await cache.connect();
var redisKey = 'testing_global';
var result = await cache.get(redisKey);
await cache.set(redisKey, 'testing new class')
console.log("\nDone");
cache.disconnect();
res.send('done');
})
But, this does not:
import { createClient } from 'redis';
const cache = createClient({
url: "rediss://" + process.env.REDIS_HOST_NAME + ":6380",
password: process.env.REDIS_KEY,
});
export const getValues = functions.https.onRequest(
(req: any, response) => {
cors(req, response, async () => {
response.set('Access-Control-Allow-Origin', origin);
var searchText = req.body['search'];
var offset = req.body['offset'];
var fetch = req.body['fetch'];
var x = req.body['x'];
var y = req.body['y'];
var districts = req.body['districtFilter'];
var sort = req.body['sort'];
if (offset) {
if (!/^\d+$/.test(offset))
throw new Error('bad number');
} else {
offset = 0;
}
if (fetch) {
if (!/^\d+$/.test(fetch))
throw new Error('bad number');
} else {
fetch = 200;
}
if (x) {
if (!/^-?\d+$/.test(x))
throw new Error('bad number');
} else {
x = null;
}
if (y) {
if (!/^-?\d+$/.test(y))
throw new Error('bad number');
} else {
y = null;
}
if (districts && districts.length > 0) {
districts = sanitizeStringArray(districts);
} else {
districts = null;
}
if (sort) {
switch (sort) {
case "scoreAsc":
case "scoreDesc":
case "priceAsc":
case "priceDesc":
break;
default:
sort = '';
}
}
var redisKey = `get_values_${searchText}_${offset}_${fetch}_${x}_${y}_${districts}_${sort}`;
var cacheResult: any = null;
await cache.connect();
// because I want to end up in the 'else', for testing
cacheResult = await cache.getFromCache('someOtherKey');
if (null !== cacheResult) {
response.send({
"status": "success",
"totalCount": cacheResult.totalCount,
"data": cacheResult.result
});
cache.disconnect();
} else {
var connection = new Connection(sqlConfig);
var totalCount: number = 0;
connection.on('connect', function (err: any) {
// If no error, then good to proceed.
console.log("Connected");
var sql = `EXEC SomeSPC;`;
const sqlRequest = new Request(sql, function (err: any) {
if (err) {
console.log(err);
}
});
const countRequest = new Request(
`EXEC SomeOtherSPC;`
, function (err: any) {
if (err) {
console.log(err);
}
}
)
sqlRequest.connection = connection;
countRequest.connection = connection;
var result: any[] = [];
sqlRequest.on('row', function (columns: any[]) {
var rowResult: any = {};
columns.forEach(function (column: any) {
rowResult[column['metadata']['colName']] = column['value'];
});
result.push(rowResult);
});
sqlRequest.on("requestCompleted", function (rowCount: any, more: any) {
console.log(rowCount + ' rows returned');
connection.execSql(countRequest);
countRequest.on('row', function (columns: any[]) {
totalCount = columns[0]['value'];
});
countRequest.on('requestCompleted', async function (rowCount: any, more: any) {
connection.close();
cacheResult = {
totalCount: totalCount
, result: result
};
// ******************************************************************
// Does Not Work, Just Fails, Without Much to Go On
await cache.set(redisKey, 'in Promise')
response.send({
"status": "success",
"totalCount": totalCount,
"data": result
});
})
});
connection.execSql(sqlRequest);
});
connection.on('infoMessage', infoError);
connection.on('errorMessage', infoError);
connection.on('end', end);
connection.on('debug', debug);
connection.connect();
console.log("Reading rows from the Table...");
}
})
}
)
There's **update- no longer ** a fair amount of psuedo-code here, so please **update- No Need to ** disregard any inconsistent lines. I went ahead and put in the full function, including all the fluff, since trimming the fat seems to make things difficult for others to understand what is being asked.
The sql stuff all works, if i take out the cache.set() everything is fine, but that line, in the result of the Promise, just fails, and I can't figure out why.
I've tried using cache locally and globally, extracting the cache operations to a function, and then to a separate class, and in all cases, i'm getting the same result.
Is there a known reason this wouldn't work?
As far as I can understand, given you didn't provide a reproducible example, your code is binding to the requestCompleted event for the second request only after it runs execSql(): I would suggest moving the binding block before that, otherwise the event may be skipped.
I think the rendering takes place before the searching of the string on the files, i have tried different methods but don't seems to get this working. any help will be appreciated. im a noob on to the nodejs. im trying to get the id of the user and query and get all the data and there after see if he is in any of the lists given and finally render the page.
const j = [];
let name = '';
const filename = [];
var ext = '';
module.exports = function(app, express) {
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
app.post('/cusdetails', isLoggedIn, function (req, res) {
var cusid=req.body.cusid;
var insertQuerys = "SELECT * FROM customer WHERE cusid=? ORDER BY rowid DESC LIMIT 1";
connection.query(insertQuerys,[cusid],
function(err, rows){
rows.forEach( (row) => {
name=row.fncus;
});
fs.readdir('./views/iplist', function(err, files) {
if (err)
throw err;
for (var index in files) {
j.push(files[index])
}
j.forEach(function(value) {
var k = require('path').resolve(__dirname, '../views/iplist/',value);
fs.exists(k, function(fileok){
if(fileok) {
fs.readFile(k, function(err, content) {
if (err) throw err;
if (content.indexOf(name) > -1) {
ext = path.extname(k);
filename.push(path.basename(k, ext));
}
});
}
else {
console.log(" FileNotExist ");
}
});
});
});
console.log(filename);
res.render('cusdetails.ejs', {rows: rows, user:req.user , aml: filename });
});
})
You can create simple Promise wrapper and then use it inside async/await function to pause execution until resolved.
// use mysql2 package as it provides promise, less work to write promise wrappers
const mysql = require('mysql2/promise');
// create the connection to database
const connection = mysql.createConnection({
host: 'localhost',
user: 'root',
database: 'test'
});
// sample wrapper
function some(k) {
// more advisable to have local variables, why do you need this to be array?
var filename = [];
return new Promise((resolve, reject) => {
// doing this is also not recommended check nodejs documentation **fs.exists** for more info
fs.exists(k, function(fileok){
if(fileok) {
fs.readFile(k, function(err, content) {
if (err) reject(err);
if (content.indexOf(name) > -1) {
ext = path.extname(k);
filename.push(path.basename(k, ext));
resolve(filename)
}
});
}
else {
// reject(new Error("FileNotExist"))
console.log(" FileNotExist ");
}
});
})
}
// note the use of async
app.post('/cusdetails', isLoggedIn, async function (req, res) {
var cusid=req.body.cusid;
var insertQuerys = "SELECT * FROM customer WHERE cusid=? ORDER BY rowid DESC LIMIT 1";
// using await to pause excution, waits till query is finished
const [rows] = await connection.query(insertQuerys,[cusid])
rows.forEach( (row) => {
name=row.fncus;
});
// then you can
var result = await some(k)
...
Note however this way you loose the advantage of concurrent execution, as it's kindoff blocking. If the result of one call is not used in another, you can execute in parallel and await for result to achieve sequencing like
const [rows] = connection.query(insertQuerys,[cusid])
var result = some(k)
console.log(await rows) // do something
console.log(await result) // do something
JavaScript is asynchronous. This means that if you have a function with a callback (i.e. your query), the callback will be called asynchronously, at an unknown time, while the other code executes.
You need to look up some tutorials how to deal with callbacks, to get a proper understanding of it. Another method is using async/await and/or promises.
Basically, if you take the following code:
console.log("this will print first");
setTimeout(function () {
console.log("this will print last");
}, 1000);
console.log("this will print second");
If you run the code above, the top level is executed synchronously, so, it first calls console.log, then it executes setTimeout, which is synchronous. It sets a timeout, then says "I'm ready", and the code continues to the other console.log. After 1 second (1000 milliseconds), the callback in the setTimeout function is executed, and only then that console.log is called. You can not make the rest of the code wait this way, you need to restructure your code or read into promises.
I am trying to build login system using node.js and i am stuck in this call back function error i have managed to build get login info and check them with data base but when i am verifying password it's taking some time so in there i need to use call back function but even i used callback function it's giving me the same error which is since validation or database call taking time it's executing other stuff in the in my case if conditions.
i have tried to implement this another way just tried to console.log order and all executing opposite this my result first
this is the order that it's run
3
2
undefined
1
but i need run this exactly opposite should i use promises instead of callback?
const {ipcMain} = require('electron');
const Password = require("node-php-password");
const connection = require("./connection");
var hash;
var done;
var self = module.exports = {
getuser_information:function(user_name,pwd,callback){
connection.query("SELECT * FROM `super_admin` WHERE ad_un = ?", user_name, function(err, result, fildes) {
if (err) throw err;
let numbers_retuned = result.length;
hash = result[0].desk;
console.log(1);
});
callback(hash,self.true_or_not);
},
hashverif:function(hash,true_or_not){
true_or_not();
console.log(2);
},
true_or_not:function(){
console.log(3);
return 1
}
}
UPDATE after your comment
You have two errors
Your get_stored_password function returns nothing when your callback function is called that's why console.log(function_returning_nothing()) outputs undefined
You forgot to pass done to your callback function in
get_stored_password's definition callback(done)
const {
ipcMain
} = require('electron');
const Password = require("node-php-password");
const connection = require("./connection");
var hash;
var done;
var self = module.exports = {
get_stored_password: function(name, pwd, callback) {
connection.query("SELECT * FROM `super_admin` WHERE ad_un = ?", name, function(err, result, fildes) {
if (err) throw err;
let numbers_retuned = result.length;
hash = result[0].desk;
if (numbers_retuned == 1) {
var test = pwd;
done = Password.verify(test, hash);
} else {
console.log('no');
return 0;
}
//you must pass an argument to your callback function
// and return done var to get an output when you log this function
callback(done);
return done;
});
},
chek_if_true: function(done) {
console.log(done);
if (done) {
return true;
} else {
return false;
}
}
}
That's why when you log done to the console it's undefined
I am trying to generate a session ID when an API call is made to my 'generateSession' endpoint. I want to make sure that I don't have any duplicate session ID, so I query the database checking for a match on the generated ID. If there isn't a match, the ID is valid and I make a second query to add an active user with said session ID.
Although my first query executes, the 'id_is_valid' boolean never gets set to true so my program gets stuck in the while loop.
I am fairly new to JavaScript, but from some research, I'm pretty sure the problem is due to the asynchronous nature of the database call. However, I'm not sure where to go from there. Could anyone with more js knowledge offer me some direction?
Thanks!
var express = require('express');
var router = express.Router();
var myDB = require('../db-connection');
function generateSession() {
var session_id = '';
var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
for (var i = 0; i < 30; i++) session_id += possible.charAt(Math.floor(Math.random() * possible.length));
return session_id;
}
router.get('/generateSession', function(req, res){
var session_id = '';
var id_is_valid = false;
while (!id_is_valid){
session_id = generateSession();
myDB.query("SELECT * FROM activeUser WHERE session_id = ?", [session_id], function(error, results, field){
if(error) throw error;
else{
if (results.length === 0) is_is_valid = true;
}
});
}
myDB.query("INSERT INTO activeUser (is_registered, session_id) VALUES (0, ?)", [session_id], function(error, results, fields){
if (error) res.send('{"success": false}');
else res.send('{"success": true, "session_id": "' + session_id + '"}');
});
});
Although my first query executes, the id_is_valid boolean never gets set to true so my program gets stuck in the while loop.
This is because db call, by nature is asynchronous. If you run the following program you will know it.
'use strict';
let id_is_valid = false;
let count = 0;
while (!id_is_valid) {
count++;
console.log(`No of time setTimeout Invoked ${count}`);
setTimeout(function() { // simulating a db call that takes a second for execution
id_is_valid = true;
}, 1000);
}
console.log('This line wont be printed');
Outputs
No of time setTimeout Invoked 61415
No of time setTimeout Invoked 61416
No of time setTimeout Invoked 61417
^C //I killed it.
Like damitj07, I too suggest using npms like shortId for uniquely generating the sessionId. This will help you to eliminate a database call.
But if your business logic restricts and you need it in the current fashion you wrote. I think we can use async & await
'use strict';
let isFound = false;
let count = 0;
function doDbQuery() {
return new Promise((resolve, reject) => {
setTimeout(function () {
resolve(true);
}, 2000);
});
}
async function run() {
while (!isFound) {
count++;
console.log(`No of time setTimeout Invoked ${count}`);
isFound = await doDbQuery();
}
console.log('This line WILL BE printed');
}
run();
Output
No of time setTimeout Invoked 1
This line WILL BE printed
Making those changes to your code,
router.get('/generateSession', async function (req, res) {
var session_id = '';
var id_is_valid = false;
while (!id_is_valid) {
session_id = generateSession();
id_is_valid = await checkSessionIdInDb(session_id);
}
myDB.query('INSERT INTO activeUser (is_registered, session_id) VALUES (0, ?)', [session_id], function (error, results, fields) {
if (error) {
res.send('{"success": false}');
} else {
res.send('{"success": true, "session_id": "' + session_id + '"}');
}
});
});
function checkSessionIdInDb() {
return new Promise((resolve, reject) => {
myDB.query('SELECT * FROM activeUser WHERE session_id = ?', [session_id], function (error, results, field) {
if (error) {
return reject(error);
} else {
if (results.length === 0) {
resolve(true);
}
resolve(false);
}
});
});
From what I could make out of your code what you want to do is basically create a new user session for logged in user with a unique session Id, also making sure that the session ID does not already exist in the collection.
So the solution to this can be to the first query to check if the session ID is already present in Active_Users Collection, if not make a save call to save the user with generated session Id.
var express = require('express');
var router = express.Router();
var myDB = require('../db-connection');
function generateSession() {
var session_id = '';
var possible = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
for (var i = 0; i < 30; i++) session_id += possible.charAt(Math.floor(Math.random() * possible.length));
return session_id;
}
router.get('/generateSession', function(req, res) {
var session_id = '';
myDB.query("SELECT * FROM activeUser WHERE session_id = ?", [session_id], function(error, results, field) {
if (error) {
throw error;
} else {
if (results.length === 0) {
//if session id is not present , insert new user
session_id = generateSession();
myDB.query("INSERT INTO activeUser (is_registered, session_id) VALUES (0, ?)", [session_id], function(error, results, fields) {
if (error) res.send('{"success": false}');
else res.send('{"success": true, "session_id": "' + session_id + '"}');
});
}
//else do nothing or inser nothing
}
});
});
But Ideally, if you are generating a truly random session id each time when you make a request to the /generate session, is it really necessary to check for duplicates in the collection.
Now if the logic to generate a random ID is not perfect, you can always use modules like shortId to do the work for you. This will avoid unnecessary database call and save on performance and your code will be much neater.
I need to build an application that does these things (in order):
on load:
01- connect to MongoDB 'db'
02- creates a collection 'cas'
03- check if a web page has updates, if yes go to step 04, if not go to step 07
04- do web scraping (using Cheerio) of the web site and get a $ variable like that $ = cheerio.load(body);
05- elaborate this object to get only informations I'm interested in and organize them in a jsons object like this one:
var jsons = [
{year: 2015, country: Germany, value: 51},
{year: 2015, country: Austria, value: 12},
{year: 2016, country: Germany, value: 84},
{year: 2016, country: Bulgaria, value: 104},
...
];
06- insert each of these elements ({year: 2015, country: Germany, value: 51}, ...) in the collection 'cas' of database 'db'
07- download the data (for example in a csv file)
08- create a web page for data visualization of these data using D3.js
09- disconnect from 'db'
If Node.js were synchronous, I could write something like this:
var url = 'http://...';
var jsons = [];
connectDb('db');
createCollection('db', 'cas');
if(checkForUpdates(url)) {
var $ = scrape(url);
jsons = elaborate($);
for(var i = 0; i < jsons.length; i++) {
saveDocumentOnDbIfNotExistsYet('db', 'cas', jsons[i]);
}
}
downloadCollectionToFile('db', 'cas', './output/casData.csv');
createBarChart('./output/casData.csv');
disconnectDb('db');
But Node.js is asynchronous so this code would not work properly.
I've read that I can use Promise to get the code to run in a certain order.
I read the documentation about the Promise and some sites that showed simple tutorials.
The structure of a Promise is:
// some code (A)
var promise = new Promise(function(resolve, reject) {
// some code (B)
});
promise.then(function() {
// some code (C)
});
promise.catch(function() {
// some code (D)
});
// some code (E)
If I understood correctly, in this case the execution (if Node.js were synchronous) would be equivalent to:
// some code (A)
// some code (E)
if(some code (B) not produce errors) {
// some code (C)
}
else {
// some code (D)
}
or (swap between code A and E, because they are asynchronous)
// some code (E)
// some code (A)
if(some code (B) not produce errors) {
// some code (C)
}
else {
// some code (D)
}
So now I wonder what is the right structure for my application.
I thought about:
var cheerio = require('cheerio');
var express = require('express');
var fs = require('fs');
var MongoClient = require('mongodb').MongoClient;
var dbUrl = 'mongodb://localhost:27017/';
var dbName = 'db';
var collectionName = 'cas';
const app = express(); // run using > node app.js
// connect to db
var connect = function(url) {
return new Promise(function(resolve, reject) {
MongoClient.connect(url + dbName, function(err, db) {
if(err) {
reject(err);
}
else {
console.log('Connected');
resolve(db);
}
});
});
}
// create collection
connect.then(function(db) {
db.createCollection(collectionName, function(err, res) {
if(err) {
throw err;
}
else {
console.log('Collection', collectionName, 'created!');
}
});
});
// connection error
connect.catch(function(err) {
console.log('Error during connection...');
throw err;
});
It's right? If yes, how can I proceed with other steps?
I can I improve my code?
EDIT 1
Following the example of Андрей Щербаков, I modified my code in this way:
app.js:
// my files
var db = require('./middlewares/db.js');
var url = 'mongodb://localhost:27017/';
var dbName = 'db';
var collectionName = 'cas';
const start = async function() {
const connect = await db.connectToMongoDb(url, dbName);
const cas = await connect.createYourCollection(collectionName);
const isPageHasUpdates = oneMoreFunction(); // i don't know how you gonna check it
if(isPageHasUpdates) {
await step 4;
await step 5;
await step 6;
}
await step 7
return something; // if you want
}
start()
.then(res => console.log(res)) // here you can use result of your start function if you return something or skip this then
.catch(err => console.log(err)); // do something with your error
middlewares/db.js:
var MongoClient = require('mongodb').MongoClient;
let dbInstance;
var methods = {};
methods.connectToMongoDb = function(url, dbName) {
if(dbInstance) {
return dbInstance;
}
else {
MongoClient.connect(url + dbName, function(err, db) {
if(!err) {
dbInstance = db;
return db;
}
});
}
}
methods.createYourCollection = function(collectionName) {
?.createCollection(collectionName, function(err, res) {
if(err) {
throw err;
}
});
}
module.exports = methods;
But I'm not sure I'm doing well.
How can I separate function in different files? For example I want to put all the function about db in file middlewares/db.js. But I have some problems in line ?.createCollection(collectionName, function(err, res).
If you are running node version 7.6 or higher, better way will be to use async await which works with promises.
So your code will look like
const start = async() => {
const connect = await connectToMongoDb(url);
const cas = await connect.createYourCollection();
const isPageHasUpdates = oneMoreFunction(); // i don't know how you gonna check it
if(isPageHasUpdates) {
await step 4;
await step 5;
await step 6;
}
await step 7
return something; // if you want
}
start()
.then(res => console.log(res)) // here you can use result of your start function if you return something or skip this then
.catch(err => console.log(err)); // do something with your error
Sure any function you are gonna await should be promisified as you did with your connect function( but if you are using https://www.npmjs.com/package/mongodb functions already promisified)
Update
The best way will be to use mongoose, but if you want to work with native mongodb you can write your mongodb like this https://pastebin.com/BHHc0uVN (just an example)
You can expand this example as you want.
You can create function createCollection
const createCollection = (connection, collectionName) => {
return connection.createCollection(collectionName); // actually i'm not sure that this function exists in mongodb driver
}
And usage will be:
const mongodbLib = require('./lib/mongodb'); //path to db.js file
mongodbLib.init()
.then(connection => mongodbLib.createCollection(connection, 'cas'))
.then(() => doSmthElse())
Or if you are sure that init is done(you can do it once before you main script like starting server or whatever you doing)
const mongodbLib = require('./lib/mongodb'); //path to db.js file
const connection = mongodbLib.getConnection();
Or if you want to simple work with collection like in step 6, add your cas collection(like user in example file). But this you can use when your init function is done as well.
So usage will be
const mongodbLib = require('./lib/mongodb');
const cas = mongodbLib.collections.cas;
cas().insertMany(docs)
.then()
.catch()