For clarity I have other cloud functions that all run intermittently (i.e from 'cold' in around 2-6 seconds, and all use the same boilerplate set up of importing an admin instance and exporting the function as a module)
I've seen other similar posts but this is really bugging me. I have a cloud function like so:
const admin = require('../AdminConfig');
const { reportError } = require('../ReportError');
module.exports = (event) => {
const uid = event.params.uid;
const snapshot = event.data;
if (snapshot._newData === null ) {
return null;
}
console.log('Create org begin running: ', Date.now());
const organisation = event.data.val();
const rootRef = admin.database().ref();
const ref = rootRef.child('/organisations').push();
const oid = ref.key;
const userData = {
level: 'owner',
name: organisation.name,
};
const orgShiftInfo = {
name: organisation.name,
startDay: organisation.startDay || 'Monday',
};
const updatedData = {};
updatedData[`/users/${uid}/currentOrg`] = oid;
updatedData[`/users/${uid}/organisations/${oid}`] = userData;
updatedData[`/organisations/${oid}`] = organisation;
updatedData[`/org_shift_info/${oid}`] = orgShiftInfo;
rootRef.update(updatedData, (err) => {
if (err) {
return rootRef.child(`/users/${uid}/addOrgStatus`).set({ error: true })
.then(() => {
console.log(`error adding organisation for ${uid}: `, err);
return reportError(err, { uid });
});
}
console.log('Create org wrote succesfully: ', Date.now());
return rootRef.child(`/users/${uid}/addOrgStatus`).set({ success: true });
});
}
I understand the 'cold start' thing but I think something is seriously wrong that it's taking 25 seconds. The logs don't return any error and are as so:
Is there some deeper way I can debug this to try and figure out why it's taking so long? It's unusable at the moment. Thanks a lot.
Solved:
Sorry,
I misunderstood the API a bit. I should have watched the promise video first!
I needed to put
return rootRef.update...
instead of
rootRef.update...
Related
I have a POST ENDPOINT in my API where i want to register all work journals a worker made.
export const registerPoint = async (req: Request, res: Response) => {
const user = res.locals.decoded;
const {id} = user;
const ponto = new Ponto;
ponto.datePoint = new Date();
ponto.user_id = id;
//this numPoint should restart every day for that user...
ponto.numPoint = 1;
res.json({ msg: "Register point Route", user, id });
return;
}
how can i control how many times this requisition was made by a worker?
I want to control the variable numPoint, when the user makes this requisition it should increase by 01 and then in the final of the day returns it to 0.
Anyone knows about a solution or about a npm package that can handle this?
EDIT: Im storing all the data with SEQUELIZE + MYSQL.
As a starting point you could use a simple database for storing the data such as https://www.npmjs.com/package/sqlite3 or MySQL.
Running jobs daily you could consider https://www.npmjs.com/package/node-cron , for example having a daily job (outside of an API call function);
var cron = require('node-cron');
cron.schedule('0 0 1 * *', () => {
console.log('running every minute to 1 from 5');
});
From what I understand, you need a logging/audit trail mechanism. Since you are using MySQL you can create a new table with columns like (datetime, user_id, action). Every time the user does any action, it will be logged here. Then you can easily find out the count by aggregation. You won't need to reset any count for any user, if data doesn't exist for a given date then it's count will be 0.
I've made a solution that it worked for what i want.
Here is the solution below:
export const registerPoint = async (req: Request, res: Response) => {
const decodedUser = res.locals.decoded;
const user = await User.findByPk(decodedUser.id);
const TODAY_START = new Date().setHours(0, 0, 0, 0);
const NOW = new Date();
const markedPointsOfDay = await Ponto.findAll({
where: {
datePoint: {
[Op.gt]: TODAY_START,
[Op.lt]: NOW
},
}
});
const markedPointsOfDayByUser = await Ponto.findAll({
where: {
datePoint: {
[Op.gt]: TODAY_START,
[Op.lt]: NOW
},
UserId: (user !== null) ? user.id : decodedUser.id
}
})
if (!markedPointsOfDay || markedPointsOfDay.length === 0) {
const ponto = new Ponto;
ponto.datePoint = new Date();
if (user) {
console.log(user)
ponto.UserId = user.id as number;
console.log(ponto.UserId);
}
if (markedPointsOfDayByUser) {
ponto.numPoint = markedPointsOfDayByUser.length + 1;
}
const newPoint = await ponto.save();
res.json({ msg: "Ponto registrado com sucesso", msg2: "Caiu no IF de quando nao encontrou ponto do DIA", newPoint })
return;
}
if (markedPointsOfDay) {
const ponto = new Ponto;
ponto.datePoint = new Date();
if (user) {
ponto.UserId = user.id as number;
}
if (markedPointsOfDayByUser) {
ponto.numPoint = markedPointsOfDayByUser.length + 1;
}
const newPoint = await ponto.save();
res.json({ msg: "ponto registrado", markedPoint: newPoint, markedPointsOfDayByUser });
return;
}
return;
}
I am trying to cache the results of the Twitter query as they have rate limiting and I cannot seem to figure it out.
I am following this tutorial here.
The Cache.js file looks like so (taken from tutorial):
import NodeCache from 'node-cache';
class Cache {
constructor(ttlSeconds) {
this.cache = new NodeCache({ stdTTL: ttlSeconds, checkperiod: ttlSeconds * 0.2, useClones: false });
}
get(key, storeFunction) {
const value = this.cache.get(key);
if (value) {
return Promise.resolve(value);
}
return storeFunction().then((result) => {
this.cache.set(key, result);
return result;
});
}
del(keys) {
this.cache.del(keys);
}
delStartWith(startStr = '') {
if (!startStr) {
return;
}
const keys = this.cache.keys();
for (const key of keys) {
if (key.indexOf(startStr) === 0) {
this.del(key);
}
}
}
flush() {
this.cache.flushAll();
}
}
export default Cache;
My Twitter query file looks like so :
import TwitterCount from "../models/TwitterModel.js";
import { TwitterApi } from 'twitter-api-v2';
import dotenv from "dotenv";
dotenv.config();
import CacheService from '../middleware/Cache.js';
const twitterClient = new TwitterApi(process.env.TWITTER_API_BEARER_TOKEN);
const readOnlyClient = twitterClient.readOnly;
const ttl = 60 * 60 * 1; //cache for 1 hour
const cache = new CacheService(ttl);
export const getCount = async(req, res) => {
try {
const twit = await TwitterCount.findOne({
attributes:['followersCount']
});
const key = `Twit_${String(twit.dataValues.followersCount)}`;
const twitterFollowers = await readOnlyClient.v2.followers('1563787278857785350'); //HejOfficial account Twitter
const results = cache.get( key , () => twitterFollowers.data.length );
if (twit.dataValues.followersCount === results) {
console.log('same results');
} else {
await TwitterCount.create({
followersCount: results
});
console.log("Twitter Data added Successful");
}
res.json(twit);
cache.del(key);
} catch (error) {
console.log(error);
}
}
I presume that I am not using the key properly. Please assist if you have encountered this issue before, or provide a better caching alternative.
Thank you.
Try to use the account id as the key, rather than follower count. This way you are caching the accounts followers for an hour.
Something like this should get you going with the cache, it doesn't have the database inserts, but they can be handled now in the getCount method.
export const getCount = async(req, res) => {
const result = await getCountFromAPI("1563787278857785350") //HejOfficial account Twitter
res.json(result);
};
const getCountFromAPI = async(account) => {
const key = `Twit_${account}`;
return cache.get(key, async() => {
const twitterFollowers = await readOnlyClient.v2.followers(account);
return twitterFollowers;
});
};
Note that if you save the results to your database every time, it defeats the purpose of the cache. I'd suggest only using the API for this data, and if you want to use database as well, maybe the cache is unnessesary? Save the follower count and account id with time-updated value, and if time updated is more than one hour ago, then query the value again from Twitter.
Can someone please tell me what is wrong in my code before I go back to MongoDB?
Project is in Node.js (Next.js)
This is how I set firebase (it works for authentication with Google Login for instance):
import { initializeApp } from 'firebase/app';
const credentials = {
...
}
const firebase = initializeApp(credentials);
export default firebase;
then this is my api js file where it throws error "db.ref" is not a function:
import firebase from '#/firebase/firebase'
import { getDatabase, ref, onValue, update, child, orderByChild, equalTo, once } from "firebase/database"
export default async (req, res) => {
const db = getDatabase(firebase);
if (req.method === 'POST') {
const body = req.body
const playlistTracks = body.playlist
const playlistName = body.name
const uid = body.uid
const data = ...
console.log(data)
var ref = db.ref().child('users');
ref.child(uid).orderByChild('n').equalTo(playlistName).once("child_added", function(snapshot) {
let listId = snapshot.key;
db.ref("users/" + uid + "/" + listId).update(data);
res.send({ risp : 'ok' })
});
}
}
realtime database structure is:
- users
- <user uid>
- <playlist uid>
c: []
n: "playlist name"
so I'm trying to first retrieve the correct playlist by it's name ("n" value) comparing all "n" with the name of the given playlist, then I'd need to update (overwrite) it with my object (data)
UPDATE:
So I found the other methods Web version 9 (modular) in the documentation, as suggested by Frank van Puffelen below, but it now thorws a error
#firebase/database: FIREBASE WARNING: Exception was thrown by user
callback. Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they
are sent to the client
My code now is like this:
try {
const myQuery = query(ref(db, 'users/' + uid), orderByChild('n'), equalTo(playlistName));
onChildAdded(myQuery, (data) => {
let listId = data.key;
const updates = {};
updates["users/" + uid + "/" + listId] = dataToUpdate;
update(ref(db), updates);
}, {
onlyOnce: true
});
res.send({ risp : 'ok' })
} catch (e) {
res.status(400).end();
}
also tried like this, but it's the same error:
const myQuery = query(ref(db, 'users/' + uid), orderByChild('n'), equalTo(playlistName));
onChildAdded(myQuery, (data) => {
let listId = data.key;
update(ref(db, "users/" + uid + "/" + listId), dataToUpdate)
.then(() => {
res.send({ risp: 'ok' })
})
.catch((error) => {
res.status(400).end();
});
}, {
onlyOnce: true
});
You're using the new modular API, so can't use namespaced accessors like db.ref() anymore. Instead use ref(db, 'users').
I highly recommend keeping the documentation handy while upgrading this code to the new API version, as it has example of the modular and namespaced API side by side. The upgrade guide is probably also a worthwhile read).
I have an iteration that can take up to hours to complete.
Example:
do{
//this is an api action
let response = await fetch_some_data;
// other database action
await perform_operation();
next = response.next;
}while(next);
I am assuming that the operation doesn't times out. But I don't know it exactly.
Any kind of explanation of nodejs satisfying this condition is highly appreciated. Thanks.
Update:
The actual development code is as under:
const Shopify = require('shopify-api-node');
const shopServices = require('../../../../services/shop_services/shop');
const { create } = require('../../../../controllers/products/Products');
exports.initiate = async (redis_client) => {
redis_client.lpop(['sync'], async function (err, reply) {
if (reply === null) {
console.log("Queue Empty");
return true;
}
let data = JSON.parse(reply),
shopservices = new shopServices(data),
shop_data = await shopservices.get()
.catch(error => {
console.log(error);
});
const shopify = new Shopify({
shopName: shop_data.name,
accessToken: shop_data.access_token,
apiVersion: '2020-04',
autoLimit: false,
timeout: 60 * 1000
});
let params = { limit: 250 };
do {
try {
let response = await shopify.product.list(params);
if (await create(response, shop_data)) {
console.log(`${data.current}`);
};
data.current += data.offset;
params = response.nextPageParameters;
} catch (error) {
console.log("here");
console.log(error);
params = false;
};
} while (params);
});
}
Everything is working fine till now. I am just making sure that the execution will ever happen in node or not. This function is call by a cron every minute, and data for processing is provided by queue data.
I have a function,
var findUserDevice = function(userDeviceId){
var device = db.DeviceUser.find({
where: {
id: userDeviceId
}
}).then(function(device) {
if (!device) {
return 'not find';
}
return device.dataValues;
});
};
but this function does not return anything...
var UserDevice = findUserDevice(req.body.deviceUserId);
console.log(UserDevice);// undefined
The operation you are trying to do is async, which means that you need to use a callback. Since sequelize is build on top of Promises, you should actually write your code like this :
var findUserDevice = function(userDeviceId){
// return the promise itself
return db.DeviceUser.find({
where: {
id: userDeviceId
}
}).then(function(device) {
if (!device) {
return 'not find';
}
return device.dataValues;
});
};
And later use it like :
findUserDevice(req.body.deviceUserId).then( function(UserDevice) {
console.log(UserDevice);
});
It's 2020, async & await are becoming more popular. You can change your code to:
const findUserDevice = async function (userDeviceId) {
const device = await db.DeviceUser.findOne({
where: {
id: userDeviceId
}
});
if (device === null) {
return 'device not found';
}
return device.dataValues;
};
(async () => {
// ...
const UserDevice = await findUserDevice(req.body.deviceUserId);
console.log(UserDevice);
// ...
})()
IMHO, the code above is way more readable.
If you are getting undefined instead of 'not find' on the console, it means your function is returning a value. The problem might be dataValues is actually undefined. You need to check for the content of device.
Hint: Try returning just device or device.id
PS. If you want to do the search based on id, should go for findById() function of your model.
var device = db.DeviceUser.findById(userDeviceId).then(function(device) {
if (!device) {
return 'not find';
}
return device.dataValues;
});
This function received params id, this worker for me:
const { customer } = require('../models');
const get = async function(req, res){
let id = req.params.id;
[err, singleCustomer] = await to(customer.findByPk(id, { raw : true }));
return ReS(res, { message :'Obtener cliente: : ', data : JSON.stringify(singleCustomer) });
}