I'm trying to create a test, to verify that I've put an item in a dynamoDB table. In order to do so, right after I make a call that should put an Item (vehicle) in the database, I am trying to get the vehicle from the DB.
In my test I want to have a maximum number of retries (5). I want this while loop to be block the thread until my query has resolved to give a vehicle, or tried 5 times. Inside my test I have:
let count = 0
let car
while (!car || count < 5) {
setTimeout(async () => {
car = await findVehicle(greenCar.vehicleInfo)
}, 3000)
count++
}
And findVehicle is an asynchronous function that does a get from the dynamoDB table
If you want to wait on each iteration you can do this:
let count = 0;
let car;
while (!car || count < 5) {
await new Promise((resolve) =>
setTimeout(async () => {
car = await findVehicle(greenCar.vehicleInfo);
resolve();
}, 3000));
count++
}
So you are resolving the promise you are awaiting after you get your data. Also your function must be async in order to use await. Hope this helps.
Related
so I have a collection where I will have a lot of documents, but for now lets suppose it has about 100. (It had less than that when I was testing).
So, I need to get all the documents of the collection, put in a array, sort that and then send it to the websocket for the frontend. But the array is going with wrong data.
This is my code:
const emitSales = async (socket) => {
let salesArray = [];
const saleExists = (contract) => {
return salesArray.some(element => element.contract === contract);
}
const addSale = (contract) => {
const element = salesArray.find(e => e.contract === contract);
element.sales = element.sales+1;
}
const sales = await fiveMinSchema.find({}).lean();
if(sales) {
for await (x of sales) {
if(saleExists(x.contract)) {
addSale(x.contract);
continue;
}
const collection = await Collection.findOne({
contract: x.contract
});
let newsale = {
contract: x.contract,
title: collection.title,
description: collection.description,
image: collection.image,
sales: 1,
}
salesArray.push(newsale);
}
socket.emit("5min", salesArray.sort((a,b) => {
return b.sales-a.sales;
}).slice(0,10));
}
}
So, when I execute this function only once, for example, the array returns the correct values. But if I execute the function like 2 times in a row (like very fast), it starts returning the array with wrong data. (like mixing the data).
And as I using websocket, this function will execute like every 2 seconds (for example). How can I fix this problem? Like it seems to be executing more than one time simultaneously and mixing the data, idk..
I have a collection of teams containing around 80 000 documents. Every Monday I would like to reset the scores of every team using firebase cloud functions. This is my function:
exports.resetOrgScore = functions.runWith(runtimeOpts).pubsub.schedule("every monday 00:00").timeZone("Europe/Oslo").onRun(async (context) => {
let batch = admin.firestore().batch();
let count = 0;
let overallCount = 0;
const orgDocs = await admin.firestore().collection("teams").get();
orgDocs.forEach(async(doc) => {
batch.update(doc.ref, {score:0.0});
if (++count >= 500 || ++overallCount >= orgDocs.docs.length) {
await batch.commit();
batch = admin.firestore().batch();
count = 0;
}
});
});
I tried running the function in a smaller collection of 10 documents and it's working fine, but when running the function in the "teams" collection it returns "Cannot modify a WriteBatch that has been committed". I tried returning the promise like this(code below) but that doesn't fix the problem. Thanks in advance :)
return await batch.commit().then(function () {
batch = admin.firestore().batch();
count = 0;
return null;
});
There are three problems in your code:
You use async/await with forEach() which is not recommended: The problem is that the callback passed to forEach() is not being awaited, see more explanations here or here.
As detailed in the error you "Cannot modify a WriteBatch that has been committed". With await batch.commit(); batch = admin.firestore().batch(); it's exactly what you are doing.
As important, you don't return the promise returned by the asynchronous methods. See here for more details.
You'll find in the doc (see Node.js tab) a code which allows to delete, by recursively using a batch, all the docs of a collection. It's easy to adapt it to update the docs, as follows. Note that we use a dateUpdated flag to select the docs for each new batch: with the original code, the docs were deleted so no need for a flag...
const runtimeOpts = {
timeoutSeconds: 540,
memory: '1GB',
};
exports.resetOrgScore = functions
.runWith(runtimeOpts)
.pubsub
.schedule("every monday 00:00")
.timeZone("Europe/Oslo")
.onRun((context) => {
return new Promise((resolve, reject) => {
deleteQueryBatch(resolve).catch(reject);
});
});
async function deleteQueryBatch(resolve) {
const db = admin.firestore();
const snapshot = await db
.collection('teams')
.where('dateUpdated', '==', "20210302")
.orderBy('__name__')
.limit(499)
.get();
const batchSize = snapshot.size;
if (batchSize === 0) {
// When there are no documents left, we are done
resolve();
return;
}
// Delete documents in a batch
const batch = db.batch();
snapshot.docs.forEach((doc) => {
batch.update(doc.ref, { score:0.0, dateUpdated: "20210303" });
});
await batch.commit();
// Recurse on the next process tick, to avoid
// exploding the stack.
process.nextTick(() => {
deleteQueryBatch(resolve);
});
}
Note that the above Cloud Function is configured with the maximum value for the time out, i.e. 9 minutes.
If it appears that all your docs cannot be updated within 9 minutes, you will need to find another approach, for example using the Admin SDK from one of your server, or cutting the work into pieces and run the CF several times.
I have a function (called rankCheck), which takes three parameters:
Guild object (aka a Discord server)
UserId
Callback Function
The function will fetch the last 500 messages from every text channel in the provided guild. It will then will then only keep any messages that start with "!rank" and were sent by the provided UserId. Finally, it will count the remaining messages and pass the integer to the callback function.
async function rankChecks(guild, userId = *REMOVED FOR PRIVACY*, callback){
sumOfRankChecks = 0;
guild.channels.cache.each(channel => { //for each text channel, get # of rank checks for userId in last 500 msgs.
if (channel.type === "text"){
fetchMessages(channel, 500).then(msgs => {
let filteredMsgs = msgs.filter(msg => msg.content.startsWith("!rank") && msg.member.user.id == userId);
sumOfRankChecks = sumOfRankChecks + filteredMsgs.length;
});
}
});
callback(sumOfRankChecks);
}
Since discord only allows fetching 100 messages at once, I use this function (fetchMessages) to bypass this limit, by sending multiple requests, and then combining the results into one.
async function fetchMessages(channel, limit) {
const sum_messages = [];
let last_id;
while (true) {
const options = { limit: 100 };
if (last_id) {
options.before = last_id;
}
const messages = await channel.messages.fetch(options);
sum_messages.push(...messages.array());
last_id = messages.last().id;
if (messages.size != 100 || sum_messages >= limit) {
break;
}
}
return sum_messages;
}
When I call the rankCheck function, the return value is always 0
rankChecks(msg.guild, *REMOVED FOR PRIVACY*, function(int){
console.log(int);
});
Output:
0
However when I add a console.log into my rankCheck function:
async function rankChecks(guild, userId = *REMOVED FOR PRIVACY*, callback){
sumOfRankChecks = 0;
guild.channels.cache.each(channel => { //for each text channel, get # of rank checks for userId in last 500 msgs.
if (channel.type === "text"){
fetchMessages(channel, 500).then(msgs => {
let filteredMsgs = msgs.filter(msg => msg.content.startsWith("!rank") && msg.member.user.id == userId);
sumOfRankChecks = sumOfRankChecks + filteredMsgs.length;
console.log(sumOfRankChecks) //NEW CONSOLE.LOG!!!!!!!!!!!!!!!
});
}
});
callback(sumOfRankChecks);
}
Output:
3
5
This is the output I was expecting. Since I have 2 text channels in my server, I got 2 logs. If you had 3 channels, you would get 3 logs, etc. 3 messages from channel #1, and 2 messages from channel #2, therefore in total, there are 5 messages.
5 should be the integer that is passed into the callback function, but 0 is passed instead. Why is this?
Your callback function is being called before you even change sumOfRankChecks. Collection#each (and Map#forEach() and the gang) cannot wait for Promises to resolve because of the way they're built. Your code also wouldn't wait anyway, because you're not using await.
Despite what one might think is happening, guild.channels.each() is called, and callback() is called immediately after. This is the source of your confusion.
For more about async vs sync, you can check out the explanation in my answer here. You must use a for loop and await, or refactor your code so that async/await syntax is not necessary.
NOTE: The Discord.js documentation hyperlinked is for recently released v12. If your Discord.js isn't up to date, switch to the correct version at the top of the page for accurate info.
Please check if my understanding about the following for loop is correct.
for(let i=0; i<1000; i){
sample_function(i, function(result){});
}
The moment the for loop is invoked, 1000 events of sample_function will be qued in the event loop. After about 5 seconds a user gives a http request, which is qued after those "1000 events".
Usually this would not be a problem because the loop is asynchronous.
But lets say that this sample_function is a CPU intensive function. Therefore the "1000 events" are completed consecutively and each take about 1 second.
As a result, the for loop will block for about 1000 seconds.
Would there be a way to solve such problem? For example would it be possible to let the thread take a "break" every 10 loops? and allow other new ques to pop in between? If so how would I do it?
Try it this:
for(let i=0; i<1000; i++)
{
setTimeout(sample_function, 0, i, function(result){});
}
or
function sample_function(elem, index){..}
var arr = Array(1000);
arr.forEach(sample_function);
There is a technique called partitioning which you can read about in the NodeJs's document, But as the document states:
If you need to do something more complex, partitioning is not a good option. This is because partitioning uses only the Event Loop, and you won't benefit from multiple cores almost certainly available on your machine.
So you can also use another technique called offloading, e.g. using worker threads or child processes which also have certain downsides like having to serialize and deserialize any objects that you wish to share between the event loop (current thread) and a worker thread or a child process
Following is an example of partitioning that I came up with which is in the context of an express application.
const express = require('express');
const crypto = require('crypto');
const randomstring = require('randomstring');
const app = express();
const port = 80;
app.get('/', async (req, res) => {
res.send('ok');
})
app.get('/block', async (req, res) => {
let result = [];
for (let i = 0; i < 10; ++i) {
result.push(await block());
}
res.send({result});
})
app.listen(port, () => {
console.log(`Listening on port ${port}`);
console.log(`http://localhost:${port}`);
})
/* takes around 5 seconds to run(varies depending on your processor) */
const block = () => {
//promisifying just to get the result back to the caller in an async way, this is not part of the partitioning technique
return new Promise((resolve, reject) => {
/**
* https://nodejs.org/en/docs/guides/dont-block-the-event-loop/#partitioning
* using partitioning techinique(using setImmediate/setTimeout) to prevent a long running operation
* to block the eventloop completely
* there will be a breathing period between each time block is called
*/
setImmediate(() => {
let hash = crypto.createHash("sha256");
const numberOfHasUpdates = 10e5;
for (let iter = 0; iter < numberOfHasUpdates; iter++) {
hash.update(randomstring.generate());
}
resolve(hash);
})
});
}
There are two endpoints / and /block, if you hit /block and then hit / endpoint, what happens is that the / endpoint will take around 5 seconds to give back response(during the breathing space(the thing that you call it a "break"))
If setImmediate was not used, then the / endpoint would respond to a request after approximately 10 * 5 seconds(10 being the number of times block function is called in the for-loop)
Also you can do partitioning using a recursive approach like this:
/**
*
* #param items array we need to process
* #param chunk a number indicating number of items to be processed on each iteration of event loop before the breathing space
*/
function processItems(items, chunk) {
let i = 0;
const process = (done) => {
let currentChunk = chunk;
while (currentChunk > 0 && i < items?.length) {
--currentChunk;
syncBlock();
++i;
}
if (i < items?.length) {
setImmediate(process);//the key is to schedule the next recursive call (by passing the function to setImmediate) instead of doing a recursive call (by simply invoking the process function)
}
}
process();
}
And if you need to get back the data processed you can promisify it like this:
function processItems(items, chunk) {
let i = 0;
let result = [];
const process = (done) => {
let currentChunk = chunk;
while (currentChunk > 0 && i < items?.length) {
--currentChunk;
const returnedValue = syncBlock();
result.push(returnedValue);
++i;
}
if (i < items?.length) {
setImmediate(() => process(done));
} else {
done && done(result);
}
}
const promisified = () => new Promise((resolve) => process(resolve));
return promisified();
}
And you can test it by adding this route handler to the other route handlers provided above:
app.get('/block2', async (req, res) => {
let result = [];
let arr = [];
for (let i = 0; i < 10; ++i) {
arr.push(i);
}
result = await processItems(arr, 1);
res.send({ result });
})
I have orders collection and products collection in my application. The user can have multiple products in their single order. What I want to do is calculating the amount of each product reading through products collection and then perform the further action. Below is what I got as of now.
exports.myfunc = functions.firestore.document('collection/{collid}')
.onCreate(event => {
let data = event.data.data();
const products = data.products;
const prices = [];
_.each(products, (data1, index) => {
const weight = data1.weight;
const isLess = data1.isLess;
firebaseAdmin.firestore().collection('collection').doc(data1.productId).onSnapshot(data2 => {
let amount = weight === '1/2' ? data2.data().price1 : data2.data().price1 * weight;
amount += isLess ? 50 : 0;
prices.push(amount);
});
});
//Do some task after _.each with new total
});
But am not able to achieve synchronous task here, so that I can store actual amount for the product against its order and calculate total to store in document.
Could anyone please tell me how I achieve the above-said scenarios? How I can work along with promise and then callback?
You can map the products array to promises, like this:
var productPromises = products.map(product => {
return new Promise((resolve, reject) => {
firebaseOperation()...onSnapshot(resolve)
})
})
Promise.all(productPromises).then(results => {
// process all results at once
})
First, don't use onSnapshot() with Cloud Functions. That attaches a listener that stay listening indefinitely, until you remove it. That's not what you want at all, because functions can't execute indefinitely.
Instead, use get(), which returns a promise when the fetch is complete.
Also, you could consider accumulating all the documents you want to access into an array and use getAll() (with the spread operator on the array) to fetch them all.