This question already has answers here:
Using async/await with a forEach loop
(33 answers)
How do I return the response from an asynchronous call?
(41 answers)
Closed 15 days ago.
I am using puppetter in node.js to get the data from a site, i am getting the data, but i am having a little problem with the promisse, i guess.
i gonna get this data and split it in two "variables" (country and league), but first a need to get the data from the site.
whats happening is that when i console.log inside the "foreach" its show me the data that i need, but out of it, it console.log a undefined value.I searched and saw that its because it is a promisse, but i am not being able to do it proppely.
i commented in the code what i am tryng to do....
try {
//using library puppeteer to get data from flashscore.com.br or the site you want
//you need to put the headless option as false to see the browser opening
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto("http://m.flashscore.com.br");
//getting the element that contains the games
const element = await page.$("div.soccer > div#score-data");
const dataCountryAndLeague = await element?.$$(`h4`,);
var data: string | null
dataCountryAndLeague?.forEach(element => {
element.getProperty('textContent').then(textContent => textContent.jsonValue().then(async (cld) => {
console.log(cld); //getting the country and league (THE Data its showed as ia want)
data = cld; //trying to pass the data to a variable
})
);
console.log(data); //trying to print the data outside the forEach, but its undefined
});
} catch (error) {
console.log(error + "Erro identificado ao tentar acessar os dados da página");
return error + "Erro identificado ao tentar acessar os dados da página";
}
Im am expecting to do what i explained before.
since alredy, thankyou for your atention.
We're gonna have to wait for all the promises to complete. For example:
async function handle(element) {
const textContent = await element.getProperty('textContent');
return textContent.jsonValue();
}
const promises = dataCountryAndLeague?.map(element => handle(element)) ?? [];
const data = await Promise.all(promises);
console.log(data) // All your data goes here
Related
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
How can I access the value of a promise?
(14 answers)
Closed 2 months ago.
I am attempting to set a variable scannedCode to a scanned qr code result. However, the steps after that happens continues even before scannedCode is in use, due to the promises. How can I make it so nothing continues until the variable finishes and is usable?
function stuff(image) {
var scannedCode = qrRun(image);
console.log("I want to console.log scannedcode and it be the actual thing completed")
}
async function qrRun(image) {
const img = await jimp.read(fs.readFileSync(image));
const qr = new QRReader();
const value = await new Promise((resolve, reject) => {
qr.callback = (err, v) => err != null ? reject(err) : resolve(v);
qr.decode(img.bitmap);
});
console.log(value.result);
return value.result;
}
Or if there is no way to do that, how would I just remove the Promise from the function currently there so I don't need a .then without breaking the qrrun.
async function stuff(image) {
const scannedCode = await qrRun(image);
}
I want fetch all child nodes in a given path from Firebase Realtime Database and which I get using a similar method to this.
But my main issue is I want to merge then into a single string and I tried out something like this:
if(userInput === '/showUsers') {
const usersRef = `Data/users/`
let returnText = `Your users:`
admin.database().ref(usersRef).on("value", function (snapshot) {
snapshot.forEach(function (e) {
const xSnap = e.val()
const xName = xSnap.Name
const xId = xSnap.id
console.log(`${xName} - ${xId}`)
returnText.concat(`\n${xName}\n${xId}`)
console.log(returnText)
})
})
return res.status(200).send({
method: 'sendMessage',
chat_id,
reply_to_message_id: messageIdtoReply,
text: `${returnText}`,
parse_mode: 'HTML'
})
}
So all the child nodes are getting fetched and they get logged into the console but the returnText always remain to it's predefined value. I have to do this because I want to return the data into a Telegram bot so it must be a single string as I cannot return multiple values to telegram bot.
I don't want to use a for-loop because those nodes won't be named as 1,2,3 or so on that I can use a loop.
Also how do I fetch only first 10 users/records a time so that it won't overflow Telegram message limit?
Expected Output:
All I want is to get the data into a single message.
I am not very familiar with typescript to be honest but I will try to answer your question.
Getting values from firebase is asynchronous which means
1 const usersRef = `Data/users/`
2 let returnText = `Your users:`
3
4 admin.database().ref(usersRef).on("value", function (snapshot) {
5 snapshot.forEach(function (e) {
6 const xSnap = e.val()
7 const xName = xSnap.Name
8 const xId = xSnap.id
9
10 console.log(`${xName} - ${xId}`)
11 returnText.concat(`\n${xName}\n${xId}`)
12 console.log(returnText)
13 })
14 })
15 console.log(returnText)
If you access returnText variable on line 15 for example it might be executed before the above function.
I think you should put this in a function with a call back that gets you returnText when everything finishes executing.
For Example:
function concatResults(cb){
const usersRef = `Data/users/`
let returnText = `Your users:`
admin.database().ref(usersRef).on("value", function (snapshot) {
snapshot.forEach(function (e) {
const xSnap = e.val()
const xName = xSnap.Name
const xId = xSnap.id
console.log(`${xName} - ${xId}`)
returnText.concat(`\n${xName}\n${xId}`)
console.log(returnText)
})
cb(returnText)
})
}
Here we have function concatResults with a parameter cb callback which is called when the foreach finishes it's job.
How to use?
Simply:
concatResults(function(result){
console.log(result);
})
Here we have implemented our callback that will be called after concatResults function finishes it's job.
In JavaScript, the .concat() function does not alter the value of any string. It just returns the concatenated values (w3 reference). Thus, you should use it as follows:
returnText = returnText.concat(`\n${xName}\n${xId}`);
Also, due to the single threaded nature of JS, in order to return the concatenated text you should perform the return inside the admin.database().ref(usersRef).on("value", function (snapshot) { ... } block or better yet, use a Promise as explained here
I have a list of promises and currently I am using promiseAll to resolve them
Here is my code for now:
const pageFutures = myQuery.pages.map(async (pageNumber: number) => {
const urlObject: any = await this._service.getResultURL(searchRecord.details.id, authorization, pageNumber);
if (!urlObject.url) {
// throw error
}
const data = await rp.get({
gzip: true,
headers: {
"Accept-Encoding": "gzip,deflate",
},
json: true,
uri: `${urlObject.url}`,
})
const objects = data.objects.filter((object: any) => object.type === "observed-data" && object.created);
return new Promise((resolve, reject) => {
this._resultsDatastore.bulkInsert(
databaseName,
objects
).then(succ => {
resolve(succ)
}, err => {
reject(err)
})
})
})
const all: any = await Promise.all(pageFutures).catch(e => {
console.log(e)
})
So as you see here I use promise all and it works:
const all: any = await Promise.all(pageFutures).catch(e => {
console.log(e)
})
However I noticed it affects the database performance wise so I decided to resolve every 3 of them at a time.
for that I was thinking of different ways like cwait, async pool or wrting my own iterator
but I get confused on how to do that?
For example when I use cwait:
let promiseQueue = new TaskQueue(Promise,3);
const all=new Promise.map(pageFutures, promiseQueue.wrap(()=>{}));
I do not know what to pass inside the wrap so I pass ()=>{} for now plus I get
Property 'map' does not exist on type 'PromiseConstructor'.
So whatever way I can get it working(my own iterator or any library) I am ok with as far as I have a good understanding of it.
I appreciate if anyone can shed light on that and help me to get out of this confusion?
First some remarks:
Indeed, in your current setup, the database may have to process several bulk inserts concurrently. But that concurrency is not caused by using Promise.all. Even if you had left out Promise.all from your code, it would still have that behaviour. That is because the promises were already created, and so the database requests will be executed any way.
Not related to your issue, but don't use the promise constructor antipattern: there is no need to create a promise with new Promise when you already have a promise in your hands: bulkInsert() returns a promise, so return that one.
As your concern is about the database load, I would limit the work initiated by the pageFutures promises to the non-database aspects: they don't have to wait for eachother's resolution, so that code can stay like it was.
Let those promises resolve with what you currently store in objects: the data you want to have inserted. Then concatenate all those arrays together to one big array, and feed that to one database bulkInsert() call.
Here is how that could look:
const pageFutures = myQuery.pages.map(async (pageNumber: number) => {
const urlObject: any = await this._service.getResultURL(searchRecord.details.id,
authorization, pageNumber);
if (!urlObject.url) { // throw error }
const data = await rp.get({
gzip: true,
headers: { "Accept-Encoding": "gzip,deflate" },
json: true,
uri: `${urlObject.url}`,
});
// Return here, don't access the database yet...
return data.objects.filter((object: any) => object.type === "observed-data"
&& object.created);
});
const all: any = await Promise.all(pageFutures).catch(e => {
console.log(e);
return []; // in case of error, still return an array
}).flat(); // flatten it, so all data chunks are concatenated in one long array
// Don't create a new Promise with `new`, only to wrap an other promise.
// It is an antipattern. Use the promise returned by `bulkInsert`
return this._resultsDatastore.bulkInsert(databaseName, objects);
This uses .flat() which is rather new. In case you have no support for it, look at the alternatives provided on mdn.
First, you asked a question about a failing solution attempt. That is called X/Y problem.
So in fact, as I understand your question, you want to delay some DB request.
You don't want to delay the resolving of a Promise created by a DB request... Like No! Don't try that! The promise wil resolve when the DB will return a result. It's a bad idea to interfere with that process.
I banged my head a while with the library you tried... But I could not do anything to solve your issue with it. So I came with the idea of just looping the data and setting some timeouts.
I made a runnable demo here: Delaying DB request in small batch
Here is the code. Notice that I simulated some data and a DB request. You will have to adapt it. You also will have to adjust the timeout delay. A full second certainly is too long.
// That part is to simulate some data you would like to save.
// Let's make it a random amount for fun.
let howMuch = Math.ceil(Math.random()*20)
// A fake data array...
let someData = []
for(let i=0; i<howMuch; i++){
someData.push("Data #"+i)
}
console.log("Some feak data")
console.log(someData)
console.log("")
// So we have some data that look real. (lol)
// We want to save it by small group
// And that is to simulate your DB request.
let saveToDB = (data, dataIterator) => {
console.log("Requesting DB...")
return new Promise(function(resolve, reject) {
resolve("Request #"+dataIterator+" complete.");
})
}
// Ok, we have everything. Let's proceed!
let batchSize = 3 // The amount of request to do at once.
let delay = 1000 // The delay between each batch.
// Loop through all the data you have.
for(let i=0;i<someData.length;i++){
if(i%batchSize == 0){
console.log("Splitting in batch...")
// Process a batch on one timeout.
let timeout = setTimeout(() => {
// An empty line to clarify the console.
console.log("")
// Grouping the request by the "batchSize" or less if we're almost done.
for(let j=0;j<batchSize;j++){
// If there still is data to process.
if(i+j < someData.length){
// Your real database request goes here.
saveToDB(someData[i+j], i+j).then(result=>{
console.log(result)
// Do something with the result.
// ...
})
} // END if there is still data.
} // END sending requests for that batch.
},delay*i) // Timeout delay.
} // END splitting in batch.
} // END for each data.
Im trying to extract the contents of variable topPost and place it into const options under url. I cant seem to get it to work. Im using the snoowrap/Reddit API and image-downloader.
var subReddit = r.getSubreddit('dankmemes');
var topPost = subReddit.getTop({time: 'hour' , limit: 1,}).map(post => post.url).then(console.log);
var postTitle = subReddit.getTop({time: 'hour' , limit: 1 }).map(post => post.title).then(console.log);
const options = {
url: topPost,
dest: './dank_memes/photo.jpg'
}
async function downloadIMG() {
try {
const { filename, image } = await download.image(options)
console.log(filename) // => /path/to/dest/image.jpg
} catch (e) {
console.error(e)
}
}
the recommended formatting for the image downloader is as follows:
const options = {
url: 'http://someurl.com/image.jpg',
dest: '/path/to/dest'
}
async function downloadIMG() {
try {
const { filename, image } = await download.image(options)
console.log(filename) // => /path/to/dest/image.jpg
} catch (e) {
console.error(e)
}
}
downloadIMG()
so it looks like i have to have my url formatted in between ' ' but i have no idea how to get the url from var topPost and place it in between those quotes.
any ideas would be greatly appreciated.
Thanks!
topPost is a Promise, not the final value.
Promises existence is to work with asynchronous data easily. Asynchronous data is data that returns at a point in the future, not instantly, and that's why they have a then method. When a Promise resolves to a value, the then callback is called.
In this case, the library will connect to Reddit and download data from it, which is not something that can done instantly, so the code will continue running and later will call the then callback, when the data has finished downloading. So:
var subReddit = r.getSubreddit('dankmemes');
// First we get the top posts, and register a "then" callback to receive all these posts
subReddit.getTop({time: 'hour' , limit: 1,}).map(post => post.url).then((topPost) => {
// When we got the top posts, we connect again to Reddit to get the top posts title.
subReddit.getTop({time: 'hour' , limit: 1 }).map(post => post.title).then((postTitle) => {
// Here you have both topPost and postTitle (which will be both arrays. You must access the first element)
console.log("This console.log will be called last");
});
});
// The script will continue running at this point, but the script is still connecting to Reddit and downloading the data
console.log("This console.log will be called first");
With this code you have a problem. You first connect to Reddit to get the top post URL, and then you connect to Reddit again to get the post Title. Is like pressing F5 in between. Simply think that if a new post is added between those queries, you will get the wrong title (and also you are consuming double bandwidth consumption, which is not optimal too). The correct way of doing this is to get both the title and the url on the same query. How to do so?, like this:
var subReddit = r.getSubreddit('dankmemes');
// We get the top posts, and map BOTH the url and title
subReddit.getTop({time: 'hour' , limit: 1,}).map(post => {
return {
url: post.url,
title: post.title
};
}).then((topPostUrlAndTitle) => {
// Here you have topPostUrlAndTitle[0].url and topPostUrlAndTitle[0].title
// Note how topPostUrlAndTitle is an array, as you are actually asking for "all top posts" although you are limiting to only one.
});
BUT this is also weird to do. Why don't you just get the post data directly? Like so:
var subReddit = r.getSubreddit('dankmemes');
// We get the top posts
subReddit.getTop({time: 'hour' , limit: 1,}).then((posts) => {
// Here you have posts[0].url and posts[0].title
});
There's a way to get rid of JavaScript callback hell with async/await, but I'm not going to enter into matter because for a newbie is a bit difficult to explain why is not synchronous code although it seems to look like so.
I've got the following firebase function to run once a file is uploaded to firebase storage.
It basically gets its URL and saves a reference to it in firestore. I need to save them in a way so that I can query them randomly from my client. Indexes seem to be to best fit this requirement.
for the firestore reference I need the following things:
doc ids must go from 0 to n (n beeing the index of the last
document)
have a --stats-- doc keeping track of n (gets
incremented every time a document is uploaded)
To achieve this I've written the following node.js script:
const incrementIndex = admin.firestore.FieldValue.increment(1);
export const image_from_storage_to_firestore = functions.storage
.object()
.onFinalize(async object => {
const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const splittedPath = filePath!.split("/");
// se siamo nelle immagini
// path = emotions/$emotion/photos/$photographer/file.jpeg
if (splittedPath[0] === "emotions" && splittedPath[2] === "photos") {
const emotion = splittedPath[1];
const photographer = splittedPath[3];
const file = bucket.file(filePath!);
const indexRef = admin.firestore().collection("images")
.doc("emotions").collection(emotion).doc("--stats--");
const index = await indexRef.get().then((doc) => {
if (!doc.exists) {
return 0;
} else {
return doc.data()!.index;
}
});
if (index === 0) {
await admin.firestore().collection("images")
.doc("emotions")
.collection(emotion)
.doc("--stats--")
.set({index: 0});
}
console.log("(GOT INDEX): " + index);
let imageURL;
await file
.getSignedUrl({
action: "read",
expires: "03-09-2491"
})
.then(signedUrls => {
imageURL = signedUrls[0];
});
console.log("(GOT URL): " + imageURL);
var docRef = admin.firestore()
.collection("images")
.doc("emotions")
.collection(emotion)
.doc(String(index));
console.log("uploading...");
await indexRef.update({index: incrementIndex});
await docRef.set({ imageURL: imageURL, photographer: photographer });
console.log("finished");
return true;
}
return false;
});
Getting to the problem:
It works perfectly if I upload the files one by one.
It messes up the index if I upload more than one file at once, because two concurrent uploads will read the same index value from --stats-- and one will overwrite the other.
How would you solve this problem? would you use another approach instead of the indexed one?
You should use a Transaction in which you:
read the value of the index (from "--stats--" document),
write the new index and
write the value of the imageURL in the "emotion" doc.
See also the reference docs about transactions.
This way, if the index value is changed in the "--stats--" document while the Transaction is being executed, the Cloud Function can catch the Transaction failure and generates an error which finishes it.
In parallel, you will need to enable retries for this background Cloud Function, in order it is retried if the Transaction failed in a previous run.
See this documentation item https://firebase.google.com/docs/functions/retries, including the video from Doug Stevenson which is embedded in the doc.