I am fairly new to node and using express with it.
I have a simple API which gets data from mongo in this way:
From a Router I call a async method in AccountServices(which is in a different file) called getAccountDetails
getAccountDetails in turn gets data from mongo making an async call.
I don't want to pass req,res objects to the getAccountDetails function, So I've use async await and it works perfectly.
What I want to understand is does my whole thread/api goes into wait state for that async await function to resolve?
If So, how can I overcome this.
async/await functions are asynchronous which means that the rest of your code doesn't wait for them to finish. They are very useful for operations like fetching data since you can't predict the time it'll take which would potentially end up freezing your app for seconds or even minutes.
Related
I need to make part of my method's code asynchronous, so it will execute in a non-blocking manner. For this purpose I've tried to create a "dummy" Promise and put the specified code in then block. I have something like this:
public static foo(arg1, arg2) {
const prom = new Promise((resolve, reject) => {
if (notValid(arg1, arg2)) {
reject();
}
resolve(true);
});
prom.then(() => {
...my code using arg1 and arg2...
});
}
However, then block always executes synchronously and blocks whole app, even though each and every JS documentation tells that then always runs asynchronously. I've also tried to replace the Promise with this:
Promise.resolve().then(() => {
...my code using arg1 and arg2...
});
but got same result.
The only way I've managed to make then block work asynchronously is by using setTimeout:
const pro = new Promise(resolve => {
setTimeout(resolve, 1);
});
pro.then(() => {
...my code using arg1 and arg2...
})
What can be the reason behind then block working synchronously? I don't want to proceed with using setTimeout, because it is kind of a "dirty" solution, and I think that there should be a way to make then run asynchronously.
Application is NodeJS with Express, written using Typescript.
I need to make part of my method's code asynchronous, so it will execute in a non-blocking manner. For this purpose I've tried to create a "dummy" Promise and put the specified code in then block.
Promises don't really make things asynchronous in and of themselves. What they do is wrap around something that's already asynchronous, and give a convenient way to tell when that thing is done. If you wrap a promise around something synchronous, your code is mostly still synchronous, just with a few details about when the .then callback executes.
even though each and every JS documentation tells that then always runs asynchronously.
By that, they mean that it waits for the current call stack to finish, and then runs the code in the .then callback. The technical term for what it's doing is a "microtask". This delay is done so that the order of operations of your code is the same whether the promise is already in a resolved state, or if some time needs to pass before it resolves.
But if your promise is already resolved (eg, because it's wrapped around synchronous code), the only thing your .then callback will be waiting for is the currently executing call stack. Once the current call stack synchronously finishes, your microtask runs synchronously to completion. The event loop will not be able to progress until you're done.
The only way I've managed to make then block work asynchronously is by using setTimeout
setTimeout will make things asynchronous*, yes. The code will be delayed until the timer goes off. If you want to wrap this in a promise you can, but the promise is not the part that makes it asynchronous, the setTimeout is.
I don't want to proceed with using setTimeout, because it is kind of a "dirty" solution
Ok, but it's the right tool for the job.
* when i say setTimeout makes it asynchronous, i just mean it delays the execution. This is good enough for many cases, but when your code eventually executes, it will tie up the thread until it's done. So if it takes a really long time you may need to write the code so it just does a portion of the work, and then sets another timeout to resume later
My route calls a function startJob() that immediately returns a response "Job started." It also calls a function runJob(), which continues with the job flow (async).
Obviously, I can't just do const jobResults = await startJob(), since that function just returns "Job started." The actual results of the job are not available until all the subsequent functions complete.
So I need a way to check the results of the test after all the functions complete, meaning I need to know when all the functions complete.
The last function in the flow logResults() is not going to work to spy on, because I need it to call through (including an async call). Jasmine doesn't allow me to call it through and then call another function that checks the results.
My idea was to add a function at the end of logResults() called finishJob and spy on that. finishJob() wouldn't do anything, so I wouldn't need to call it through.
Pseudo-code:
spyOn(finishJob).and.callFake( [function that checks the results] )
However, it seems like bad practice to add a function to the code purely for testing reasons.
Other possible solutions would be using some kind of node watchers, a state change machine, or just doing a timeout (also bad practice? but quick implementation...)
I found a solution I'm happy with. I decided to start the testing at a different point in the flow.
Instead calling await startJob() to start the test, I called await runJob(), which kicks off the asynchronous process and gets the results.
That way, I could know when the process finishes (using await keyword obvs) and check the results at that time.
I am not sure if it's a bug in mongoose or if I am doing something wrong. Once I start using async functions when iterating on a cursor with eachAsync I experience memory leaks (quickly goes up to 4gb and then crashes). After trying some things I noticed that this wouldn't happen if I don't use an async function as callback.
No Memory leak:
const playerCursor: QueryCursor<IPlayerProfileModel> = PlayerProfile.find({}, projection).lean().cursor();
await playerCursor.eachAsync(
(profile: IPlayerProfileModel) => {
return;
},
{ parallel: 50 }
);
Memory leak:
const playerCursor: QueryCursor<IPlayerProfileModel> = PlayerProfile.find({}, projection).lean().cursor();
await playerCursor.eachAsync(
async (profile: IPlayerProfileModel) => {
return;
},
{ parallel: 50 }
);
Obviously above code doesn't make any sense but I need to perform an asynchronous operation within the function.
Question:
What is causing the memory leak / how can I avoid it?
It has to do with how async functions work.
Quoting the documentation:
When the async function returns a value, the Promise will be resolved
with the returned value.
Meaning, values returned by async functions will be automatically wrapped into a Promise.
In your first code sample, your code returns undefined whereas in the second code sample your code returns Promise.resolve(undefined).
What is causing the memory leak?
I didn't take a look at mongoose code but the documentation states:
If fn returns a promise, will wait for the promise to resolve before iterating on to the next one.
Since your first example does not return a Promise, I am thinking your callback is executed on each result all at once rather than sequentially.
How can I avoid it?
I'd recommend using async/wait as you used it on your second code sample.
After taking a look at the code (looking for an answer myself), if provided with a callback that doesn't return a promise eachAsync will run the callback as many times as fast as possible as it can.
This line is where your callback is executed. The next line checks whether it's a Promise and if it's not then execute right away callback which effectively calls your eachAsync callback on the next result. If your callback has any sort of async operation but returns right away then you end up with thousands and thousands of async operations running all at once.
On top of that, you set the option parallel to 100 so it executes eachAsync callback one hundred times in parallel.
This isn't a bug on mongoose because there are cases where this behavior is wanted and it does provide with a sequential processing using Promise. The documentation should mention the caveat of using a callback which doesn't return a Promise.
To go a little further, express uses next on middleware callbacks for the purpose of sequencing them.
I am trying to use async-await style of code throughout my app which is simple way to do async operation without any callback methods. I was able to do this for creating a wrapper apis for some external apis using "request-promise" module. But I am not sure if there are any similar thing available for contacting to database. I am completely trying to avoid callbacks.
Please let me know if anyone can help me in this regard. Let me know if you need any more information.
await requires its passed function to return a promise.
Seems like there are 2 things you can do.
Make your function, the one that interacts with the db(the one that performs asynchronous task) return a new Promise and then await the promise returned.
Use a library that performs MYSQL operations and return a promise . ex: https://www.npmjs.com/package/promise-mysql
I'm working on an Express app. At startup it connects to a Redis server and to a PostgreSQL sever. I'd like to wait for both connections to succeed before starting the express server. Now, if I were waiting on just one callback, I could initiate the Express server within that callback. But how to best do this when waiting on more than one async operation? I'm new to this. Is there a good idiomatic pattern for doing this sort of thing? Maybe grouping promises together or..?
Promises are what you want.
You can use .all() on an array of promises to wait for them all to complete. You didn't mention what Promise library you're using, but it's fairly universal. here's the Bluebird documentation: https://github.com/petkaantonov/bluebird/blob/master/API.md#all---promise
Promises are probably the idiomatic way to solve this. You will have to "promisify" your functions that return callbacks to turn them into something that returns and resolves promises, but then Promise.all() will work just fine. I personally use Bluebird for my nodejs development and regularly promisify existing functions or whole modules that use asynchronous callbacks.
If you aren't otherwise using a library that can promisify non-promised functions, then you can also just use a counter to keep track of when all your callbacks are done. This is the "older fashioned" way of doing things, but works just fine too. Such an operation works like this:
function setup(fn) {
// initialize counter to number of async operations
var counter = 3;
function checkDone() {
--counter;
if (counter <= 0) {
fn();
}
}
firstAsyncFunction(...., checkDone);
secondAsyncFunction(...., checkDone);
thirdAsyncFunction(...., checkDone);
}
setup(function() {
// this will get called when all the initial async operations are done
});