FileStream Promise resolving early - node.js

I am experiencing a rather weird problem in nodeJS, and I cannot quite figure out why.
Consider the following code:
(async () => {
console.log ("1");
await new Promise ((resolve, reject) => {
setTimeout (() => {
console.log ("2");
resolve ();
}, 1000);
});
console.log ("3");
process.exit ();
})();
This code does exactly what it is supposed to do. It prints 123, in that order. After the print of 1, it waits approximately one second. Perfect. Now let's see the following example:
const fs = require ("fs");
(async () => {
const stream = fs.createWriteStream ("file.txt");
stream.write ("Test");
console.log ("1");
await new Promise ((resolve, reject) => {
stream.on ("finish", () => {
console.log ("2");
resolve ();
});
});
console.log ("3");
process.exit ();
})();
From my understanding, this code should either complete, or - in case the finish event never gets fired - run infinitely. What happens is the exact opposite: It prints 1, then quits. Shouldn't it at least print another 3 before quitting, since this is the end of the script?
Important: I know that the promise will not resolve, because .end() is not called on the stream. I want to know why the script finishes anyway.
Can anyone explain this behaviour to me?

The best explanation is probably to write this without the async/await keywords and for you to undertstand these don't do anything "magical" and are simply "sugar" for a different way to resolve a Promise as opposed to .then().
const fs = require ("mz/fs");
const stream = fs.createWriteStream("file.txt");
stream.write("Test");
console.log("1");
new Promise ((resolve, reject) => {
stream.on ("finish", () => {
console.log("2");
resolve();
});
}).then(() => {
console.log("2");
process.exit();
});
That's the exact same thing right! So where's the catch.
The thing you are really missing is there is "nothing" that says when you open a file handle it "must" be explicitly closed before the program can exit. As such, there is "nothing to wait for" and the program completes but does not "branch" into the part that is still awaiting the Promise to resolve().
The reason why it only logs "1" is because the remaining branch "is" waiting for the Promise to resolve, but it's just never going to get there before the program finishes.
Of course that all changes when you actually call stream.end() immediately after the write or ideally by "awaiting" any write requests that may be pending:
const fs = require ("mz/fs");
(async () => {
const stream = fs.createWriteStream ("file.txt");
await stream.write ("Test"); // await here before continuing
stream.end()
console.log ("1");
await new Promise ((resolve, reject) => {
stream.on ("finish", () => {
console.log ("2");
//resolve ();
});
});
console.log ("3");
//process.exit ();
})();
That of course will log each output in the listing, as you should well know.
So If you were expecting to see the "3" in the log, the reason why it does not is because of the await where we don't ever close the stream. Again probably best demonstrated by getting rid of the await:
const fs = require ("mz/fs");
(async () => {
const stream = fs.createWriteStream ("file.txt");
await stream.write ("Test");
stream.end()
console.log ("1");
new Promise ((resolve, reject) => { // remove await - execution hoisted
stream.on ("finish", () => {
console.log ("2");
//resolve ();
});
});
console.log ("3");
//process.exit ();
})();
Then you "should" see:
1
3
2
At least on most systems unless you have an "extreme" lag. But generally the "finish" should get fired before the next line was reached after "awaiting" the write.
NOTE: Just using the mz library here for demonstration of an an await on the write() method without wrapping a callback. Generally speaking the callback execution should resolve just the same.

Related

Why doesn't this promise work inside async method?

I just started learning node.js and am trying to figure out how promises work.
I'm already familiar with async/await and am using axios to fetch data. Once the data is fetched, I want to write the data to a file and when that's done, log successful to the console.
To learn more about promises, I want to make my own that logs ++ to the console and then integrate that promise into the above.
So I wrote this, but when I run it with node, it only logs ++. I tried some variants, but don't understand what's happening.
const second = async () => {
console.log('++')
}
const processData = async () => {
const req = await axios.get('http://localhost:5004/swagger');
let reqJson = JSON.stringify(req.data);
fs.writeFile('newSwagger.json', reqJson, (err) => {
if (err) throw err;
console.log('successful');
});
}
let firstPromise = new Promise (function (resolve, reject) {
second(() => {
resolve(processData());
});
});
firstPromise.then(function() {
second();
});
Why does the above code not produce the following output?
++
successful
++
const axios = require("axios");
const fs = require("fs").promises;
const second = async () => {
console.log('++')
}
const processData = async () => {
const req = await axios.get('http://google.com');
let reqJson = JSON.stringify(req.data);
try {
await fs.writeFile('newSwagger.json', reqJson);
console.log('succesful');
} catch (e) {
throw e;
}
}
let firstPromise = new Promise (function (resolve, reject) {
second().then(processData).then(resolve);
});
firstPromise.then(second)
.then(() => console.log('finished'));
results in
++
successful
++
finished
Explanations
second is a parameterless function that returns a promise. It isn't itself a promise. This means that when you're doing second(() => resolve(processData())), you're never actually invoking the processData function. You're just executing the second function and handing it a function that would run processData but instead it never executes the passed in function as something like this would:
const second = async (callback) {
callback()
}
Also the async keyword for second is unnecessary since you're not using an await within. The async keyword basically is syntactic sugar which wraps the function into returning a promise kind of like:
const second = () => {
console.log('++')
return new Promise();
so in my corrected example, that's why I can suddenly do second().then(doSomething) however because there's nothing awaited within, it's the exact same as just second() followed by doSomething().
Also, the resolve inside of new Promise((resolve, reject) => ...) is kind of like setting a return, but it doesn't necessarily wait for anything. So if you want the second '++' to wait until processData is completely done and 'successful' is logged, you need to chain off the processData's returned promise like .then(processData).then(resolve).
Finally if you do the above, you may still notice that you don't see your 'successful' string printed - or well you might sometimes and at unexpected times. This is because you've also added callbacks into the mix along with promises and async/await. the ultimate effect being that your console.log('successful') will run at some point but you have nothing waiting for it nor chained after it. You could promisify the call to fs.writeFile yourself but you could also just use the built-in promise version of node's fs module - see this for more.
If it helps focus only on how the promises work(without async/await and callbacks also), here's a version with only promises to achieve the same resulting output:
const axios = require("axios");
const fs = require("fs").promises;
const second = () => {
console.log('++')
}
const processData = () => {
return new Promise(
resolve => axios.get('http://google.com').then((req) => {
let reqJson = JSON.stringify(req.data);
return reqJson;
}).then((reqJson) => {
fs.writeFile('newSwagger.json', reqJson).then(resolve);
}).then(() => console.log('successful'))
);
}
second();
processData().then(() => {
second();
console.log('finished')
})

Firebase onCall method finishes before code execution completes

I have a rather intensive Node function that includes multiple aync tasks that waits for the previous action to complete prior to going to the next step. It's crucial that the entire task finishes when it is called. I'm seeing issues where the onCall function will end before the code execution finishes. I can see this in the logs, as I'm console logging the various methods called from top to bottom until the function is "complete". The parent function is here:
exports.myHandyFunction = functions.https.onCall((data, context) => {
return new Promise((resolve, reject) => {
const response = async () => {
try {
let result = await some_main_function_elsewhere_in_code();
resolve(result)
} catch (err) {
console.log('parent promise caught error:')
console.log(err)
reject(err)
}
}
response()
})
})
I've increased the function's timeout from 60s to 240s hoping that this is purely a function timeout issue? This is pretty troubling, as I have some complex logic being called that ends up not fully completing. Any help would be greatly appreciated.
Your response function is async. When you call it, you're not awaiting for it's execution. This is leading on the premature end of the oncall method execution.
Instead of wrapping the business logic inside an async function, wrapped inside a promise, you can change the code like this:
exports.myHandyFunction = functions.https.onCall((data, context) => {
return new Promise(async (resolve, reject) => {
try {
let result = await some_main_function_elsewhere_in_code();
resolve(result)
} catch (err) {
console.log('parent promise caught error:')
console.log(err)
reject(err)
}
})
})
Basically, the Promise is declared as async, so we can use the await constructor inside of it. Then, we just have to call the some_main_function_elsewhere_in_code method, awaiting for it.

simple UnhandledPromiseRejectionWarning

When I try to run this, i get "UnhandledPromiseRejectionWarning". But last "catch" received control anyway with correct "err" object.
I have got two warnings:
UnhandledPromiseRejectionWarning
PromiseRejectionHandledWarning
If I comment "return delay(5000);" all works fine.
Why does Node.JS handle "promiseErr" before I do that?
const delay = (ms: number) => new Promise(res => setTimeout(res, ms));
let promiseErr = new Promise( (resolve, reject) => {
reject(new Error("__error__"));
});
let promiseOk = new Promise( (resolve, reject) => {
resolve();
});
promiseOk
.then( () => {
return delay(5000);
})
.then( () => {
return promiseErr;
})
.then( () => {
console.log("OK");
})
.catch( (err) => {
console.log(`ERR ${err}`);
});
program output
You have a rejected promise asynchronously (after 5000ms). Node.js detects unhandled promises by looking at promises that were not handled synchronously or within a microtick (inside a then).
Since Node.js cannot know "for sure" when a promise rejection is unhandled at the moment it errors on the safe side and gives you the warning. It is generally better to add catch handlers to promise errors synchronously as much as possible.
You can suppress the warning by adding a .catch(() => {}) to promiseErr when you create it which will handle the exception:
var promiseErr = ...
promiseErr.catch(() => {}); // add a catch handler without changing the original
In general - the solution is to not write code in such an error prone way. If promiseOk or delay rejects (which Node has no way of knowing) - then the exception in promiseErr itself is indeed unhandeld - so this is not really a safe way to write code.

How to asynchronously createReadStream in node.js with async/await

I am having difficulty with using fs.creadReadStream to process my csv file asynchronously:
async function processData(row) {
// perform some asynchronous function
await someAsynchronousFunction();
}
fs.createReadStream('/file')
.pipe(parse({
delimiter: ',',
columns: true
})).on('data', async (row) => {
await processData(row);
}).on('end', () => {
console.log('done processing!')
})
I want to perform some asynchronous function after reading each record one by one before the createReadStream reaches on('end').
However, the on('end') gets hit before all of my data finishes processing. Does anyone know what I might be doing wrong?
Thanks in advance!
.on('data, ...) does not wait for your await. Remember, an async function returns a promise immediately and .on() is not paying any attention to that promise so it just keeps merrily going on.
The await only waits inside the function, it does not stop your function from returning immediately and thus the stream thinks you've process the data and keeps sending more data and generating more data events.
There are several possible approaches here, but the simplest might be to pause the stream until processData() is done and then restart the stream.
Also, does processData() return a promise that is linked to the completion of the async operation? That is also required for await to be able to do its job.
The readable stream doc contains an example of pausing the stream during a data event and then resuming it after some asynchronous operation finishes. Here's their example:
const readable = getReadableStreamSomehow();
readable.on('data', (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
readable.pause();
console.log('There will be no additional data for 1 second.');
setTimeout(() => {
console.log('Now data will start flowing again.');
readable.resume();
}, 1000);
});
I ran into the same problem recently. I fixed it by using an array of promises, and waiting for all of them to resolve when .on("end") was triggered.
import parse from "csv-parse";
export const parseCsv = () =>
new Promise((resolve, reject) => {
const promises = [];
fs.createReadStream('/file')
.pipe(parse({ delimiter: ',', columns: true }))
.on("data", row => promises.push(processData(row)))
.on("error", reject)
.on("end", async () => {
await Promise.all(promises);
resolve();
});
});

await inside async controller function [duplicate]

I am working on a simple TCP client for a server and am using the latest node 7.6 because of the async/await functions. I'm new to node and asynchronous coding, so I apologize if this is stupidly easy.
I want to run a function that calls the callServer() function with specific parameters, wait until it finishes getting the data, and return the data as a variable.
Here is my code:
'use strict'
const net = require('net')
var formattedJson = funcRequestToJson("Server.GetStatus", false)
doThings()
async function doThings() {
var info = await callServer()
}
async function callServer() {
var client = new net.Socket()
client.connect(1705, '192.168.28.16', () => {
console.log('connected to server')
client.write(formattedJson)
})
client.on('data', (data) => {
client.destroy()
//return await data
console.log(data.toString())
})
client.on('close', () => {
})
}
// method and paramBool are always required
// macAddress, paramKey, paramValue are required for if paramBool is true
function funcRequestToJson(method, paramBool, macAddress, paramKey, paramValue) {
var objectRequest = {}
objectRequest[ "jsonrpc" ] = "2.0"
objectRequest[ "method" ] = method
objectRequest[ "id" ] = 0
if (paramBool == true) {
objectRequest[ "params" ] = {
"client": macAddress,
[paramKey]: paramValue
}
}
var json = (JSON.stringify(objectRequest) + '\r\n')
return json
}
So I didn't declare objectRequest() as async because it's not waiting on the server, but I think callServer() should be async, right? I know this can be done with promises, but I wanted to use async/await and this seems to be right.
Now, I want to return the data that comes from inside callServer() and client.on('data', (data) but I can't seem to figure out how to do it asynchronously. I would think there'd be a way to make an async function and call it with await like I tried (await return data) but it never works right.
I'm sorry if this is terribly convoluted, but I've been poring over async node tutorials for the past week and am still stuck.
Thanks!
Async/Await relies on code that uses promises for async operations. So you need to return a promise from any async operation in order to use it with async/await.
So, callServer() needs to return a promise that is resolved when the async operation inside it is done. In fact, you can only await an async operation in a function if that function returns a promise. await saves you from having to write .then() handlers on promises, but async/await does not magically know when async operations are done. You still have to wrap them in promises.
Here's an example of how you could make callServer() return a promise:
async function callServer(formattedJson) {
return new Promise((resolve, reject) => {
let client = new net.Socket()
client.connect(1705, '192.168.28.16', () => {
console.log('connected to server')
client.write(formattedJson)
})
client.on('data', (data) => {
resolve(data);
client.destroy()
})
client.on('close', () => {
})
client.on('error', reject);
});
}
Sample Usage:
try {
var info = await callServer(funcRequestToJson("Server.GetStatus", false));
} catch(e) {
// error here
}
Just to show you what await is doing, in ES5 (without async/await), the sample usage would be this:
callServer(funcRequestToJson("Server.GetStatus", false)).then(info => {
// info is available here
}).catch(err => {
// error here
});
So, await is just letting you avoid writing the .then() handler. Internally in the execution of the code, there's just a promise that the interpreter sets up a .then() handler for so it will know when that promise is resolved. This is syntactical sugar, not really any magic with async operations. You still have to use promises for all your async operations. You can use await instead of .then() and your code will appear more sequential even though it's really the same under the covers.
People think that await allows one to write asynchronous code as if it was synchronous, but that isn't really the case, especially if you handle errors and understand concurrency issues. It will look more synchronous, but isn't actually any more synchronous than it was with .then() in ES5.
And, watch out for error handling. In people's quest to write synchronous looking code, people seem to completely forget to handle rejected promises in their async operations (which are handled with try/catch when using await). I'm personally not yet convinced that ES6 is a step forward when it comes to error handling as the early indications are that ES6 seems to encourage people to just forget about error handling or get lazy and not do it, whereas it's easier when using .then() to just know that there should be a .catch() somewhere for it to be solid code. Maybe that's just a learning process, but it seems to be an early issue when people use await.

Resources