I'm using the Node.js readline interface to read a file line-by-line using an async for-of loop. But I want to be able to control the flow and I'm not sure how to break and continue the loop where it left off.
Simplified example:
const fileStream = fs.createReadStream('input.txt')
const rl = readline.createInterface({
input: fileStream,
crlfDelay: Infinity
})
for await (const line of rl) {
console.log(line) // This works
break
}
for await (const line of rl) {
console.log(line) // This does not print anything
}
See this replit for a complete example.
How do I use the same readline interface to continue the loop where it left off?
Async generators close the underlying stream once you exit the loop. So I went for this instead:
const it = this.lineReader[Symbol.asyncIterator]()
while (true) {
const res = await it.next()
if (res.done) break
line = res.value.trim()
}
Related
Hi so I have this piece of code that I want to run in order, however I always end up running the last line first and then the first one. Some guidance with this would be great.
await fs.readFile(file, "UTF-8", (err, lines) => {
lines = lines.split(/\r?\n/);
lines.forEach( (line) => {
line = line.split('\t');
let book = new Book(line[0], line[1], line[2], line[3],
line[4],line[5],line[6],line[7],line[8],line[9],
line[10],line[11]);
console.log(book);
}
);
});
await pgdb.closeDatabase();
Close database looks like this
closeDatabase = async() => {
console.log("Closed")
this.client.end();
}
Thanks!
You are mixing callbacks and async/await
with callbacks, only the code inside the callback will execute after the file is retrieved. The control flow will immediately jump to the next line.
You are not awaiting anything.
try:
import fs from "fs/promises" //or require()
// code...
const file = await fs.readFile(file, "UTF-8")
// do something with file contents
// do some more sync or async tasks
await pgdb.closeDatabase();
After creating a stream (A), creating another stream (B) and reading stream (B), the reading process stops from the stream (A).
How can I solve this problem?
Node.js v14.18.1
import * as readline from 'readline';
import { Readable } from 'stream';
async function main() {
const streamA = Readable.from('a');
const readerA = readline.createInterface({
input: streamA,
crlfDelay: Infinity
});
var stopCase = false;
if (stopCase) {
const streamB = Readable.from('b');
const readerB = readline.createInterface({
input: streamB,
crlfDelay: Infinity
});
console.log('readB');
for await (const line of readerB) {
console.log(line);
}
}
console.log(`readerA.closed = ${'closed' in readerA}`);
console.log('readA');
for await (const line of readerA) {
console.log(line);
}
console.log('success');
}
main();
Output(stopCase=true):
readB
b
readerA.closed = true
readA
Output(stopCase=false):
readerA.closed = false
readA
a
success
The issue is that as soon as you do this:
const readerA = readline.createInterface({
input: streamA,
crlfDelay: Infinity
});
Then, streamA is now ready to flow and readerA is ready to generate events as soon as you hit the event loop. When you go into the stopCase block and hit the for await (const line of readerB), that will allow streamA to flow which will allow readerA to fire events.
But, you aren't listening for the readerA events when they fire and thus it finishes the streamA content it had while you aren't listening.
You can see how it works better if you don't create readerA until after you're done with the stopCase block. Because then streamA and readerA aren't yet flowing when you hit the await inside of the stopCase block.
This is what I would call a growing pain caused by trying to add promises onto the event driven streams. If you leave the stream in a flowing state and you were going to use await to read those events, but you then await some other promise, all your events on that first stream fire when you aren't yet listening. It doesn't know you're waiting to use await on it. You set it up to flow so as soon as the interpreter hits the event loop, it starts flowing, even though you aren't listening with await.
I've run into this before in my own code and the solution is to not set a stream up to flow until you're either just about to use await to read it or until you have a more traditional event handler configured to listen to any events that flow. Basically, you can't configure two streams for use with for await (...) at the same time. Configure one stream, use it with your for await (...), then configure the other. And, be aware of any other promises used in your processing of the for await (...) loop too. There are lots of ways to goof up when using that structure.
In my opinion, it would work more reliably if a stream was actually put in a different state to be used with promises so it will ONLY flow via the promise interface. Then, this kind of thing would not happen. But, I'm sure there are many challenges with that implementation too.
For example, if you do this:
import * as readline from 'readline';
import { Readable } from 'stream';
async function main() {
var stopCase = true;
console.log(`stopCase = ${stopCase}`);
if (stopCase) {
const streamB = Readable.from('b');
const readerB = readline.createInterface({
input: streamB,
crlfDelay: Infinity
});
console.log('readB');
for await (const line of readerB) {
console.log(line);
}
}
const streamA = Readable.from('a');
const readerA = readline.createInterface({
input: streamA,
crlfDelay: Infinity
});
console.log(`streamA flowing = ${streamA.readableFlowing}`);
console.log(`readerA.closed = ${!!readerA.closed}`);
console.log('readA');
for await (const line of readerA) {
console.log(line);
}
console.log('success');
}
main();
Then, you get all the output:
stopCase = true
readB
b
streamA flowing = true
readerA.closed = false
readA
a
success
The reason you never get the console.log('success') is probably because you hit the for await (const line of readerA) { ...} loop and it gets stopped there on a promise that has no more data. Meanwhile, nodejs notices that there is nothing left in the process that can create any future events so it exits the process.
You can see that same concept in play in an even simpler app:
async function main() {
await new Promise(resolve => {
// do nothing
});
console.log('success');
}
main();
It awaits a promise that never completes and there are no event creating things left in the app so nodejs just shuts down with ever logging success.
I have a string with the alphabet and certain letters are in parenthesis. It means these letters should be uppercase. But the user has to be asked first if he wants it.
The problem is, I have to call readline inside a loop and this doesn't work.
const readline = require('readline');
const text = 'a(b)cdefg(h)ijklmnopqrst(u)vwxyz';
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
text.replace(/\((\w)\)/g, function (match, p1) {
let answer;
rl.question(`Do you want to change ${p1}? [yes/no] `, input => {
answer = input;
});
return answer === "yes" ? p1.toUpperCase() : p1;
});
console.log("Result:", text);
It just prints out Do you want to change b? [yes/no] Result: a(b)cdefg(h)ijklmnopqrst(u)vwxyz.
How can this be solved?
As Nick has explained in the comments the main issue of your code it that it does not handle the async readline operation correctly - here's how I would do this:
use a promisifed version of readline to prevent nested callbacks
use a while loop to step through all regex matches and await the user input in order to convert the chars to upper case or not
Something like (still needs error handling, but should give you an idea):
const readline = require('readline').createInterface({
input: process.stdin,
output: process.stdout
});
function question(query) {
return new Promise(resolve => {
readline.question(query, resolve);
})
}
const specialCharsRegex = /\((\w)\)/g;
const text = 'a(b)cdefg(h)ijklmnopqrst(u)vwxyz';
(async () => {
let result = text;
let currMatch;
while ((currMatch = specialCharsRegex.exec(text)) !== null) {
const convertInput = await question(`Do you want to change ${currMatch[1]}? [yes/no] `);
result = result.replace(currMatch[0], v => {
return convertInput === "yes" ? currMatch[1].toUpperCase() : currMatch[1];
});
}
console.log("Result: ", result);
readline.close();
})();
The main issue with your code is that you're trying to do something asynchronous in a callback that is expected to be synchronous - that is, you're trying to read user input in the .replace() callback, but reading user input doesn't happen immediately, so the callback to .question() occurs sometime after your question has been asked and your replace callback has terminated/finished. Making rl.question() synchronous would fix this (Node.js synchronous prompt) and would allow you to prompt against the exact words you're replacing.
If you want to keep things asynchronous, you could first match the letters you want to replace using .matchAll(), that way you'll have the text/groups, and the indexes of the characters you want to change. Then, you can use a loop to iterate the matches, and prompt the user whether they want to change the group. For each group, you can update the text accordingly by updating the text at the specified index. You could use replace, but if your text has duplicates eg: "a(b)c(b)e", and you said "no" for the first (b) but yes for the second (b), then you'll either need to replace both occurrences, or the first match. By using the index, you can specify exactly which group to change:
const readline = require('readline');
const rl = readline.createInterface({
input: process.stdin, //or fileStream
output: process.stdout
});
const question = q => new Promise(res => rl.question(q, res));
let text = 'a(b)cdefg(h)ijklmnopqrst(u)vwxyz';
(async () => {
const matches = text.matchAll(/\((\w)\)/g);
for(const {0: match, 1: p1, index} of matches) {
const ans = await question(`Do you want to change ${p1}? [yes/no] `);
if(ans === "yes")
text = text.slice(0, index) + match.toUpperCase() + text.slice(index + match.length);
}
console.log(text);
})();
I built a simple asynchronous implementation of the readlines module built into nodejs, which is simply a wrapper around the event-based module itself. The code is below;
const readline = require('readline');
module.exports = {
createInterface: args => {
let self = {
interface: readline.createInterface(args),
readLine: () => new Promise((succ, fail) => {
if (self.interface === null) {
succ(null);
} else {
self.interface.once('line', succ);
}
}),
hasLine: () => self.interface !== null
};
self.interface.on('close', () => {
self.interface = null;
});
return self;
}
}
Ideally, I would use it like so, in code like this;
const readline = require("./async-readline");
let filename = "bar.txt";
let linereader = readline.createInterface({
input: fs.createReadStream(filename)
});
let lines = 0;
while (linereader.hasLine()) {
let line = await linereader.readLine();
lines++;
console.log(lines);
}
console.log("Finished");
However, i've observed some erratic and unexpected behavior with this async wrapper. For one, it fails to recognize when the file ends, and simply hangs once it reaches the last line, never printing "Finished". And on top of that, when the input file is large, say a couple thousand lines, it's always off by a few lines and doesn't successfully read the full file before halting. in a 2000+ line file it could be off by as many as 20-40 lines. If I throw a print statement into the .on('close' listener, I see that it does trigger; however, the program still doesn't recognize that it should no longer have lines to read.
It seems that in nodejs v11.7, the readline interface was given async iterator functionality and can simply be looped through with a for await ... of loop;
const rl = readline.createInterface({
input: fs.createReadStream(filename);
});
for await (const line of rl) {
console.log(line)
}
How to get synchronous readline, or "simulate" it using async, in nodejs?
Im reading a text file in NodeJs using readline module.
var lineReader = require('readline').createInterface({
input: require('fs').createReadStream('log.txt')
});
lineReader.on('line', function (line) {
console.log(line);
});
lineReader.on('close', function() {
console.log('Finished!');
});
Is there any way to set the time of the reading?
For example i want to read each line every 5msec.
You can pause the reader stream as soon as you read a line. Then resume it 5ms later. Repeat this till the end of file. Make sure to adjust highWaterMark option to a lower value so that the file reader stream doesn't read multiple lines at once.
var lineReader = require('readline').createInterface({
input: require('fs').createReadStream('./log.txt',{
highWaterMark : 10
})
});
lineReader.on('line', line => {
lineReader.pause(); // pause reader
// Resume 5ms later
setTimeout(()=>{
lineReader.resume();
}, 5)
console.log(line);
});
You can use observables to do this. Here's an example of the kind of buffering I think you want with click events instead of file line events. Not sure if there's a cleaner way to do it that avoids the setInterval though....
let i = 0;
const source = Rx.Observable
.fromEvent(document.querySelector('#container'), 'click')
.controlled();
var subscription =
source.subscribe(() => console.log('was clicked ' + i++));
setInterval(() => source.request(1), 500);
Here's a fiddle and also a link to docs for rx:
https://jsfiddle.net/w6ewg175/
https://github.com/Reactive-Extensions/RxJS/blob/master/doc/api/core/operators/controlled.md