How to create a file stream and write to the stream asynchronously? - node.js

I am new to TypeScript/JavaScript and Node.
Now I am trying to create a file stream and write "Hello!" to the stream asynchronously.
#!/usr/bin/env node
import fs from 'fs';
function createStream(filePath: string): Promise<fs.WriteStream> {
return new Promise<fs.WriteStream>((resolve, reject) => {
const out = fs.createWriteStream(filePath);
out.on('close', () => {
console.log(filePath + ' closed');
resolve(out);
});
out.on('error', (err: any) => {
console.log(filePath + ' ' + err);
reject(err);
});
});
}
createStream('/tmp/test.txt').then((out:fs.WriteStream) => {
console.log(out);
out.write('Hello!');
out.end();
})
This code does create /tmp/test.txt but prints out nothing and the file is empty.
What is the problem with this code ?

You don't need to resolve a promise with the fs.WriteStream since its creation is synchronous. Just call fs.createWriteStream() directly and pass the instance to your function to create a promise that settles when the stream closes or errors:
#!/usr/bin/env node
import fs from 'fs';
import stream from 'stream';
function promisify(s: stream.Stream): Promise<void> {
return new Promise<void>((resolve, reject) => {
const onClose = () => {
s.off('error', onError);
resolve();
};
const onError = (error: Error) => {
s.off('close', onClose);
reject(error);
};
s.once('close', onClose);
s.once('error', onError);
});
}
const out = fs.createWriteStream('/tmp/test.txt');
promisify(out).then(() => {
console.log('Done');
});
out.write('Hello!');
out.end();

Related

graphql-upload-minimal: how to destroy the stream when upload file size limit is reached?

On a Node.js GraphQL API, using express + #graphql-yoga/node + graphql-upload-minimal, I have the uploads working very well, but when I upload a huge file, and the upload file size limit is reached, the stream continues until be finished, and it imposes makes unnecessary waiting to finish the entire stream.
I tried the below code, but the 'reject' does't destroy the stream:
stream.on('limit', function () {
const incomplete_file = `${folder}/${my_filename}`;
fs.unlink(incomplete_file, function () {});
reject(
new GraphQLYogaError(`Error: Upload file size overflow`, {
code: 'UPLOAD_SIZE_LIMIT_OVERFLOW',
}),
);
});
Full module below:
import getFilename from './getFilename';
import fs from 'fs';
import { GraphQLYogaError } from '#graphql-yoga/node';
export default async function uploadSingleFile(
folder: string,
file: any,
): Promise<any> {
return new Promise(async (resolve, reject) => {
const { createReadStream, filename, fieldName, mimetype, encoding } =
await file;
const my_filename = getFilename(filename, fieldName);
let size = 0;
let stream = createReadStream();
stream.on('data', function (chunk: any) {
size += chunk.length;
fs.appendFile(`${folder}/${my_filename}`, chunk, function (err) {
if (err) throw err;
});
});
stream.on('close', function () {
resolve({
nameOriginal: filename,
nameUploaded: my_filename,
mimetype: mimetype,
});
});
stream.on('limit', function () {
const incomplete_file = `${folder}/${my_filename}`;
fs.unlink(incomplete_file, function () {});
reject(
new GraphQLYogaError(`Error: Upload file size overflow`, {
code: 'UPLOAD_SIZE_LIMIT_OVERFLOW',
}),
);
});
})
.then((data) => data)
.catch((e) => {
throw new GraphQLYogaError(e.message);
});
}
How I can force imediate end of stream? There is any method to destroy the stream?
Thanks for the help!

How to use promisify() with the spawn() function for the 'child_process'?

I have the following code example, and I have now ideas on how to resolve this using utils.promisify(); ONLY! Not Promise!
const spawn = child_process.spawn('docker', ['--version']);
spawn.stdout.on('data', (data) => {
process.stdout.write(data);
});
spawn.on('error', () => {
process.stderr.write(error);
process.exit(1);
});
The second code example works fine.
const promisifiedExecFile = promisify(child_process.execFile);
async function test() {
const version = await promisifiedExecFile('docker', ['--version']);
console.log(version);
}
test();
I couldn't quickly find out why the promisify function does now work properly with the spawn function. But you can create your own promisify function like this:
TS
import { spawn as spwn } from 'child_process';
const spawn = (
cmd: string,
args: ReadonlyArray<string>,
) => new Promise((resolve, reject) => {
const cp = spwn(cmd, args);
const error: string[] = [];
const stdout: string[] = [];
cp.stdout.on('data', (data) => {
stdout.push(data.toString());
});
cp.on('error', (e) => {
error.push(e.toString());
});
cp.on('close', () => {
if (error.length) reject(error.join(''));
else resolve(stdout.join(''));
});
});
(async () => {
try {
const stdOut = await spawn('docker', ['--version']);
console.log('stdOut: ', stdOut);
} catch (error) {
console.log('error:', error);
process.exit(1);
}
})();
JS
const { spawn: spwn } = require('child_process');
const spawn = (
cmd,
args,
) => new Promise((resolve, reject) => {
const cp = spwn(cmd, args);
const error = [];
const stdout = [];
cp.stdout.on('data', (data) => {
stdout.push(data.toString());
});
cp.on('error', (e) => {
error.push(e.toString());
});
cp.on('close', () => {
if (error.length) reject(error.join(''));
else resolve(stdout.join(''));
});
});
(async () => {
try {
const stdOut = await spawn('docker', ['--version']);
console.log('stdOut: ', stdOut);
} catch (error) {
console.log('error: ', error);
process.exit(1);
}
})();
Node.js' built-in util package has a promisify() function that converts callback-based functions to promise-based functions. This lets you use promise chaining and async/await with callback-based APIs.
I think that we can't use the promisify() with the spawn() function.
For example we can use promisify() with execFile() instead of spawn():
async asyncExecFile(tool) {
const execFile = util.promisify(child_process.execFile);
return await execFile(tool, ['--version'])
.catch(() => {
this.printError(`The "${tool}" don't exist in the current environment. \n`);
process.exit(0);
});
}
It is not possible because there is nothing to promisify. promisify works on functions where it takes a callback and spawn does not take a callback.
You use spawn by taking the returned ChildProcess then adding listeners to the ChildProcess' readable streams (stdout, stderr, stdio...)
Omar Omeiri's answer is similar to how execFile works inside node itself, so you can just use the promisified execFile instead. (if you need unlimited buffer, pass in maxBuffer: Infinity inside options)

how to access a local variable value outside the scope of its function in nodejs

I want to compare the data of two files and for that, I'm reading that file using the fs module but since I want to compare the values so I thought to store the value in an external variable but when I do console.log(budget_details) I get nothing in console. Please someone help. Please point me out if my approach is wrong and if we don't need to do that in nodejs. I'm new to nodejs.
import csv from 'csv-parser'
import fs from 'fs';
let budget_details
const budgetProcessing = (budget_file_path) => {
try{
fs.createReadStream(budget_file_path)
.pipe(csv())
.on('data', (row) => {
budget_details = row
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
catch(error){
console.log(error)
}
}
budgetProcessing('budget.csv')
console.log(budget_details)
Let's first explain why you don't get the expected result, it doesnt have to do with scope actually:
import csv from 'csv-parser'
import fs from 'fs';
let budget_details
const budgetProcessing = (budget_file_path) => {
try{
fs.createReadStream(budget_file_path)
.pipe(csv())
.on('data', (row) => {
budget_details = row
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
catch(error){
console.log(error)
}
}
budgetProcessing('budget.csv')
console.log(budget_details)
fs.createReadStream is not itslef exactly asynchronous but then we pipe the returned stream to csv-parser which does event based parsing, so even if you call budgetProcessing before the console.log(budget_details) the stream reading has most likely not runned yet and budget_details is still undefined.
To fix this, you could move this console.log(budget_details) where it is set like so:
let budget_details
const budgetProcessing = (budget_file_path) => {
try{
fs.createReadStream(budget_file_path)
.pipe(csv())
.on('data', (row) => {
budget_details = row
console.log(budget_details)
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
catch(error){
console.log(error)
}
}
budgetProcessing('budget.csv')
but then the variable itself wouldnt serve any real purpose so instead you could do this:
const budgetProcessing = (budget_file_path, callback) => {
try{
fs.createReadStream(budget_file_path)
.pipe(csv())
.on('data', (row) => {
callback(row)
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
catch(error){
console.log(error)
}
}
budgetProcessing('budget.csv', (budget_details) => {
console.log(budget_details) // or anything with budget_details
})
Lastly, I want to make clear that the callback will be called for each row of the csv as specified in csv-parser's documentation
your code is not asynchronous. Anything with 'on', which takes a function, would indicate that it is event driven. You need something like:
import csv from 'csv-parser'
import fs from 'fs';
let budget_details
const budgetProcessing = (budget_file_path) => new Promise((resolve, reject) => {
try {
fs.createReadStream(budget_file_path)
.pipe(csv())
.on('data', (row) => {
budget_details = row
})
.on('end', () => {
console.log('CSV file successfully processed');
resolve()
});
} catch (error) {
console.log(error)
reject(error)
}
})
budgetProcessing('budget.csv')
.then(() => console.log(budget_details))

In node.js, why is my data not getting passed back after a asynchronous file read using a Promise

I know for sure that my pullData module is getting the data back from the file read but the function calling it, though it has an await, is not getting the data.
This is the module (./initialise.js) that reads the data:
const fs = require('fs');
const getData = () => {
return new Promise((resolve, reject) => {
fs.readFile('./Sybernika.txt',
{ encoding: 'utf8', flag: 'r' },
function (err, data) {
if (err)
reject(err);
else
resolve(data);
});
});
};
module.exports = {getData};
And this is where it gets called (app.js):
const init = require('./initialise');
const pullData = async () => {
init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
};
const start = async() => {
let data = await pullData();
console.log(data);
}
start();
putting 'console.log(data)' just before return(data) in the resolve part of the call shows the data so I know it's being read OK. However, that final console.log shows my data variabkle as being undefined.
Any suggestions?
It's either
const pullData = async () => {
return init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
};
or
const pullData = async () =>
init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
Both versions make sure a promise returned by then/catch is passed down to the caller.

nodejs async await inside createReadStream

I am reading a CSV file line by line and inserting/updating in MongoDB. The expected output will be
1. console.log(row);
2. console.log(cursor);
3.console.log("stream");
But getting output like
1. console.log(row);
console.log(row); console.log(row); console.log(row); console.log(row); ............ ............
2. console.log(cursor);
3.console.log("stream");
Please let me know what i am missing here.
const csv = require('csv-parser');
const fs = require('fs');
var mongodb = require("mongodb");
var client = mongodb.MongoClient;
var url = "mongodb://localhost:27017/";
var collection;
client.connect(url,{ useUnifiedTopology: true }, function (err, client) {
var db = client.db("UKCompanies");
collection = db.collection("company");
startRead();
});
var cursor={};
async function insertRec(row){
console.log(row);
cursor = await collection.update({CompanyNumber:23}, row, {upsert: true});
if(cursor){
console.log(cursor);
}else{
console.log('not exist')
}
console.log("stream");
}
async function startRead() {
fs.createReadStream('./data/inside/6.csv')
.pipe(csv())
.on('data', async (row) => {
await insertRec(row);
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
In your startRead() function, the await insertRec() does not stop more data events from flowing while the insertRec() is processing. So, if you don't want the next data event to run until the insertRec() is done, you need to pause, then resume the stream.
async function startRead() {
const stream = fs.createReadStream('./data/inside/6.csv')
.pipe(csv())
.on('data', async (row) => {
try {
stream.pause();
await insertRec(row);
} finally {
stream.resume();
}
})
.on('end', () => {
console.log('CSV file successfully processed');
});
}
FYI, you also need some error handling if insertRec() fails.
That is expected behavior in this case because your on data listener triggers the insertRec asynchronously as and when data is available in stream. So that is why your first line of insert method is getting executed kind of in parallel. If you want to control this behavior you can use highWaterMark (https://nodejs.org/api/stream.html#stream_readable_readablehighwatermark) property while creating the read stream. This way you will get 1 record at a time but I am not sure what your use case is.
something like this
fs.createReadStream(`somefile.csv`, {
"highWaterMark": 1
})
Also you are not awaiting your startRead method. I would wrap it inside the promise and resolve it in end listener else you will not know when the processing got finished. Something like
function startRead() {
return new Promise((resolve, reject) => {
fs.createReadStream(`somepath`)
.pipe(csv())
.on("data", async row => {
await insertRec(row);
})
.on("error", err => {
reject(err);
})
.on("end", () => {
console.log("CSV file successfully processed");
resolve();
});
});
}
From Node 10+ ReadableStream got property Symbol.asyncIterator and is's allow processing stream using for-await-of
async function startRead() {
const readStream = fs.createReadStream('./data/inside/6.csv');
for await (const row of readStream.pipe(csv())) {
await insertRec(row);
}
console.log('CSV file successfully processed');
}

Resources