In the following code, the csv2pg function is async
const postUsages = async (req: MulterRequest, res: Response, next: any) => {
try {
const result = await csv2pg(req, next);
res.status(200).json({
msg: 'File uploaded/import successfully!',
file: req.file,
});
} catch (err) {
res.status(400).json({
msg: 'File uploaded/import failed!',
file: req.file,
});
}
};
The problem is, the await doesn't seem to await the function being finished before returning the res.status(200)
Here is the content of the csv2pg function that is being called (the async bit in this code is the forEach and the pool.query)
const csv2pg = (req: MulterRequest, next: any): any => {
let provider_id = req.body!.provider_id;
const filePath = appRoot + '/reports/' + req.file.filename;
const stream = fs.createReadStream(filePath);
const csvData: any[] = [];
const csvStream = csv
.parse()
.on('data', (data: any) => {
csvData.push(data);
})
.on('error', (err: any) => {
throw err.message;
})
.on('end', () => {
csvData.shift();
csvData.forEach((row) => {
pool.query(
`INSERT INTO usage (date_and_time, consumption, reading_quality, provider_id)
VALUES ((TO_TIMESTAMP($1, 'DD/MM/YYYY HH24:MI') AT TIME ZONE 'Australia/Melbourne')::TIMESTAMP WITH TIME ZONE, $2, $3, ${provider_id})`,
row,
(err: any) => {
if (err) {
next(new Error(err));
}
}
);
});
fs.unlinkSync(filePath);
});
stream.pipe(csvStream);
};
Another problem with this is that when there's an issue, there's another error message on top saying that the headers were already set (of course, the server already returned to the client by that time)
Ideally, when an error is raised, the whole function should stop, return the error message, and wait for new incoming requests
In order to be able to wait for csv2pg to finish all async operations, it must return a promise. Since the async actions are in a loop, we have to use Promise.all. Try this:
const csv2pg = (req: MulterRequest, next: any): any => {
return new Promise((resolve, reject) => {
let provider_id = req.body.provider_id;
const filePath = appRoot + "/reports/" + req.file.filename;
const stream = fs.createReadStream(filePath);
const csvData: any[] = [];
const csvStream = csv
.parse()
.on("data", (data: any) => {
csvData.push(data);
})
.on("error", (err: any) => {
reject(err.message);
})
.on("end", () => {
csvData.shift();
const promisesArray = csvData.map((row) => {
return new Promise((resolve, reject) =>
pool.query(
`INSERT INTO usage (date_and_time, consumption, reading_quality, provider_id)
VALUES ((TO_TIMESTAMP($1, 'DD/MM/YYYY HH24:MI') AT TIME ZONE 'Australia/Melbourne')::TIMESTAMP WITH TIME ZONE, $2, $3, ${provider_id})`,
row,
(err: any) => {
if (err) {
reject(new Error(err));
} else {
resolve();
}
}
)
);
});
Promise.all(promisesArray).then(() => {
fs.unlinkSync(filePath);
resolve();
})
});
stream.pipe(csvStream);
});
};
Please note, I also wrapped your pool.query in a promise, since this function doesn't return a promise by default.
The reason could be that you are using this code inside the for-loop. Using Promise.all would fix the problem. I would recommend this post to you: https://www.freecodecamp.org/news/promise-all-in-javascript-with-example-6c8c5aea3e32/.
It explains everything in detail.
Related
I know for sure that my pullData module is getting the data back from the file read but the function calling it, though it has an await, is not getting the data.
This is the module (./initialise.js) that reads the data:
const fs = require('fs');
const getData = () => {
return new Promise((resolve, reject) => {
fs.readFile('./Sybernika.txt',
{ encoding: 'utf8', flag: 'r' },
function (err, data) {
if (err)
reject(err);
else
resolve(data);
});
});
};
module.exports = {getData};
And this is where it gets called (app.js):
const init = require('./initialise');
const pullData = async () => {
init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
};
const start = async() => {
let data = await pullData();
console.log(data);
}
start();
putting 'console.log(data)' just before return(data) in the resolve part of the call shows the data so I know it's being read OK. However, that final console.log shows my data variabkle as being undefined.
Any suggestions?
It's either
const pullData = async () => {
return init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
};
or
const pullData = async () =>
init.getData().then((data) => {
return data;
}).catch((err) => {
console.log(err);
});
Both versions make sure a promise returned by then/catch is passed down to the caller.
I am trying to set up a route that sends data from a nested promise to my vue app.
But i'm having trouble with the getting data from the nested promises.
i tried using a callback with no success
app.get('/notification', (req, res) => {
const getData = (data) => {
console.log(data)
}
scheduler(data)
})
const scheduler = (callback) => {
sftp
.connect({ credentials })
.then(() => {
return sftp.list(root);
})
.then(async data =>
{
const filteredFile = data.filter(file => {
let currentDate = moment();
let CurrentTimeMinusFive = moment().subtract(5, "minutes");
let allAccessTimes = file.accessTime;
let parsedAccessTimes = moment(allAccessTimes);
let filteredTime = moment(parsedAccessTimes).isBetween(
CurrentTimeMinusFive,
currentDate
);
return filteredTime;
});
for (const file of filteredFile) {
let name = file.name;
let filteredThing;
await sftp
.get(`Inbound/${name}`)
.then(data => {
csv()
.fromString(data.toString())
.subscribe(function (jsonObj) {
return new Promise(function (resolve, reject) {
filteredThing = new Notification(jsonObj);
filteredThing.save()
.then(result => {
console.log(result);
callback(result) **// THIS IS THE RESULT I NEED IN MY FRONT END**
})
.catch(err => {
console.log(err);
});
resolve();
});
});
});
}
})
When i go to localhost/notification i get:
ReferenceError: data is not defined
Thanks in advance!
So i encountered a problem while doing my project.The problem is that when i try to write my data to csv file,it only write half of the data ,even sometimes only less than half of my data.I don't know what the problem is because there is no error shown in the terminal.
Below is my code
async function getFile(req, res, next) {
try {
let URI;
const listOfKeys = await listAllKeys();
let temp = []
await Promise.all(listOfKeys.map(async function (data) {
let response = await writeFile(data.Key);
temp.push(response)
}))
.then(async _ => {
fs.writeFileSync(FILE_PATH, Buffer.concat(temp));
})
.catch(err => {
console.log(err)
})
return res.json({ message: 'halo' });
} catch (err) {
console.log('hmm.... error', err);
return next(new APIError(err, httpStatus.INTERNAL_SERVER_ERROR, true));
};
};
And this is the writeFile function
function writeFile(key) {
return new Promise((resolve, reject) => {
s3.getObject({ Bucket: process.env.AWS_BUCKET, Key: key }, (err, data) => {
if (err) reject(err)
else resolve(data.Body)
})
});
};
If possible, i would like to know the detail of my problem and how to fix it.Thanks.
It looks to me like you can do it like this (function names have been modified to make sense to me):
const fsp = require('fs').promises;
async function getDataAndWriteFile(req, res, next) {
try {
let URI;
const listOfKeys = await listAllKeys();
let responses = await Promise.all(listOfKeys.map(function (data) {
return getData(data.Key);
}));
await fsp.writeFile(FILE_PATH, Buffer.concat(responses);
res.json({ message: 'halo' });
} catch(err) {
console.log('hmm.... error', err);
next(new APIError(err, httpStatus.INTERNAL_SERVER_ERROR, true));
}
}
function getData(key) {
return new Promise((resolve, reject) => {
s3.getObject({ Bucket: process.env.AWS_BUCKET, Key: key }, (err, data) => {
if (err) reject(err)
else resolve(data.Body)
})
});
}
Summary of changes:
Change function names to better describe what they do
Use let responses = await Promise.all() to get the data from the promise array.
Use the promise interface in the fs module with await fsp.writeFile() to write the data out to your file.
Use try/catch to catch all the promise rejections in one place
Possible Open Issues:
Writing this Buffer.concat(responses) to disk seems kind of odd. Is that really what you want in this file?
I am trying to get the array after it is filled in a function but it is empty when I print it out.
Code:
var usersList = [];
app.post('/submitLogin',function(req,res) {
getUsersGroups();
console.log(usersList);
});
function getUsersGroups() {
const users = new Promise((resolve, reject) => {
dbConnection
.getUsers()
.then(data => {
resolve(data)
})
});
const groups = new Promise((resolve, reject) => {
dbConnection
.getGroups()
.then(data => {
resolve(data)
})
});
Promise.all([users, groups])
.then(data => {
usersList = data[0];
groupsList = data[1];
console.log(usersList)
});
}
However, the console.log in the getUsersGroups(), prints out the filled array but it is empty in the app.post('/submitLogin...')
Why is this happening assuming that getUsersGroups() runs before I try to print out the array?
You are not observing the async execution of your db call.
The console.log(usersList); happens BEFORE the db call, so it is empty.
You must adapt your code like this and post the code AFTER the db execution:
app.post('/submitLogin', function (req, res) {
getUsersGroups().then((data) => {
console.log(data)
res.send(data.usersList)
})
})
function getUsersGroups () {
const users = new Promise((resolve, reject) => {
dbConnection
.getUsers()
.then(data => {
resolve(data)
})
})
const groups = new Promise((resolve, reject) => {
dbConnection
.getGroups()
.then(data => {
resolve(data)
})
})
return Promise.all([users, groups])
.then(data => {
console.log(data[0])
return {
usersList: data[0],
groupsList: data[1]
}
})
}
I strongly suggest to don't change a global variable like usersList (that I removed from my example)
because if you receive two request contemporaneously, the second one could overwrite the data of the first one and
cause many side effect.
app.post('/submitLogin', async function(req,res) {
await getUsersGroups();
console.log(usersList);
});
try this
I am trying to download tracks via the soundcloud API, and then launch a callback once an indeterminant amount of tracks is downloaded. When I run the below code, I see "All done" being console logged before anything else, even though I intend for it to be the last thing... What am I doing wrong?
// Deps
import fs from 'fs'
import SC from 'node-soundcloud'
import request from 'request'
// Write mp3 function
function writeMP3(track) {
return new Promise((resolve, reject) => {
console.log('Starting download: ', track.title)
request.get(track.download_url)
.on('error', err => {
// reject('Download error: ', err)
})
.on('finish', () => {
() => resolve('Download complete')
})
.pipe(fs.createWriteStream(`./data/temp/${track.title}_${track.user.username}.mp3`))
})
}
async function asyncTrackFetch(track) {
return await writeMP3(track)
}
// Array of promises to callback upon
const trackActions = []
SC.init({
id: 'MY_ID',
secret: 'MY_SECRET'
})
SC.get('/tracks', (err, tracks) => {
if (err) {
throw new Error(err)
} else {
console.log('Tracks fetched: ', tracks.length)
tracks.map(track => {
if (track.downloadable) {
console.log('downloadable')
trackActions.push(asyncTrackFetch(track))
}
})
}
})
// Perform requests async
Promise.all(trackActions).then(() => {
console.log('All done')
console.log(fs.readdirSync('./data/temp'))
})
Promise.all(trackActions) waits on whatever promises are in trackActions, but trackActions is empty at the time you make the call. You're only adding promises to the array after your SC.get callback gets called.
Try putting your Promise.all... block inside the SC.get callback like this:
SC.get('/tracks', (err, tracks) => {
if (err) {
throw new Error(err)
} else {
console.log('Tracks fetched: ', tracks.length)
tracks.map(track => {
if (track.downloadable) {
console.log('downloadable')
trackActions.push(asyncTrackFetch(track))
}
})
Promise.all(trackActions).then(() => {
console.log('All done')
console.log(fs.readdirSync('./data/temp'))
})
}
})
It's worth mentioning as well that your line throw new Error(err) will crash the program since there's nowhere for that error to be caught.
As Antonio Val mentioned, there are better ways to do this. If you promisify the node-soundcloud library then the last part of your code could look like this:
SC.get('/tracks').then(tracks => {
// No need for trackedActions array.
return Promise.all(tracks.filter(track => track.downloadable)
.map(track => asyncTrackFetch(track)))
}).then(fetchedTracks => {
console.log('All done fetching tracks', fetchedTracks)
}).catch(err => {
// Handle error.
})
Or inside an async function,
try {
const tracks = await SC.get('/tracks')
const fetchPromises = tracks
.filter(track => track.downloadable)
.map(track => asyncTrackFetch(track))
const fetchedTracks = await Promise.all(fetchPromises)
console('All done fetching tracks.', fetchedTracks)
} catch (err) {
// Handle error
}
I think the easiest way would be to move Promise.all after tracks.map loop finished.
A more elegant solution would be to promisify SC.get as well and use async await along all your code.
UPDATE:
Couldn't test it so not sure if it works, but it would be something like this:
import fs from 'fs'
import SC from 'node-soundcloud'
import request from 'request'
function writeMP3(track) {
return new Promise((resolve, reject) => {
console.log('Starting download: ', track.title)
request.get(track.download_url)
.on('error', err => {
// reject('Download error: ', err)
})
.on('finish', () => {
() => resolve('Download complete')
})
.pipe(fs.createWriteStream(`./data/temp/${track.title}_${track.user.username}.mp3`))
})
}
function getTracks() {
return new Promise((resolve, reject) => {
SC.get('/tracks', (err, tracks) => {
if (err) {
return reject(err)
}
console.log('Tracks fetched: ', tracks.length)
resolve(tracks)
})
})
}
SC.init({
id: 'MY_ID',
secret: 'MY_SECRET'
})
With async await:
async function start() {
const tracks = await getTracks();
for (let track of tracks) {
await writeMP3(track)
}
}
start()
.then(() => {
console.log('All done')
console.log(fs.readdirSync('./data/temp'))
})
.catch((err) => {
// insert error handler here
})
If you just want to use Promises:
getTracks
.then((tracks) => {
const promiseArray = tracks.map((track) => {
return writeMP3(track)
})
return Promise.all(promiseArray)
})
.then(() => {
console.log('All done')
console.log(fs.readdirSync('./data/temp'))
})
.catch((err) => {
// insert error handler here
})