I have an simple REST application and I want to read files in my directory and send them back to frontend. There's code I'm using:
const fs = Promise.promisifyAll(require('fs'))
const router = require('express').Router()
router.get('/list', async (req, res, next) => {
const files = await fs.readdirAsync('presentations')
res.json(files)
})
The problem is: my frontend receive 'Promise', but if I try to debug it shows me that files is an array.
I've tried not to use async/await syntax like that:
router.get('/list', (req, res, next) => {
fs.readdirAsync('presentations')
.then(files => {
res.json(files)
})
})
But result was the same: frontend still get Promise.
UPD: Problem was with frontend axios instance. It didn't resolve promise, so await for results solved a problem.
So, there are three parts. Reading, storing and sending.
Here's the reading part:
var fs = require('fs');
function readFiles(dirname, onFileContent, onError) {
fs.readdir(dirname, function(err, filenames) {
if (err) {
onError(err);
return;
}
onFileContent(filename);
});
});
});
}
Here's the storing part:
var data = {};
readFiles('dirname/', function(filename) {
data[filename] = filname;
}, function(err) {
throw err;
});
The sending part is up to you. You may want to send them one by one or after reading completion.
If you want to send files after reading completion you should either use sync versions of fs functions or use promises. Async callbacks is not a good style.
Related
I can set the value for a key in my Redis server (when I flushall, run this code, and then get key in redis-cli, I get back the proper values), but when I try to get key values through my NodeJs server, it never even logs out 'got data'.
I considered maybe this was because these functions were running asynchronously, and I was asking to get values that weren't yet stored in the cache, but that does not explain why it wouldn't print 'got data' ever.
My console logs-> 'start'->'data saved'->'end' (and no 'got data', ever)
In redis-cli-> flushall->get test->(nil)->run app.js(in the other terminal)->get test->"1, 2, 3, 4, 5"
I get no errors at all, the code runs, but does not do what I want it to.
Also, don't know if this is relevant, but when connecting to the Redis server, just Redis.createClient() only created a client but did not connect and when I looked it up, the general idea I got was that newer versions did not connect automatically and you had to manually redisClient.connect().
I struggled with this a bit at the start but seemed to have sorted this problem, but just thought I'd mention it, if I messed up somehow, please correct me, as I'm pretty new to NodeJs and codng in general.
My code:
const redisClient = Redis.createClient();
redisClient.connect();
const data = [1, 2, 3, 4, 5];
app.get('/', async(req, res, next) => {
console.log('start')
await redisClient.set('test', data);
console.log('data saved');
await redisClient.get('test', (error, test)=>{
console.log('got data');
console.log(test);
});
console.log('end');
});
Thanks!
I have seen your code. Based on my investigation you should remove the callback and keep await only while you get the data from redis.
I have investigated more this issue and have found that client.get() and client.set() function runs asynchronously. Hence it would achieve this way.
client.set('foo', 'bar', (err, reply) => {
if (err) throw err;
console.log(reply);
client.get('foo', (err, reply) => {
if (err) throw err;
console.log(reply);
});
});
But everytime is not the same use-case that we should set and get the value immediately.
To get rid of this, Following are the options.
Promises and async/await
you can promisify a subset of node_redis functions one at a time using native Node.js promises and util.promisify:
example:
const redis = require('redis');
const { promisify } = require('util');
const runApplication = async () => {
const client = redis.createClient();
const setAsync = promisify(client.set).bind(client);
const getAsync = promisify(client.get).bind(client);
await setAsync('foo', 'bar');
const fooValue = await getAsync('foo');
console.log(fooValue);
};
I have used the await here and solve an issue. In addition to that you can use redis.get().then() also to fetch the data rather than a callback.
I am also attaching the link with an example provided by redis repo
https://github.com/redis/node-redis/blob/master/examples/connect-as-acl-user.js
Following is the code, I have tested and it is working fine now.
redis.js
const redis = require("redis");
const redisClient = redis.createClient({
url: "redis://host:6379",
password: "password",
});
redisClient.connect();
// const { promisify } = require("util");
// promisify(redisClient.get).bind(redisClient);
// promisify(redisClient.set).bind(redisClient);
module.exports = redisClient;
index.js
const express = require("express");
const app = express();
const redisClient = require("./redis");
app.get("/set", async (req, res, next) => {
try {
const data = req.query.p;
await redisClient.set("test", data);
res.status(200).json({
message: "data cached",
data: data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.get("/get", async (req, res, next) => {
try {
// const data = await redisClient.get("test");
const data = await reddisClient.get("test").then((data) => {
return data;
});
res.status(200).json({
message: "Cached data retrieved",
data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.listen(process.env.PORT || 3000, () => {
console.log("Node server started");
});
Please find attached a screenshot of the output.
So the final thought is that, when we are using callback and wants to execute the code synchronously you should either use callback inside callback (but it is created callback hell, so it would not suggested anymore) or you should use promise/async await/native promisify library of nodejs.
Please visit below link to get the simplest understanding and example.
https://docs.redis.com/latest/rs/references/client_references/client_nodejs/
Hope my question clear your mind. I am happy to accept the relevant suggestion to improve an answer.
I'm building a Slackbot that makes a call to an Express app, which then needs to 1) fetch some other data from the Slack API, and 2) insert resulting data in my database. I think I have the flow right finally using async await, but the operation is timing out because the original call from the Slackbot needs to receive a response within some fixed time I can't control. It would be fine for my purposes to ping the bot with a response immediately, and then execute the rest of the logic asynchronously. But I'm wondering the best way to set this up.
My Express route looks like:
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
const slack = require('../../services/slack_helpers');
// POST api/slack/foo
router.post('/foo', async (req, res) => {
let [body, images] = await slack.grab_context(req);
knex('texts')
.insert({ body: body,
image_ids: images })
.then(text => { res.send('worked!'); }) // This sends a response back to the original Slackbot call
.catch(err => { res.send(err); })
});
module.exports = router;
And then the slack_helpers module looks like:
const { WebClient } = require('#slack/web-api');
const Slack = new WebClient(process.env.SLACKBOT_TOKEN);
async function grab_context(req) {
try {
const context = await Slack.conversations.history({ // This is the part that takes too long
channel: req.body.channel_id,
latest: req.headers['X-Slack-Request-Timestamp'],
inclusive: true,
limit: 5
});
} catch (error) {
return [error.toString(), 'error'];
}
return await parse_context(context);
};
function parse_context(context) {
var body = [];
context.messages.forEach(message => {
body.push(message.text);
});
body = body.join(' \n');
return [body, ''];
}
module.exports = {
grab_context
};
I'm still getting my head around asynchronous programming, so I may be missing something obvious. I think basically something like res.send perhaps needs to come before the grab_context call? But again, not sure the best flow here.
Update
I've also tried this pattern in the API route, but still getting a timeout:
slack.grab_context(req).then((body, images) => {
knex ...
})
Your timeout may not be coming from where you think. From what I see, it is coming from grab_context. Consider the following simplified version of grab_context
async function grab_context_simple() {
try {
const context = { hello: 'world' }
} catch (error) {
return [error.toString(), 'error']
}
return context
}
grab_context_simple() /* => Promise {
<rejected> ReferenceError: context is not defined
...
} */
You are trying to return context outside of the try block where it was defined, so grab_context will reject with a ReferenceError. It's very likely that this error is being swallowed at the moment, so it would seem like it is timing out.
The fix is to move a single line in grab_context
async function grab_context(req) {
try {
const context = await Slack.conversations.history({
channel: req.body.channel_id,
latest: req.headers['X-Slack-Request-Timestamp'],
inclusive: true,
limit: 5
});
return await parse_context(context); // <- moved this
} catch (error) {
return [error.toString(), 'error'];
}
};
I'm wondering the best way to set this up.
You could add a higher level try/catch block to handle errors that arise from the /foo route. You could also improve readability by staying consistent between async/await and promise chains. Below is how you could use async/await with knex, as well as the aforementioned try/catch block
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
const slack = require('../../services/slack_helpers');
const insertInto = table => payload => knex(table).insert(payload)
const onFooRequest = async (req, res) => {
try {
let [body, images] = await slack.grab_context(req);
const text = await insertInto('texts')({
body: body,
image_ids: images,
});
res.send('worked!');
} catch (err) {
res.send(err);
}
}
router.post('/foo', onFooRequest);
module.exports = router;
I'm trying to get json data from database (sql) with node.js and then pass it to app.get (express.js) but without success.
I have two files urls.js which should get all urls from database and in app.js I'm trying to create api endpoint with express.js. I have managed to get json data in app.js if I write query there and run it but I do not know how to separate it into two files.
code that works in app.js
app.get('/api/urls', (request, response) => {
db.query('SELECT * FROM urls', (error, result) => {
if (error) throw error;
response.send(result);
});
});
I've tried to separate it into two files so in urls.js (model like) I could have something like
class Urls {
async getUrls() {
const sql = `select * from urls`;
return await db.query(sql);
}
}
module.exports = Urls;
and then call it in app.js (controller like):
const data = new Urls();
app.get(/api/urls, (req, res) => {
res.send(data.getUrls());
}
In both cases result should be json
Your getUrls function is async it will return promise,so do something like this
const data = new Urls();
app.get(/api/urls, (req, res) => {
data.getUrls().then(response=>{
res.send(response);
})
}
i have a problem, I need to read a csv file from a web page developed using express.
Basically I have
router.get('/metricas', function(req, res, next) {
var datos= lee.leedatos();
res.render('metricas', {page:'Metricas', menuId:'metricas',data:datos});
});
and the lee.leedatos() is the following
exports.leedatos= function(){
var datos;
const csv =require('fast-csv')
const stream =fs.createReadStream('mywebsite/book1.csv');
const streamCsv=csv({
headers:true,
})
.on('data',data=>{
datos=data;
//console.log(datos);
})
.on('finish',()=>{
console.log('finished')
console.log(datos)
return datos;
})
stream.pipe(streamCsv);
}
My problem is that the web page always returns before the file has been read at all :( .. and data:datos is always empty.
How can I make the call syncronous?
Thanks !
You can use either callbacks / promises to track execution of asynchronous code. This is how code would look like with promises:
router.get('/metricas', function(req, res, next) {
var datos= lee.leedatos().then(() => {
res.render('metricas', {page:'Metricas', menuId:'metricas',data:datos});
});
});
exports.leedatos = function () {
return new Promise((resolve, reject) => {
var datos;
const csv = require('fast-csv')
const stream = fs.createReadStream('mywebsite/book1.csv');
const streamCsv = csv({
headers: true,
}).on('data', data => {
datos = data;
//console.log(datos);
}).on('finish', () => {
console.log('finished')
console.log(datos)
resolve(datos);
})
stream.pipe(streamCsv);
})
}
Here is a link to blog post that might help you in understanding asynchronous functions: https://blog.risingstack.com/node-hero-async-programming-in-node-js/
So I have two files, server.js and db.js
Now Here is the code that I am having issue with :
server.js :
var DB = require('./db')
app.get("/test", (req, res) => {
console.log(DB.getPostAll())
})
db.js :
MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
module.exports.getPostAll = function getPostAll() {
return (
dbo.collection('posts').find({}).toArray(function (err, res) {
if (err) throw err;
else return res
})
)
}
})
.catch(function (err) {
})
Sorry for the silly question. But can anyone say what am I doing wrong here ?
I am trying to use two returns. Before this I tried to use a variable in place of the returns in db.js's getPostAll. But it also returns undefined.
There are plenty of bad practices in the code, I'll try to cover some.
Your db.js file triggers an asynchronous action the moment it is required.
You are not returning the Promise from the db.js file, hence your server.js file starts executing code without knowing if the connection to the DB was fulfilled/pending or rejected
You are exporting a method after a function is executed, this leads to many odd and unexpected side effects. It is best to define all your exports at the top level of the file.
The simplest way to solve your issue is:
server.js :
const connectDB = require('./db')
connectDB().then((db) => {
app.get("/test", (req, res) => {
console.log(db.getPostAll())
})
app.listen(...); // lift the server ONLY when the db is connected
});
db.js :
module.exports = function connectDB() {
return MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
return {
getPostAll() {
return dbo.collection('posts').find({}).toArray()
}
}
})
}
In the db.js file, I am exporting a function that returns a promise, this way I can tell when the connection is complete (by having the promise resolved), it returns an object with all the db methods you need (getPostAll).
In the server.js file, I am waiting for the async connection to be established before I lift the app, this way I know i have my application in a ready state when it is served, and I have the db methods readily available for my app.
You are getting undefined as your promise has not resolved. Try using an async function and await your db operation to finish. See working with async funtions . you can also try mongoose object modeling as your mongodb client.