Can set data in Redis server but can't get it - node.js

I can set the value for a key in my Redis server (when I flushall, run this code, and then get key in redis-cli, I get back the proper values), but when I try to get key values through my NodeJs server, it never even logs out 'got data'.
I considered maybe this was because these functions were running asynchronously, and I was asking to get values that weren't yet stored in the cache, but that does not explain why it wouldn't print 'got data' ever.
My console logs-> 'start'->'data saved'->'end' (and no 'got data', ever)
In redis-cli-> flushall->get test->(nil)->run app.js(in the other terminal)->get test->"1, 2, 3, 4, 5"
I get no errors at all, the code runs, but does not do what I want it to.
Also, don't know if this is relevant, but when connecting to the Redis server, just Redis.createClient() only created a client but did not connect and when I looked it up, the general idea I got was that newer versions did not connect automatically and you had to manually redisClient.connect().
I struggled with this a bit at the start but seemed to have sorted this problem, but just thought I'd mention it, if I messed up somehow, please correct me, as I'm pretty new to NodeJs and codng in general.
My code:
const redisClient = Redis.createClient();
redisClient.connect();
const data = [1, 2, 3, 4, 5];
app.get('/', async(req, res, next) => {
console.log('start')
await redisClient.set('test', data);
console.log('data saved');
await redisClient.get('test', (error, test)=>{
console.log('got data');
console.log(test);
});
console.log('end');
});
Thanks!

I have seen your code. Based on my investigation you should remove the callback and keep await only while you get the data from redis.
I have investigated more this issue and have found that client.get() and client.set() function runs asynchronously. Hence it would achieve this way.
client.set('foo', 'bar', (err, reply) => {
if (err) throw err;
console.log(reply);
client.get('foo', (err, reply) => {
if (err) throw err;
console.log(reply);
});
});
But everytime is not the same use-case that we should set and get the value immediately.
To get rid of this, Following are the options.
Promises and async/await
you can promisify a subset of node_redis functions one at a time using native Node.js promises and util.promisify:
example:
const redis = require('redis');
const { promisify } = require('util');
const runApplication = async () => {
const client = redis.createClient();
const setAsync = promisify(client.set).bind(client);
const getAsync = promisify(client.get).bind(client);
await setAsync('foo', 'bar');
const fooValue = await getAsync('foo');
console.log(fooValue);
};
I have used the await here and solve an issue. In addition to that you can use redis.get().then() also to fetch the data rather than a callback.
I am also attaching the link with an example provided by redis repo
https://github.com/redis/node-redis/blob/master/examples/connect-as-acl-user.js
Following is the code, I have tested and it is working fine now.
redis.js
const redis = require("redis");
const redisClient = redis.createClient({
url: "redis://host:6379",
password: "password",
});
redisClient.connect();
// const { promisify } = require("util");
// promisify(redisClient.get).bind(redisClient);
// promisify(redisClient.set).bind(redisClient);
module.exports = redisClient;
index.js
const express = require("express");
const app = express();
const redisClient = require("./redis");
app.get("/set", async (req, res, next) => {
try {
const data = req.query.p;
await redisClient.set("test", data);
res.status(200).json({
message: "data cached",
data: data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.get("/get", async (req, res, next) => {
try {
// const data = await redisClient.get("test");
const data = await reddisClient.get("test").then((data) => {
return data;
});
res.status(200).json({
message: "Cached data retrieved",
data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.listen(process.env.PORT || 3000, () => {
console.log("Node server started");
});
Please find attached a screenshot of the output.
So the final thought is that, when we are using callback and wants to execute the code synchronously you should either use callback inside callback (but it is created callback hell, so it would not suggested anymore) or you should use promise/async await/native promisify library of nodejs.
Please visit below link to get the simplest understanding and example.
https://docs.redis.com/latest/rs/references/client_references/client_nodejs/
Hope my question clear your mind. I am happy to accept the relevant suggestion to improve an answer.

Related

Why does Async firebase fetching is not working? (NODE JS)

Building a NodeJS REST API.
Trying to send load data from FireBase collection, then sending it to the user (as API response).
Looks like the problem is that it's not waits for the firebase fetch to resolve, but send back a response without the collection data. (tried to use ASYNC-AWAIT but its not working)
exports.getChatMessages = async (req, res, next) => {
const chatId = req.params.chatId
const getChatData = () => {
db
.collection('chats')
.doc(chatId)
.collection('messages')
.orderBy('timeStamp', 'asc')
.onSnapshot((snapshot) => {
snapshot.docs.forEach(msg => {
console.log(msg.data().messageContent)
return {
authorID: msg.data().authorID,
messageContent: msg.data().messageContent,
timeStamp: msg.data().timeStamp,
}
})
})
}
try {
const chatData = await getChatData()
console.log(chatData)
res.status(200).json({
message: 'Chat Has Found',
chatData: chatData
})
} catch (err) {
if (!err.statusCode) {
err.statusCode(500)
}
next(err)
}
}
As you can see, I've used 2 console.logs to realize what the problem, Terminal logs looks like:
[] (from console.logs(chatData))
All messages (from console.log(msg.data().messageContent))
Is there any way to block the code unti the firebase data realy fetched?
If I correctly understand, you want to send back an array of all the documents present in the messages subcollection. The following should do the trick.
exports.getChatMessages = async (req, res, next) => {
const chatId = req.params.chatId;
const collectionRef = db
.collection('chats')
.doc(chatId)
.collection('messages')
.orderBy('timeStamp', 'asc');
try {
const chatsQuerySnapshot = await collectionRef.get();
const chatData = [];
chatsQuerySnapshot.forEach((msg) => {
console.log(msg.data().messageContent);
chatData.push({
authorID: msg.data().authorID,
messageContent: msg.data().messageContent,
timeStamp: msg.data().timeStamp,
});
});
console.log(chatData);
res.status(200).json({
message: 'Chat Has Found',
chatData: chatData,
});
} catch (err) {
if (!err.statusCode) {
err.statusCode(500);
}
next(err);
}
};
The asynchronous get() method returns a QuerySnapshot on which you can call forEach() for enumerating all of the documents in the QuerySnapshot.
You can only await a Promise. Currently, getChatData() does not return a Promise, so awaiting it is pointless. You are trying to await a fixed value, so it resolves immediately and jumps to the next line. console.log(chatData) happens. Then, later, your (snapshot) => callback happens, but too late.
const getChatData = () => new Promise(resolve => { // Return a Promise, so it can be awaited
db.collection('chats')
.doc(chatId)
.collection('messages')
.orderBy('timeStamp', 'asc')
.onSnapshot(resolve) // Equivalent to .onSnapshot((snapshot) => resolve(snapshot))
})
const snapshot = await getChatData();
console.log(snapshot)
// Put your transform logic out of the function that calls the DB. A function should only do one thing if possible : call or transform, not both.
const chatData = snapshot.map(msg => ({
authorID: msg.data().authorID,
messageContent: msg.data().messageContent,
timeStamp: msg.data().timeStamp,
}));
res.status(200).json({
message: 'Chat Has Found',
chatData
})
Right now, getChatData is this (short version):
const getChatData = () => {
db
.collection('chats')
.doc(chatId)
.collection('messages')
.orderBy('timeStamp', 'asc')
.onSnapshot((snapshot) => {}) // some things inside
}
What that means is that the getChatData function calls some db query, and then returns void (nothing). I bet you'd want to return the db call (hopefully it's a Promise), so that your await does some work for you. Something along the lines of:
const getChatData = async () =>
db
.collection('chats')
// ...
Which is the same as const getChatData = async() => { return db... }
Update: Now that I've reviewed the docs once again, I see that you use onSnapshot, which is meant for updates and can fire multiple times. The first call actually makes a request, but then continues to listen on those updates. Since that seems like a regular request-response, and you want it to happen only once - use .get() docs instead of .onSnapshot(). Otherwise those listeners would stay there and cause troubles. .get() returns a Promise, so the sample fix that I've mentioned above would work perfectly and you don't need to change other pieces of the code.

Chaining async await calls in Node/Express with an external time limit

I'm building a Slackbot that makes a call to an Express app, which then needs to 1) fetch some other data from the Slack API, and 2) insert resulting data in my database. I think I have the flow right finally using async await, but the operation is timing out because the original call from the Slackbot needs to receive a response within some fixed time I can't control. It would be fine for my purposes to ping the bot with a response immediately, and then execute the rest of the logic asynchronously. But I'm wondering the best way to set this up.
My Express route looks like:
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
const slack = require('../../services/slack_helpers');
// POST api/slack/foo
router.post('/foo', async (req, res) => {
let [body, images] = await slack.grab_context(req);
knex('texts')
.insert({ body: body,
image_ids: images })
.then(text => { res.send('worked!'); }) // This sends a response back to the original Slackbot call
.catch(err => { res.send(err); })
});
module.exports = router;
And then the slack_helpers module looks like:
const { WebClient } = require('#slack/web-api');
const Slack = new WebClient(process.env.SLACKBOT_TOKEN);
async function grab_context(req) {
try {
const context = await Slack.conversations.history({ // This is the part that takes too long
channel: req.body.channel_id,
latest: req.headers['X-Slack-Request-Timestamp'],
inclusive: true,
limit: 5
});
} catch (error) {
return [error.toString(), 'error'];
}
return await parse_context(context);
};
function parse_context(context) {
var body = [];
context.messages.forEach(message => {
body.push(message.text);
});
body = body.join(' \n');
return [body, ''];
}
module.exports = {
grab_context
};
I'm still getting my head around asynchronous programming, so I may be missing something obvious. I think basically something like res.send perhaps needs to come before the grab_context call? But again, not sure the best flow here.
Update
I've also tried this pattern in the API route, but still getting a timeout:
slack.grab_context(req).then((body, images) => {
knex ...
})
Your timeout may not be coming from where you think. From what I see, it is coming from grab_context. Consider the following simplified version of grab_context
async function grab_context_simple() {
try {
const context = { hello: 'world' }
} catch (error) {
return [error.toString(), 'error']
}
return context
}
grab_context_simple() /* => Promise {
<rejected> ReferenceError: context is not defined
...
} */
You are trying to return context outside of the try block where it was defined, so grab_context will reject with a ReferenceError. It's very likely that this error is being swallowed at the moment, so it would seem like it is timing out.
The fix is to move a single line in grab_context
async function grab_context(req) {
try {
const context = await Slack.conversations.history({
channel: req.body.channel_id,
latest: req.headers['X-Slack-Request-Timestamp'],
inclusive: true,
limit: 5
});
return await parse_context(context); // <- moved this
} catch (error) {
return [error.toString(), 'error'];
}
};
I'm wondering the best way to set this up.
You could add a higher level try/catch block to handle errors that arise from the /foo route. You could also improve readability by staying consistent between async/await and promise chains. Below is how you could use async/await with knex, as well as the aforementioned try/catch block
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
const slack = require('../../services/slack_helpers');
const insertInto = table => payload => knex(table).insert(payload)
const onFooRequest = async (req, res) => {
try {
let [body, images] = await slack.grab_context(req);
const text = await insertInto('texts')({
body: body,
image_ids: images,
});
res.send('worked!');
} catch (err) {
res.send(err);
}
}
router.post('/foo', onFooRequest);
module.exports = router;

How to restore mongo in chai mocha

So I want to restore mongo database before tests begin.
I do this way:
const app = require("../app");
const chai = require("chai");
const mongoose = require("mongoose");
const User = require('../models/users');
const Region = require('../models/regions');
const testUsers = require('../testdata/users.json');
const testRegions = require('../testdata/regions.json');
describe('Restoring database', function () {
before(function(done) {
var promises = [
User.deleteMany().exec()
,Region.deleteMany().exec()
];
console.log('Cleaned database');
done();
});
before(function(done) {
testUsers.users.forEach(element => {
var ObjectId = mongoose.Types.ObjectId;
element._id = new ObjectId(element._id);
var newUser = new User(element);
newUser.save(function (err, result) {
if (err) {
console.log("err:",err);
}
});
});
console.log('Users added');
done();
});
before(function(done) {
testRegions.regions.forEach(element => {
var newRegion = new Region(element);
newRegion.save(function (err, result) {
if (err) {
console.log("err:",err);
}
});
});
console.log('Regions added');
done();
});
testdata/users.json and testdata/regions.json are simple json files including key/pair values.
Does this look good?
When I run
npm test
It does not give any error. And in the console I see this:
Restoring database
Cleaned database
Users added
Regions added
But when I look in database I get different results.
Sometimes everything looks good. All the rows are in the collections.
Sometimes a few rows are missing in one of the collections.
Sometimes one of the collections is empty.
This is a very strange behaviour.
I also tried to add in the variable "promises" each of the "newUser" and "newRegion" instead of executing them directly.
But I still get these strange results.
Whats the deal?
The issue is due to done() being called before the async statements/promises have completed.
Either use async/await or use Promises and only call done() when your async statements/promises have completed.
For example:
No done() call as we using await statements which will wait till each statement completes before continuing:
before(async function() {
let userResult = await User.deleteMany();
let regionResult = wait Region.deleteMany();
console.log('Cleaned database');
});
Or use done() with promises:
before(function(done) {
User.deleteMany()
.then(result => {
console.log('Cleaned database');
done();
});
});
The syntax in your before example is not adding Promises at all, it is simply adding those functions to an array:
var promises = [
User.deleteMany().exec()
,Region.deleteMany().exec()
];
Take a look at the following related answer to help.

Why does it return undefined?

So I have two files, server.js and db.js
Now Here is the code that I am having issue with :
server.js :
var DB = require('./db')
app.get("/test", (req, res) => {
console.log(DB.getPostAll())
})
db.js :
MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
module.exports.getPostAll = function getPostAll() {
return (
dbo.collection('posts').find({}).toArray(function (err, res) {
if (err) throw err;
else return res
})
)
}
})
.catch(function (err) {
})
Sorry for the silly question. But can anyone say what am I doing wrong here ?
I am trying to use two returns. Before this I tried to use a variable in place of the returns in db.js's getPostAll. But it also returns undefined.
There are plenty of bad practices in the code, I'll try to cover some.
Your db.js file triggers an asynchronous action the moment it is required.
You are not returning the Promise from the db.js file, hence your server.js file starts executing code without knowing if the connection to the DB was fulfilled/pending or rejected
You are exporting a method after a function is executed, this leads to many odd and unexpected side effects. It is best to define all your exports at the top level of the file.
The simplest way to solve your issue is:
server.js :
const connectDB = require('./db')
connectDB().then((db) => {
app.get("/test", (req, res) => {
console.log(db.getPostAll())
})
app.listen(...); // lift the server ONLY when the db is connected
});
db.js :
module.exports = function connectDB() {
return MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
return {
getPostAll() {
return dbo.collection('posts').find({}).toArray()
}
}
})
}
In the db.js file, I am exporting a function that returns a promise, this way I can tell when the connection is complete (by having the promise resolved), it returns an object with all the db methods you need (getPostAll).
In the server.js file, I am waiting for the async connection to be established before I lift the app, this way I know i have my application in a ready state when it is served, and I have the db methods readily available for my app.
You are getting undefined as your promise has not resolved. Try using an async function and await your db operation to finish. See working with async funtions . you can also try mongoose object modeling as your mongodb client.

Bluebird promise resolve in express route

I have an simple REST application and I want to read files in my directory and send them back to frontend. There's code I'm using:
const fs = Promise.promisifyAll(require('fs'))
const router = require('express').Router()
router.get('/list', async (req, res, next) => {
const files = await fs.readdirAsync('presentations')
res.json(files)
})
The problem is: my frontend receive 'Promise', but if I try to debug it shows me that files is an array.
I've tried not to use async/await syntax like that:
router.get('/list', (req, res, next) => {
fs.readdirAsync('presentations')
.then(files => {
res.json(files)
})
})
But result was the same: frontend still get Promise.
UPD: Problem was with frontend axios instance. It didn't resolve promise, so await for results solved a problem.
So, there are three parts. Reading, storing and sending.
Here's the reading part:
var fs = require('fs');
function readFiles(dirname, onFileContent, onError) {
fs.readdir(dirname, function(err, filenames) {
if (err) {
onError(err);
return;
}
onFileContent(filename);
});
});
});
}
Here's the storing part:
var data = {};
readFiles('dirname/', function(filename) {
data[filename] = filname;
}, function(err) {
throw err;
});
The sending part is up to you. You may want to send them one by one or after reading completion.
If you want to send files after reading completion you should either use sync versions of fs functions or use promises. Async callbacks is not a good style.

Resources