So I have two files, server.js and db.js
Now Here is the code that I am having issue with :
server.js :
var DB = require('./db')
app.get("/test", (req, res) => {
console.log(DB.getPostAll())
})
db.js :
MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
module.exports.getPostAll = function getPostAll() {
return (
dbo.collection('posts').find({}).toArray(function (err, res) {
if (err) throw err;
else return res
})
)
}
})
.catch(function (err) {
})
Sorry for the silly question. But can anyone say what am I doing wrong here ?
I am trying to use two returns. Before this I tried to use a variable in place of the returns in db.js's getPostAll. But it also returns undefined.
There are plenty of bad practices in the code, I'll try to cover some.
Your db.js file triggers an asynchronous action the moment it is required.
You are not returning the Promise from the db.js file, hence your server.js file starts executing code without knowing if the connection to the DB was fulfilled/pending or rejected
You are exporting a method after a function is executed, this leads to many odd and unexpected side effects. It is best to define all your exports at the top level of the file.
The simplest way to solve your issue is:
server.js :
const connectDB = require('./db')
connectDB().then((db) => {
app.get("/test", (req, res) => {
console.log(db.getPostAll())
})
app.listen(...); // lift the server ONLY when the db is connected
});
db.js :
module.exports = function connectDB() {
return MongoClient.connect(uri, { useNewUrlParser: true })
.then(function (db) {
console.log("Connected")
var dbo = db.db('test')
return {
getPostAll() {
return dbo.collection('posts').find({}).toArray()
}
}
})
}
In the db.js file, I am exporting a function that returns a promise, this way I can tell when the connection is complete (by having the promise resolved), it returns an object with all the db methods you need (getPostAll).
In the server.js file, I am waiting for the async connection to be established before I lift the app, this way I know i have my application in a ready state when it is served, and I have the db methods readily available for my app.
You are getting undefined as your promise has not resolved. Try using an async function and await your db operation to finish. See working with async funtions . you can also try mongoose object modeling as your mongodb client.
Related
I can set the value for a key in my Redis server (when I flushall, run this code, and then get key in redis-cli, I get back the proper values), but when I try to get key values through my NodeJs server, it never even logs out 'got data'.
I considered maybe this was because these functions were running asynchronously, and I was asking to get values that weren't yet stored in the cache, but that does not explain why it wouldn't print 'got data' ever.
My console logs-> 'start'->'data saved'->'end' (and no 'got data', ever)
In redis-cli-> flushall->get test->(nil)->run app.js(in the other terminal)->get test->"1, 2, 3, 4, 5"
I get no errors at all, the code runs, but does not do what I want it to.
Also, don't know if this is relevant, but when connecting to the Redis server, just Redis.createClient() only created a client but did not connect and when I looked it up, the general idea I got was that newer versions did not connect automatically and you had to manually redisClient.connect().
I struggled with this a bit at the start but seemed to have sorted this problem, but just thought I'd mention it, if I messed up somehow, please correct me, as I'm pretty new to NodeJs and codng in general.
My code:
const redisClient = Redis.createClient();
redisClient.connect();
const data = [1, 2, 3, 4, 5];
app.get('/', async(req, res, next) => {
console.log('start')
await redisClient.set('test', data);
console.log('data saved');
await redisClient.get('test', (error, test)=>{
console.log('got data');
console.log(test);
});
console.log('end');
});
Thanks!
I have seen your code. Based on my investigation you should remove the callback and keep await only while you get the data from redis.
I have investigated more this issue and have found that client.get() and client.set() function runs asynchronously. Hence it would achieve this way.
client.set('foo', 'bar', (err, reply) => {
if (err) throw err;
console.log(reply);
client.get('foo', (err, reply) => {
if (err) throw err;
console.log(reply);
});
});
But everytime is not the same use-case that we should set and get the value immediately.
To get rid of this, Following are the options.
Promises and async/await
you can promisify a subset of node_redis functions one at a time using native Node.js promises and util.promisify:
example:
const redis = require('redis');
const { promisify } = require('util');
const runApplication = async () => {
const client = redis.createClient();
const setAsync = promisify(client.set).bind(client);
const getAsync = promisify(client.get).bind(client);
await setAsync('foo', 'bar');
const fooValue = await getAsync('foo');
console.log(fooValue);
};
I have used the await here and solve an issue. In addition to that you can use redis.get().then() also to fetch the data rather than a callback.
I am also attaching the link with an example provided by redis repo
https://github.com/redis/node-redis/blob/master/examples/connect-as-acl-user.js
Following is the code, I have tested and it is working fine now.
redis.js
const redis = require("redis");
const redisClient = redis.createClient({
url: "redis://host:6379",
password: "password",
});
redisClient.connect();
// const { promisify } = require("util");
// promisify(redisClient.get).bind(redisClient);
// promisify(redisClient.set).bind(redisClient);
module.exports = redisClient;
index.js
const express = require("express");
const app = express();
const redisClient = require("./redis");
app.get("/set", async (req, res, next) => {
try {
const data = req.query.p;
await redisClient.set("test", data);
res.status(200).json({
message: "data cached",
data: data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.get("/get", async (req, res, next) => {
try {
// const data = await redisClient.get("test");
const data = await reddisClient.get("test").then((data) => {
return data;
});
res.status(200).json({
message: "Cached data retrieved",
data,
});
} catch (err) {
console.error(err);
res.status(500).json({
message: "Something went wrong",
});
}
});
app.listen(process.env.PORT || 3000, () => {
console.log("Node server started");
});
Please find attached a screenshot of the output.
So the final thought is that, when we are using callback and wants to execute the code synchronously you should either use callback inside callback (but it is created callback hell, so it would not suggested anymore) or you should use promise/async await/native promisify library of nodejs.
Please visit below link to get the simplest understanding and example.
https://docs.redis.com/latest/rs/references/client_references/client_nodejs/
Hope my question clear your mind. I am happy to accept the relevant suggestion to improve an answer.
Schema operations in routes proceed without a hitch. But when I reuse the same code (just a find() call for example) and execute it in a file ('node test.js'), I get - "MongooseError: Operation refreshtokens.findOne() buffering timed out after 10000ms."
So I tried checking mongoose.connection.readyState and sure enough it equals 0 (disconnected). Why? When again, mongoose.connection is never touched in my routes and I've never had any issues there (it's only initialized in server.js).
Ultimately I'm trying to create an executable that I'll schedule, and it will need to access the database.
Nothing unusual about the code (I don't think ;) -
server.js
const express = require('express')
const app = express()
const mongoose = require('mongoose')
mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
const db = mongoose.connection
db.on('error', (error) => console.error(error))
db.once('open', () => console.log('Connected to Database'))
...
./models/refreshtoken.js
const mongoose = require('mongoose')
const refreshTokenSchema = new mongoose.Schema({
token: {
type: String,
required: true
}
})
module.exports = mongoose.model('RefreshToken', refreshTokenSchema)
Test executable (test.js)
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
(async () => {
await testGetToken();
console.log("ran testGetToken()");
})().catch(e => {
// Deal with the fact the chain failed
console.log(e)
});
async function testGetToken() {
try {
console.log("here in testGetToken() call")
console.log("pre if mongoose.connection = " + mongoose.connection.readyState);
if (mongoose.connection.readyState != 1) {
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
console.log("post if mongoose.connection = " + mongoose.connection.readyState);
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
The executable will get and print the token after I added in the if statement with await mongoose.connect but then the process doesn't close out (I have to ctrl-c twice). Why? (And as noted at top, without the await mongoose.connect, I get the MongooseError.)
Best thing to know is why does mongoose have a mind of its own and is disconnecting only in files I execute (and how to correct it). Added mongoose.connection.on('disconnected'...) event code in server.js too and it never trips.
Next best thing would be if we didn't expect any deleterious effects in proceeding as such, how to get the executable's process to close out. I can do more homework here but I tend to think it's best to resolve problems rather than ignore them (see: Best thing)? :)
Appreciate any help you can offer. Thank you.
It either has an error you aren’t catching or can’t find one. I’ve had the same problem and in my case I was misusing the findOne() an it wasn’t matching up. Try using a mongo GUI and console.log() for you results.
I think I figured this out. In a one-off script, you need to connect and close a separate mongoose connection (with mongoose.connect and mongoose.connection.close).
I thought it would leverage my existing mongoose connection, especially when I had put the test.js code in (and was executing) my route file, but even there not the case. It must be that a heroku scripting process (or what would this be called?) is separate from a web process.
I also thought if I had mongoose.connection.close() in my script, again thinking they were one in the same, then it would take that connection down and schema calls in my route would fail. Also not the case.
Updated test.js for your reference -
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
// ALWAYS executes (think of this like your main)
(async () => {
await mongooseConnect()
await testGetToken();
mongoose.connection.close()
})().catch(e => {
console.log(e)
});
async function mongooseConnect() {
// This is what I was missing before. I thought it would leverage my existing mongoose connection from server.js
// but it most certainly would not when running a script
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
async function testGetToken() {
// gets token from database and prints
try {
console.log("here in testGetToken() call")
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
// event code that fires when mongoose is connected
mongoose.connection.on('connected', () => {
console.log('Test script connected to database');
console.log(mongoose.connection.readyState); //logs 1
});
// event code that fires when mongoose is disconnected
mongoose.connection.on('disconnected', () => {
console.log('Test script disconnected from database');
console.log(mongoose.connection.readyState); //logs 0
});
Output when I run from my command line 'heroku run node test.js' -
Test script connected to database
1
here in testGetToken() call
[
{
_id: 61355917fedd0f00166e4e08,
token: <mytokenhere>,
__v: 0
}
]
Test script disconnected from database
0
Final key point would be I also have the same "mongoose.connection.on('disconnected'..." code in my server.js with a different log message and it doesn't show up in the server logs after running this script (that was one of my fears, that the connection.close in script would take my server mongoose connection down; it doesn't).
Sometimes it also as simple as checking if your current ip is whitelisted or not.
In node.js I had code like following:
mongoose.connect(dbURI, dbOptions)
.then(() => {
console.log("ok");
},
err => {
console.log('error: '+ err)
}
);
Now i want to do it with async/await syntax. So i could start with var mcResult = await mongoose.connect(dbURI, dbOptions);, afaik it will wait for operation, until it ends with any result (much like calling C function read() or fread() in syncronous mode).
But what should I write then? What does that return to the mcResult variable and how to check for an error or success? Basically I want a similar snippet, but written with proper async/await syntax.
Also I wonder because I have auto reconnect, among dbOptions:
dbOptions: {
autoReconnect: true,
reconnectTries: 999999999,
reconnectInterval: 3000
}
Would it "stuck" on await forever, in case if database connection is unavailble? I hope you can give me a clue on what would happen and how that would work.
Basically I want a similar snippet, but written with proper async/await syntax.
(async () => {
try {
await mongoose.connect(dbURI, dbOptions)
} catch (err) {
console.log('error: ' + err)
}
})()
Please try this, Below code has basics of db connectivity and a query :
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
let url = 'mongodb://localhost:27017/test';
const usersSchema = new Schema({
any: {}
}, {
strict: false
});
const Users = mongoose.model('users', usersSchema, 'users');
/** We've created schema as in mongoose you need schemas for your collections to do operations on them */
const dbConnect = async () => {
let db = null;
try {
/** In real-time you'll split DB connection(into another file) away from DB calls */
await mongoose.connect(url, { useNewUrlParser: true }); // await on a step makes process to wait until it's done/ err'd out.
db = mongoose.connection;
let dbResp = await Users.find({}).lean(); /** Gets all documents out of users collection.
Using .lean() to convert MongoDB documents to raw Js objects for accessing further. */
db.close(); // Needs to close connection, In general you don't close & re-create often. But needed for test scripts - You might use connection pooling in real-time.
return dbResp;
} catch (err) {
(db) && db.close(); /** Needs to close connection -
Only if mongoose.connect() is success & fails after it, as db connection is established by then. */
console.log('Error at dbConnect ::', err)
throw err;
}
}
dbConnect().then(res => console.log('Printing at callee ::', res)).catch(err => console.log('Err at Call ::', err));
As we're talking about async/await then few things I wanted to mention - await definitely needs it's function to be declared as async - otherwise it would throw an error. And it's recommended to wrap async/await code inside try/catch block.
const connectDb = async () => {
await mongoose.connect(dbUri, dbOptions).then(
() => {
console.info(`Connected to database`)
},
error => {
console.error(`Connection error: ${error.stack}`)
process.exit(1)
}
)
}
connectDb().catch(error => console.error(error))
Lets assume the use of then() is prohibited, you can result to this...
const connectDb = async () => {
try {
await mongoose.connect(dbConfig.url, dbConfigOptions)
console.info(`Connected to database on Worker process: ${process.pid}`)
} catch (error) {
console.error(`Connection error: ${error.stack} on Worker process: ${process.pid}`)
process.exit(1)
}
}
I saw these posts on SO describing this error. Most of them was by the reason that JavaScript is async and mongoClient.close() called outside of callback. That's not my case, but I don't know what else can be the reason.
const MongoClient = require('mongodb').MongoClient;
const url = "mongodb://localhost:27017/";
const mongoClient = new MongoClient(url, {
useNewUrlParser: true
});
module.exports = class Mongo {
insertOne(article) {
mongoClient.connect((err, client) => {
const db = client.db('grabber');
db.collection("zr").insertOne(article, (err, res) => {
if (err) throw err;
mongoClient.close();
});
});
};
}
I observed that you open mongoClient.connect() in the insertOne() method, and also call mongoClient.close() within that method, with mongoClient as a global variable.
My hunch is that either:
There's another method that calls mongoClient that was closed by this method, or
You called the insertOne(article) twice
I can confirm that the second reason is the most likely one. Here's the code I tried:
const MongoClient = require('mongodb').MongoClient;
const url = "mongodb://localhost:27017/";
const mongoClient = new MongoClient(url, {
useNewUrlParser: true
});
class Mongo {
insertOne(article) {
mongoClient.connect((err, client) => {
const db = client.db('grabber');
db.collection("zr").insertOne(article, (err, res) => {
if (err) throw err;
mongoClient.close();
});
});
};
};
x = new Mongo()
setTimeout(function() { x.insertOne({'a': 1}); }, 1000);
setTimeout(function() { x.insertOne({'a': 2}); }, 2000);
The two setTimeout was there to ensure that the two insertOne() are called one after another. Result:
MongoError: server instance pool was destroyed
The way your code is currently structured, the node driver creates a new connection pool every time insertOne() is called. This is not optimal, and prevents the node driver to utilize connection pooling.
Instead of calling mongoClient.connect() inside insertOne(), call it globally outside of class Mongo. Pass the global connection object (the returned object from mongoClient.connect()) instead of the mongoClient object itself to your insertOne() method.
I'm trying to build an API for a single-page web app using AWS Lambda and the Serverless Framework. I want to use Redis Cloud for storage, mostly for its combination of speed and data persistence. I may use more Redis Cloud features in the future, so I'd prefer to avoid using ElastiCache for this. My Redis Cloud instance is running in the same AWS region as my function.
I have a function called related that takes a hashtag from a GET request to an API endpoint, and checks to see if there's an entry for it in the database. If it's there, it should return the results immediately. If not, it should query RiteTag, write the results to Redis, and then return the results to the user.
I'm pretty new to this, so I'm probably doing something adorably naive. Here's the event handler:
'use strict'
const lib = require('../lib/related')
module.exports.handler = function (event, context) {
lib.respond(event, (err, res) => {
if (err) {
return context.fail(err)
} else {
return context.succeed(res)
}
})
}
Here's the ../lib/related.js file:
var redis = require('redis')
var jsonify = require('redis-jsonify')
var rt = require('./ritetag')
var redisOptions = {
host: process.env.REDIS_URL,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASS
}
var client = jsonify(redis.createClient(redisOptions))
module.exports.respond = function (event, callback) {
var tag = event.hashtag.replace(/^#/, '')
var key = 'related:' + tag
client.on('connect', () => {
console.log('Connected:', client.connected)
})
client.on('end', () => {
console.log('Connection closed.')
})
client.on('ready', function () {
client.get(key, (err, res) => {
if (err) {
client.quit()
callback(err)
} else {
if (res) {
// Tag is found in Redis, so send results directly.
client.quit()
callback(null, res)
} else {
// Tag is not yet in Redis, so query Ritetag.
rt.hashtagDirectory(tag, (err, res) => {
if (err) {
client.quit()
callback(err)
} else {
client.set(key, res, (err) => {
if (err) {
callback(err)
} else {
client.quit()
callback(null, res)
}
})
}
})
}
}
})
})
}
All of this works as expected, to a point. If I run the function locally (using sls function run related), I have no problems whatsoever—tags are read from and written to the Redis database as they should be. However, when I deploy it (using sls dash deploy), it works the first time it's run after deployment, and then stops working. All subsequent attempts to run it simply return null to the browser (or Postman, or curl, or the web app). This is true regardless of whether the tag I use for testing is already in the database or not. If I then re-deploy, making no changes to the function itself, it works again—once.
On my local machine, the function first logs Connected: true to the console, then the results of the query, then Connection closed. On AWS, it logs Connected: true, then the results of the query, and that's it. On the second run, it logs Connection closed. and nothing else. On the third and all subsequent runs, it logs nothing at all. Neither environment ever reports any errors.
It seems pretty clear that the problem is with the connection to Redis. If I don't close it in the callbacks, then subsequent attempts to call the function just time out. I've also tried using redis.unref instead of redis.quit, but that didn't seem to make any difference.
Any help would be greatly appreciated.
I've now solved my own problem, and I hope I can be of help to someone experiencing this problem in the future.
There are two major considerations when connecting to a database like I did in the code above from a Lambda function:
Once context.succeed(), context.fail(), or context.done() is called, AWS may freeze any processes that haven't finished yet. This is what was causing AWS to log Connection closed on the second call to my API endpoint—the process was frozen just before Redis finished closing, then thawed on the next call, at which point it continued right where it left off, reporting that the connection was closed. Takeaway: if you want to close your database connection, make sure it's fully closed before you call one of those methods. You can do this by putting a callback in an event handler that's triggered by a connection close (.on('end'), in my case).
If you split your code into separate files and require them at the top of each file, like I did, Amazon will cache as many of those modules as possible in memory. If that's causing problems, try moving the require() calls inside a function instead of at the top of the file, then exporting that function. Those modules will then be re-imported whenever the function is run.
Here's my updated code. Note that I've also put my Redis configuration into a separate file, so I can import it into other Lambda functions without duplicating code.
The Event Handler
'use strict'
const lib = require('../lib/related')
module.exports.handler = function (event, context) {
lib.respond(event, (err, res) => {
if (err) {
return context.fail(err)
} else {
return context.succeed(res)
}
})
}
Redis Configuration
module.exports = () => {
const redis = require('redis')
const jsonify = require('redis-jsonify')
const redisOptions = {
host: process.env.REDIS_URL,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASS
}
return jsonify(redis.createClient(redisOptions))
}
The Function
'use strict'
const rt = require('./ritetag')
module.exports.respond = function (event, callback) {
const redis = require('./redis')()
const tag = event.hashtag.replace(/^#/, '')
const key = 'related:' + tag
let error, response
redis.on('end', () => {
callback(error, response)
})
redis.on('ready', function () {
redis.get(key, (err, res) => {
if (err) {
redis.quit(() => {
error = err
})
} else {
if (res) {
// Tag is found in Redis, so send results directly.
redis.quit(() => {
response = res
})
} else {
// Tag is not yet in Redis, so query Ritetag.
rt.hashtagDirectory(tag, (err, res) => {
if (err) {
redis.quit(() => {
error = err
})
} else {
redis.set(key, res, (err) => {
if (err) {
redis.quit(() => {
error = err
})
} else {
redis.quit(() => {
response = res
})
}
})
}
})
}
}
})
})
}
This works exactly as it should—and it's blazing fast, too.