I am building a discord bot, and I'm trying to connect it to a mongo DB.
The way my code is structured, I have a 'main' file, called index.js, where I run my main code, on the main code, I handle the events. Inside the event 'on ready'(which is executed once) I connect to the DB. This is how I handle the events on index.js:
for (const file of eventFiles) {
const event = require(`./events/${file}`);
if (event.once) {
client.once(event.name, (...args) => event.execute(...args, commands));
} else {
client.on(event.name, (...args) => event.execute(...args, commands));
}
}
To connect to the DB, I have an event script, called ready, that is called only once. The problem I'm having is, when I try to connect to the DB, I get stuck without a response. I'm using mongoose js and following their documentation to connect. Their documentation states that I need to use an async function to connect.
When I do that, the code gets 'stuck'. If I remove the async and await statement form the code below, it works. It would be ok to use it like that, but that's not a good pratice. Code fore ready.js follows:
const mongoose = require('mongoose')
require('dotenv').config()
const testSchema = require('../test-schema')
module.exports = {
name: 'ready',
once: true,
async execute(client, commands) {
console.log(`Ready! Logged in as ${client.user.tag}`);
await mongoose.connect(process.env.MONGO_URI);
console.log("Database Connected!");
},
};
So, is it ok to just remove the async and await?
Thanks in advance.
Disclaimer: English is not my first language, sorry if it sounds confuse.
So, I figured it out (sort of).
Changed my code to the following, and it worked:
const mongoose = require('mongoose')
require('dotenv').config()
module.exports = {
name: 'ready',
once: true,
async execute(client, commands) {
console.log(`Ready! Logged in as ${client.user.tag}`);
main().catch(err => console.log(err));
console.log("Database Connected!");
},
};
async function main() {
await mongoose.connect(process.env.MONGO_URI);
}
Basically, put the connection inside the main function, and called it on the execute method.
It worked.
Related
Im trying to add unit tests to my node express app. The app uses sequelize with sqlite as its orm/database. I am trying to unit test one of my simplest controllers directly (not even trying to test routes with supertest yet)
I was able to get one basic successful test, the issue I a having is that since my controller imports my sequelize User model, it also imports the instantiation of Sequelize. This results in sequelize trying to connect to the DB while the tests are performed. The test is executed before all the db stuff happens, and hence at the end my result is a lot of "Cannot log after tests are done" because Sequelize is still trying to connect and log things, some from the sequelize module itself and others from my code connecting and sync()ing the db.
If I remove the import and usage of the model in my controller, I no longer have those issues, so clearly importing the model is causing all those calls to initialize sequelize. How can i ask Jest to either wait until all async processes are done, or just prevent sequelize from initializing at all (I dont actually need a db connection, i will test them all with mocks)
So here is what the test looks like
describe('testing user registration', () => {
let mRes: Response<any, Record<string, any>>;
let mNext;
beforeAll(()=>{
mRes = {
status: jest.fn().mockReturnThis(),
json: jest.fn()
}
mNext = jest.fn(err => err)
jest.spyOn(UserModel, 'create').mockResolvedValue(createdUser)
})
afterAll(()=>{
jest.resetAllMocks();
})
test('with invalid email', async () => {
const result = await registerUser(mReqInvalidEmail, mRes, mNext);
expect(mNext).toBeCalledWith(invalidBodyError)
})
})
Here is what the Controller looks like:
const registerUser = async(req:Request,res:Response,next:NextFunction) : Promise<Promise<Response> | void> => {
const {body} = req
const {email, role} = body;
if(!isValidEmail(email) || !isValidRole(role)){
const error = createError(
400,
'Invalid body, please provide a valid email address a role of oneOf[\"user\",\"super\"]',
{
body: {
email: 'a valid email string',
role: 'string, oneOf [user, super]'
}
}
);
return next(error);
}
const password = generatePassword()
const hash = hashPassword(password)
const user = await User.create({email, role, password:hash}).catch((err: Error) : Error => {
return createError(500, 'woopsie', err)
})
if(user instanceof Error){
return next(user)
}
return res.status(200).json({
email,
role,
password
});
}
The model:
import {sqlzDB} from "../../database/db";
const User = sqlzDB.define('User',
{
...myfields, not including to save space
}, {
options
})
And finally, where sequelize gets initialized (declaring sqlzDB). This is all the code that is running that I need to either wait for it to finish, or just prevent it from getting called at all!
const {Sequelize} = require('sequelize');
export const sqlzDB = new Sequelize({
dialect: 'sqlite',
storage: 'database/db.sqlite'
});
sqlzDB.authenticate()
.then(():void => {
console.log('Connection to database established successfully.');
}).catch((err : Error): void => {
console.log('Unable to connect to the database: ', err);
})
sqlzDB.sync().then(()=>{
console.log('Sequelize Synced')
})
My test passes just fine. Note that for the test i wrote i dont actually need the mocks yet, Since im just trying to get the setup to work correctly.
I have tried suggestions I have seen out here like calling await new Promise(setImmediate); after my test, or also closing the connection with sqlzDB.close() which causes some different issues (it closes the connection but still tries to log)
So i dont know how to approach this at this point! Sorry for th elong question and thanks to whoever took their time to read this!
You can try a trick to avoid this issue: Close the connection in afterAll
afterEach(() => { // reset mock should be in afterEach
jest.resetAllMocks();
});
afterAll(async ()=>{
await sqlzDB.close();
await new Promise(res => setTimeout(res, 500)); // avoid jest open handle error
});
I have the following code to connect to Mongoose DB using Node.js. But I get some kind of warning or refactor notification from VS code to remove await from the the part when I try to connect.
Its say that: await has no effect in this kind of expression
From the documentation of Mongoose inside index.d.ts file;
export function connect(uri: string, options?: ConnectOptions): Promise<Mongoose>; returns promise
So I try to do like this:
// Provide connection to a new in-memory database server.
const connect = async () => {
// NOTE: before establishing a new connection close previous
await mongoose.disconnect()
mongoServer = await MongoMemoryServer.create()
try {
const mongoUri = await mongoServer.getUri()
await mongoose.connect(mongoUri, opts) // No need to use await??
} catch (error) {
console.log(error)
}
Hihi.
I'm having a problem where jest it's sometimes failing a couple of tests randomly, most of the time because of this error "mongodb memory server cannot perform operation: a background operation is currently running for collection".
In another post I read something about building differents mongo instance for each block of tests.
What I have so far is a globalsetup file where I start the mongo replica set like this:
// global.ts
import { MongoMemoryReplSet } from "mongodb-memory-server";
const replSet = new MongoMemoryReplSet({
replSet: { storageEngine: "wiredTiger" },
});
module.exports = async () => {
await replSet.waitUntilRunning();
const uri = await replSet.getUri();
process.env.MONGODB_URI = uri;
};
and my db.ts is like this
// db.ts
export const connect = async () => {
mongoose.set("useFindAndModify", false);
const conn = mongoose.connect(
process.env.MONGODB_URI || config.connectionString, connectionSettings
);
When trying to call it from a test file I do something like this
// test.spec.ts
import db from "./db";
beforeAll(async () => {
await db.connect();
});
afterAll(async (done) => {
await db.dropCollections();
await db.disconnect(done);
});
beforeEach(async () => {
await seed();
});
describe('Some test', () => {
it('Should not fail and get the seeders', () => {
// some random tests using the seeds values
})
})
What I read in that post is instead of using globalSetup use a setupFile that will run for every test instead of one globally and then I MIGHT be able to solve the concurrency issue I have with my tests.
So, to conclude, does anyone knows if there is a proper way to configure the mongodb in memory or if I am doing something THAT BAD that is allowing this to happend or if is there any kind of improvement I can do that will prevent "mongodb memory server cannot perform operation: a background operation is currently running for collection" this to happen?
Schema operations in routes proceed without a hitch. But when I reuse the same code (just a find() call for example) and execute it in a file ('node test.js'), I get - "MongooseError: Operation refreshtokens.findOne() buffering timed out after 10000ms."
So I tried checking mongoose.connection.readyState and sure enough it equals 0 (disconnected). Why? When again, mongoose.connection is never touched in my routes and I've never had any issues there (it's only initialized in server.js).
Ultimately I'm trying to create an executable that I'll schedule, and it will need to access the database.
Nothing unusual about the code (I don't think ;) -
server.js
const express = require('express')
const app = express()
const mongoose = require('mongoose')
mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
const db = mongoose.connection
db.on('error', (error) => console.error(error))
db.once('open', () => console.log('Connected to Database'))
...
./models/refreshtoken.js
const mongoose = require('mongoose')
const refreshTokenSchema = new mongoose.Schema({
token: {
type: String,
required: true
}
})
module.exports = mongoose.model('RefreshToken', refreshTokenSchema)
Test executable (test.js)
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
(async () => {
await testGetToken();
console.log("ran testGetToken()");
})().catch(e => {
// Deal with the fact the chain failed
console.log(e)
});
async function testGetToken() {
try {
console.log("here in testGetToken() call")
console.log("pre if mongoose.connection = " + mongoose.connection.readyState);
if (mongoose.connection.readyState != 1) {
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
console.log("post if mongoose.connection = " + mongoose.connection.readyState);
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
The executable will get and print the token after I added in the if statement with await mongoose.connect but then the process doesn't close out (I have to ctrl-c twice). Why? (And as noted at top, without the await mongoose.connect, I get the MongooseError.)
Best thing to know is why does mongoose have a mind of its own and is disconnecting only in files I execute (and how to correct it). Added mongoose.connection.on('disconnected'...) event code in server.js too and it never trips.
Next best thing would be if we didn't expect any deleterious effects in proceeding as such, how to get the executable's process to close out. I can do more homework here but I tend to think it's best to resolve problems rather than ignore them (see: Best thing)? :)
Appreciate any help you can offer. Thank you.
It either has an error you aren’t catching or can’t find one. I’ve had the same problem and in my case I was misusing the findOne() an it wasn’t matching up. Try using a mongo GUI and console.log() for you results.
I think I figured this out. In a one-off script, you need to connect and close a separate mongoose connection (with mongoose.connect and mongoose.connection.close).
I thought it would leverage my existing mongoose connection, especially when I had put the test.js code in (and was executing) my route file, but even there not the case. It must be that a heroku scripting process (or what would this be called?) is separate from a web process.
I also thought if I had mongoose.connection.close() in my script, again thinking they were one in the same, then it would take that connection down and schema calls in my route would fail. Also not the case.
Updated test.js for your reference -
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
// ALWAYS executes (think of this like your main)
(async () => {
await mongooseConnect()
await testGetToken();
mongoose.connection.close()
})().catch(e => {
console.log(e)
});
async function mongooseConnect() {
// This is what I was missing before. I thought it would leverage my existing mongoose connection from server.js
// but it most certainly would not when running a script
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
async function testGetToken() {
// gets token from database and prints
try {
console.log("here in testGetToken() call")
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
// event code that fires when mongoose is connected
mongoose.connection.on('connected', () => {
console.log('Test script connected to database');
console.log(mongoose.connection.readyState); //logs 1
});
// event code that fires when mongoose is disconnected
mongoose.connection.on('disconnected', () => {
console.log('Test script disconnected from database');
console.log(mongoose.connection.readyState); //logs 0
});
Output when I run from my command line 'heroku run node test.js' -
Test script connected to database
1
here in testGetToken() call
[
{
_id: 61355917fedd0f00166e4e08,
token: <mytokenhere>,
__v: 0
}
]
Test script disconnected from database
0
Final key point would be I also have the same "mongoose.connection.on('disconnected'..." code in my server.js with a different log message and it doesn't show up in the server logs after running this script (that was one of my fears, that the connection.close in script would take my server mongoose connection down; it doesn't).
Sometimes it also as simple as checking if your current ip is whitelisted or not.
I can't figure out why this app keeps running. I've tried using the why-is-node-running package but I'm not perfectly sure how to read the output properly. Here's the first output of it:
There are 30 handle(s) keeping the process running
# TCPWRAP
/node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));
/node_modules/mongodb/lib/core/connection/connect.js:29 - makeConnection(family, options, cancellationToken, (err, socket) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:182 - connect(monitor.connectOptions, monitor[kCancellationToken], (err, conn) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:206 - checkServer(monitor, e0 => {
/node_modules/mongodb/lib/core/sdam/monitor.js:92 - monitorServer(this);
My guess is it has something to do with MongoDB not closing properly. Although, when I removed all of the other functions between opening the client and closing it, it opened and closed perfectly.
Adding process.exit() at the end closes program properly, but I'd like to figure out why it isn't closing.
A summary of the app is that it is getting data from MongoDB, cleaning it, and then writing it into Firestore - so a lot of async actions going on, but I didn't see Firestore-related stuff pop up in the why-is-node-running logs.
const GrabStuffFromDBToCalculate = require("./helpers/GrabStuffFromDBToCalculate");
const SendToFirestore = require("./helpers/SendToFirestore");
const log = require("why-is-node-running");
const { MongoClient } = require("mongodb");
require("dotenv").config();
const main = async () => {
try {
const client = await MongoClient.connect(process.env.MONGODB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
const collection = await client.db("test").collection("testcollection");
const trip_object = await GrabStuffFromDBToCalculate(collection);
SendToFirestore(trip_object);
client.close();
log(); // "There are 30 handle(s) keeping the process running including node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));"
// process.exit() // this closes everything but I'd rather not have to use this
} catch (err) {
console.log(err);
client.close();
}
};
const runAsync = async () => {
await main(); // this exists because I'm usually running multiple main() functions
};
runAsync();
SendToFirestore code:
const firebase = require("firebase");
const firebaseConfig = require("../config");
module.exports = SendToFirestore = trip_object => {
if (!firebase.apps.length) {
firebase.initializeApp(firebaseConfig);
}
const db = firebase.firestore();
db.doc(`hello/${object._id}`).set({
objectid:object._id
});
};
GrabStuffFromDBToCalculate code (way simplified):
module.exports = GrabStuffFromDBToCalculate = async collection => {
const cursor = await collection
.aggregate([
// does a bunch of stuff here
])
.toArray();
const newObj = cursor[0];
return newObj;
};
Making my comment into an answer since it led to the missing piece.
Node does not shut down because you have an open Firestore connection. You will have to call terminate to allow the SDK to shut down and release resources:
db.terminate();
Which is relevant for allowing nodejs to shut itself down automatically.
Also, I'm not sure you understood that I was suggesting that you use await as in
await client.close()
before calling log() so you are sure that the client connection has been closed before you do the logging. client.close() is an asynchronous method so your original code would log() before that close was complete.