I can't figure out why this app keeps running. I've tried using the why-is-node-running package but I'm not perfectly sure how to read the output properly. Here's the first output of it:
There are 30 handle(s) keeping the process running
# TCPWRAP
/node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));
/node_modules/mongodb/lib/core/connection/connect.js:29 - makeConnection(family, options, cancellationToken, (err, socket) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:182 - connect(monitor.connectOptions, monitor[kCancellationToken], (err, conn) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:206 - checkServer(monitor, e0 => {
/node_modules/mongodb/lib/core/sdam/monitor.js:92 - monitorServer(this);
My guess is it has something to do with MongoDB not closing properly. Although, when I removed all of the other functions between opening the client and closing it, it opened and closed perfectly.
Adding process.exit() at the end closes program properly, but I'd like to figure out why it isn't closing.
A summary of the app is that it is getting data from MongoDB, cleaning it, and then writing it into Firestore - so a lot of async actions going on, but I didn't see Firestore-related stuff pop up in the why-is-node-running logs.
const GrabStuffFromDBToCalculate = require("./helpers/GrabStuffFromDBToCalculate");
const SendToFirestore = require("./helpers/SendToFirestore");
const log = require("why-is-node-running");
const { MongoClient } = require("mongodb");
require("dotenv").config();
const main = async () => {
try {
const client = await MongoClient.connect(process.env.MONGODB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
const collection = await client.db("test").collection("testcollection");
const trip_object = await GrabStuffFromDBToCalculate(collection);
SendToFirestore(trip_object);
client.close();
log(); // "There are 30 handle(s) keeping the process running including node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));"
// process.exit() // this closes everything but I'd rather not have to use this
} catch (err) {
console.log(err);
client.close();
}
};
const runAsync = async () => {
await main(); // this exists because I'm usually running multiple main() functions
};
runAsync();
SendToFirestore code:
const firebase = require("firebase");
const firebaseConfig = require("../config");
module.exports = SendToFirestore = trip_object => {
if (!firebase.apps.length) {
firebase.initializeApp(firebaseConfig);
}
const db = firebase.firestore();
db.doc(`hello/${object._id}`).set({
objectid:object._id
});
};
GrabStuffFromDBToCalculate code (way simplified):
module.exports = GrabStuffFromDBToCalculate = async collection => {
const cursor = await collection
.aggregate([
// does a bunch of stuff here
])
.toArray();
const newObj = cursor[0];
return newObj;
};
Making my comment into an answer since it led to the missing piece.
Node does not shut down because you have an open Firestore connection. You will have to call terminate to allow the SDK to shut down and release resources:
db.terminate();
Which is relevant for allowing nodejs to shut itself down automatically.
Also, I'm not sure you understood that I was suggesting that you use await as in
await client.close()
before calling log() so you are sure that the client connection has been closed before you do the logging. client.close() is an asynchronous method so your original code would log() before that close was complete.
Related
I followed this tutorial to set up MongoDB in my Next.js application: https://www.mongodb.com/developer/languages/javascript/nextjs-with-mongodb.
In a file called mongodb-config.js, I have
import { MongoClient } from 'mongodb'
const uri = process.env.MONGODB_URI
const options = {
useUnifiedTopology: true,
useNewUrlParser: true,
}
let client;
let dbPromise;
if (!process.env.MONGODB_URI) {
throw new Error('Please add your Mongo URI to .env.local')
}
// In production mode, it's best to not use a global variable.
client = new MongoClient(uri, options)
dbPromise = client.connect()
// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default dbPromise
The above is how I configure my database.
When I need to use the database for my API, I do:
import dbPromise from "database/mongodb-config";
let db;
dbPromise.then((value) => {
const client = value;
db = client.db("database_name");
})
.catch((error)=>{
console.error(error);
});
db will be the variable that links to my database.
Now, I want to simplify this process and put everything in one file.
In the portion where I set up dbPromise in mongodb-config.js, I initialize the DB variable and export it:
client = new MongoClient(uri, options)
dbPromise = client.connect()
let db;
dbPromise.then((value)=>{
const client = value;
db = client.db("datatbase_name");
})
.catch((error)=>{
console.error(error);
});
export default db
I totally expect this to work. However, when I import db to another file and use it, the db is null. However, I waited until the dbPromise is resolved to pass db a value.
Use await like this.
dbPromise = client.connect()
let db;
await dbPromise.then((value)=>{
const client = value;
db = client.db("datatbase_name");
})
.catch(error){
console.error(error);
}
export default db
because mongoose is asynchronous so we have to use await before dbPromise otherwise export will be executed first before dbPromise.
Try this, instead of returning db, return an async function, which you can all with await, in other parts to obtain, the connection to the database, the function will not try to make repeated connections, if they are already present:
let db, connection;
async function dbSetup() {
try {
if(!connection) {
connection = await client.connect();
}
if(!db) {
db = await connection.db("datatbase_name");
}
return db;
} catch(error){
console.error(error);
}
}
export default dbSetup;
Another way is if you have node version 14.8 or greater, you can use top-level await functionality, which is basically using await, not inside an async function, like this:
client = new MongoClient(uri, options);
dbPromise = await client.connect();
let db = await dbPromise.db("datatbase_name");
export default db;
I know we have various packages avalable on GitHub to query MongoDB using NodeJS. They don't solve my use case. I am looking to execute MongoDB Shell commands from NodeJS which are different from MongoDB NodeJS commands
Example: NodeJS
db.collection('user').find({})
Example: MongoDB Shell
db.getCollection('user').find({})
Notice 'collection' and 'getCollection' are different.
I want to save MongoDB Shell queries as text. How do I execute those queries once I have read them from the database?
Try this adapted from the MongoDB documentation:
const { MongoClient } = require("mongodb");
// Replace the uri string with your MongoDB deployment's connection string.
const uri =
"mongodb+srv://<user>:<password>#<cluster-url>?w=majority";
const client = new MongoClient(uri);
async function run() {
try {
await client.connect();
const db = client.db("myDatabase");
const result = await db.getCollection('user').find({});
console.log(result);
} finally {
await client.close();
}
}
run().catch(console.dir);
You can use child_process library of NodeJs but it is litte bit tricky. You must wait for mongo shell to finish its words.
const { spawn } = require("child_process");
const child = spawn("/usr/bin/mongo");
setTimeout(() => {
child.stdin.write("show dbs\n");
}, 5000);
child.stdout.on("data", (data) => {
console.log(`command line says:\n${data}`);
});
Lets Try this code -
let { MongoClient } = require('mongodb');
const url =
`mongodb+srv://${process.env.DB_USER}:${process.env.DB_USER_PASSWORD}#
${process.env.DB_CLUSTER}.mongodb.net`;
async function connectDatabase() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true,
useUnifiedTopology: true
});
const db = client.db("main");
return {
listings: db.collection("test_listings")
};
};
module.exports = { connectDatabase };
In node.js I had code like following:
mongoose.connect(dbURI, dbOptions)
.then(() => {
console.log("ok");
},
err => {
console.log('error: '+ err)
}
);
Now i want to do it with async/await syntax. So i could start with var mcResult = await mongoose.connect(dbURI, dbOptions);, afaik it will wait for operation, until it ends with any result (much like calling C function read() or fread() in syncronous mode).
But what should I write then? What does that return to the mcResult variable and how to check for an error or success? Basically I want a similar snippet, but written with proper async/await syntax.
Also I wonder because I have auto reconnect, among dbOptions:
dbOptions: {
autoReconnect: true,
reconnectTries: 999999999,
reconnectInterval: 3000
}
Would it "stuck" on await forever, in case if database connection is unavailble? I hope you can give me a clue on what would happen and how that would work.
Basically I want a similar snippet, but written with proper async/await syntax.
(async () => {
try {
await mongoose.connect(dbURI, dbOptions)
} catch (err) {
console.log('error: ' + err)
}
})()
Please try this, Below code has basics of db connectivity and a query :
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
let url = 'mongodb://localhost:27017/test';
const usersSchema = new Schema({
any: {}
}, {
strict: false
});
const Users = mongoose.model('users', usersSchema, 'users');
/** We've created schema as in mongoose you need schemas for your collections to do operations on them */
const dbConnect = async () => {
let db = null;
try {
/** In real-time you'll split DB connection(into another file) away from DB calls */
await mongoose.connect(url, { useNewUrlParser: true }); // await on a step makes process to wait until it's done/ err'd out.
db = mongoose.connection;
let dbResp = await Users.find({}).lean(); /** Gets all documents out of users collection.
Using .lean() to convert MongoDB documents to raw Js objects for accessing further. */
db.close(); // Needs to close connection, In general you don't close & re-create often. But needed for test scripts - You might use connection pooling in real-time.
return dbResp;
} catch (err) {
(db) && db.close(); /** Needs to close connection -
Only if mongoose.connect() is success & fails after it, as db connection is established by then. */
console.log('Error at dbConnect ::', err)
throw err;
}
}
dbConnect().then(res => console.log('Printing at callee ::', res)).catch(err => console.log('Err at Call ::', err));
As we're talking about async/await then few things I wanted to mention - await definitely needs it's function to be declared as async - otherwise it would throw an error. And it's recommended to wrap async/await code inside try/catch block.
const connectDb = async () => {
await mongoose.connect(dbUri, dbOptions).then(
() => {
console.info(`Connected to database`)
},
error => {
console.error(`Connection error: ${error.stack}`)
process.exit(1)
}
)
}
connectDb().catch(error => console.error(error))
Lets assume the use of then() is prohibited, you can result to this...
const connectDb = async () => {
try {
await mongoose.connect(dbConfig.url, dbConfigOptions)
console.info(`Connected to database on Worker process: ${process.pid}`)
} catch (error) {
console.error(`Connection error: ${error.stack} on Worker process: ${process.pid}`)
process.exit(1)
}
}
So I'm following the example here https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs, to optimize my lambda functions.
I've tried two approaches and tested them locally using serverless-offline and both don't seem to work.
First Approach
// endpoint file
import {connectToDatabase} from "lib/dbUtils.js";
let cachedDb = null;
export function post(event, context, callback) {
let response;
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase()
.then(//do other stuff
// lib/dbUtils.js
export async function connectToDatabase() {
if (cachedDb && cachedDb.serverConfig.isConnected()) {
console.log(" using cached db instance");
return cachedDb;
}
cachedDb = await mongoose.createConnection(
process.env.DB_URL,
async err => {
if (err) {
throw err;
}
}
);
return cachedDb;
}
Second Approach
global.cachedDb = null;
export function post(event, context, callback) {
let response;
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase()
.then(connection => createUser(event.body, connection))
// lib/dbUtils.js
export async function connectToDatabase() {
// eslint-disable-next-line
if (global.cachedDb && global.cachedDb.serverConfig.isConnected()) {
// eslint-disable-next-line
console.log(" using cached db instance");
// eslint-disable-next-line
return global.cachedDb;
}
// eslint-disable-next-line
global.cachedDb = await mongoose.createConnection(
process.env.DB_URL,
async err => {
if (err) {
throw err;
}
}
);
// eslint-disable-next-line
return global.cachedDb;
}
In both cases the using cached db instance console log does not run.
Why does this not work? Is this because of serverless-offline?
The answer is simple: serverless-offline doesn't simulate the full AWS. Use the AWS console to to make a real Lambda
The MongoDB Atlas guide is OK, but it's also worth checking the official AWS Lambda documentation describing the context option in each lambda:
callbackWaitsForEmptyEventLoop – Set to false to send the response right away when the callback executes, instead of waiting for the Node.js event loop to be empty. If false, any outstanding events will continue to run during the next invocation.
It's possible to run your code on a real Lambda and see using cached db instance on the console. Since MongoDB's JavaScript code is fairly poor, I've written out my own version below:
var MongoClient = require("mongodb").MongoClient
let db = null
var log = console.log.bind(console)
var print = function(object) {
return JSON.stringify(object, null, 2)
}
// Use your own credentials (and better yet, put them in environment variables)
const password = `notactuallyapassword`
const uri = `mongodb+srv://lambdauser:${password}#fakedomain.mongodb.net/test?retryWrites=true`
exports.handler = function(event, context, callback) {
log(`Calling MongoDB Atlas from AWS Lambda with event: ${print(event)}`)
var document = JSON.parse(JSON.stringify(event))
const databaseName = "myDatabase",
collectionName = "documents"
// See https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs
// and https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html#nodejs-prog-model-context-properties
context.callbackWaitsForEmptyEventLoop = false
return createDoc(databaseName, collectionName, document)
}
async function createDoc(databaseName, collectionName, document) {
var isConnected = db && db.serverConfig.isConnected()
if (isConnected) {
log(`Already connected to database, warm start!`)
} else {
log(`Connecting to database (cold start)`)
var client = await MongoClient.connect(uri)
db = client.db(databaseName)
}
var result = await db.collection(collectionName).insertOne(document)
log(`just created an entry into the ${collectionName} collection with id: ${result.insertedId}`)
// Don't close the connection thanks to context.callbackWaitsForEmptyEventLoop = false - this will re-use the connection on the next called (if it can re-use the same Lambda container)
return result
}
Use the Test button to run the lambda above twice in the AWS Lambda console.
The first time you run it you'll see Connecting to database (cold start)
The second time you'll see Already connected to database, warm start!
See the log output section in screenshot below:
I have A Mongoose Connection and at some point in my program I need to close it.
after logging the mongoose object several times, I have found that the following workd
mongoose.connection.base.connections[1].close();
Is There A Cleaner way to do this?
To close all connections in the Mongoose connection pool:
mongoose.disconnect();
Docs here.
I used a singleton in my code.
Exporting a promise
In one file I export a promise...
connection.js
const { createConnection } = require('mongoose');
function getConnection() {
return mongoose.createConnection(...).asPromise();
}
const connPromise = getConnection();
module.exports = connPromise;
other files...
const connPromise = require('./connection.js');
async function handler() {
// Connection starts on first module import
// This gets faster when the connection is done being created
const connection = await connPromise;
// Do stuff with the connection
return {
data: '',
and: '',
stuff: '',
};
}
When everything is shutting down, like in a lambda, also in connection.js:
process.on('SIGTERM', async () => {
console.info('[runtime] SIGTERM received');
const conn = await connPromise;
await conn.close();
console.info('[runtime] closed connection');
process.exit(0);
});
So by storing the connection in a promise, I can get it everywhere. I don't export the connection itself because getting it is async.
Exporting a getter function
I could have a function like
let connection = null;
function getConnection() {
if (!connection) {
connection = await createConnection(...).asPromise();
}
return connection;
}
module.exports = getConnection;
other files...
const getConnection = require('./connection.js');
async function handler() {
// Gets faster after the first call.
// connection created on first call.
const connection = await getConnection();
// Do stuff with the connection
return {
data: '',
and: '',
stuff: '',
};
}
Then your shutdown would become
process.on('SIGTERM', async () => {
console.info('[runtime] SIGTERM received');
if (connection) {
await connection.close();
console.info('[runtime] closed connection');
}
process.exit(0);
});
Summary
Overall, I like exporting the promise, since in every lambda function, I'm always using the database.
Since the exported promise means that the connection starts as soon as the module is imported, it's ready to go sooner than if I waited till the invocation of the getConnection() in an event handler.
For having only one connection, this is a good way to get ahold of it.