What is the correct way to make MongoDB Atlas connection available in middleware functions in Node.js/Express? - node.js

Environment
Node.js (18 LTS) / Express (^4.18.2)
MongoDB Native Driver (^4.12.0)
MongoDB Atlas (5.0.14)
Application Structure
.github
config
- mongodb_client.js
dist
middleware
node_modules
routes
src
views
.env
.gitignore
app.js
package.json
README.md
Connection Code
As a sanity check, this is the connection code that is provided in the MongoDB Atlas interface:
As a screenshot:
As code:
const { MongoClient, ServerApiVersion } = require('mongodb');
const uri = "mongodb+srv://admin:<password>#cluster0.******.mongodb.net/?retryWrites=true&w=majority";
const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
client.connect(err => {
const collection = client.db("test").collection("devices");
// perform actions on the collection object
client.close();
});
Desired Behaviour
The code snippet provided in the MongoDB Atlas interface performs the connection and subsequent database calls in one file.
However, I would like to:
Create a file that contains the MongoDB Atlas connection (e.g mongodb_client.js)
Export the connection so that it can be used in middleware files (e.g my_middleware_01.js)
So, in pseudo code, I imagine it would look something like this:
config / mongodb_client.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
const mongodb_client = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
// export the connection somehow
export { mongodb_client };
middleware / my_middleware_01.js
import { mongodb_client } from '../config/mongodb_client.js';
const api_myResource_get = async (req, res) => {
mongodb_client.open();
let collection = mongodb_client.db('myDatabase').collection('myCollection');
let result = await collection.findOne(query, options);
res.json({ result: result });
mongodb_client.close();
};
export { api_myResource_get };
What I've Tried
It seems I was grappling with this dynamic over a year ago and posted my solution here:
https://stackoverflow.com/a/70135909
However, I think that conventions have changed since then.
For example when instantiating the client, the current method seems to be:
const client = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
whereas previously it was something like:
await MongoClient.connect(connection_string);
I've Google searched:
how to make mongodb connection available in node.js middleware?
But all the results seem to reference this older convention and I'd like to ensure I am using best practice (and most recent conventions).
Related Questions and Resources
Passing Mongo DB Object DB to Express Middleware
What is best way to handle global connection of Mongodb in NodeJs
How do I manage MongoDB connections in a Node.js web application?
What is the difference between MongoClient and the client object which we get in the callback of MongoClient.connect() method
How to properly reuse connection to Mongodb across NodeJs application and modules
MongoDB Driver Connection Documentation
EDIT 01:
Below is one attempt which is returning the error:
TypeError: Cannot read properties of null (reading 'db')
config / mongodb_connection.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
class mongodb_connection {
static async open() {
if (this.conn) return this.conn;
this.conn = await MongoClient.connect(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
return this.conn;
}
}
mongodb_connection.conn = null;
mongodb_connection.url = connection_string;
export { mongodb_connection };
middleware / api_mongodb_get.js
import { mongodb_connection } from '../../config/mongodb_connection.js';
const api_mongodb_get = async (req, res) => {
try {
mongodb_connection.open();
const collection = mongodb_connection.conn.db('pages').collection('pages');
const result = await collection.findOne({ "my_key": "my value" });
res.json({ data: result });
mongodb_connection.close();
} catch (error) {
console.error(error);
res.status(500).send(error);
}
};
export { api_mongodb_get };
EDIT 02:
The following 'works' but I don't know if it is best practice or not.
In other words, I don't know if I am overlooking something that will cause undesired behavior.
config / mongodb_connection.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
const mongodb_connection = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
export { mongodb_connection };
middleware / api_mongodb_get.js
import { mongodb_connection } from '../../config/mongodb_connection.js';
const api_mongodb_get = async (req, res) => {
try {
mongodb_connection.connect(async err => {
const collection = mongodb_connection.db('pages').collection('pages');
const result = await collection.findOne({ "my_key": "my value" });
res.json({ data: result });
mongodb_connection.close();
});
} catch (error) {
console.error(error);
res.status(500).send(error);
}
};
export { api_mongodb_get };

Insetead of using mongodb, use mongoose library to establish the connection.
Here is an example to establish the connection with mongodb cluster:
connectDb.js:
const dotenv = require('dotenv').config();
const DB_URL = process.env.DB_URL;
const mongoose = require('mongoose');
const connectDb = async () => {
try {
const connection = await mongoose.connect(DB_URL)
console.log(`Connected to database Successfully: ${connection}`)
} catch (error) {
console.log(error)
}
}
module.exports = connectDb;
and I think I don't need to mention that the DB_URL is the URL which is provided by the mondodb cluster.

Related

TypeScript: All declarations of 'Query' must have identical type parameters

I get "All declarations of 'Query' must have identical type parameters." error when I hover over "Query" in interface section.
import mongoose from "mongoose";
import * as redis from "redis";
declare module "mongoose" {
interface Query {
cache():this;
useCache:boolean;
}
};
const client = redis.createClient({ url: process.env.REDIS });
client.on("connect", () => console.log("Redis Connection Is Successful!"));
client.on("err", (err) => console.log("Redis Client Error:", err));
client.connect();
//Hooking into mongoose's query generation and execution process
//in order to make the caching reusable in the codebase
const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.cache = function() {
this.useCache = true;
return this;
}

How to get query result from postgraphile running as a library

I have postgraphile running as an express middleware. For example:
const pgMiddleware = postgraphile(pool, SCHEMA, postgraphileConfig);
app.use(pgMiddleware);
How to get or intercept the result of a query or mutation without having a separate client?
For example when I send the below query
query {
personById(id: 1){
firstname
}
}
I want to be able to get the data sent back inside the same express app. How can I do that?
I believe what you are asking for is to be able to execute GraphQL operations against a PostGraphile schema from other routes/middlewares in Express without needing to make additional http requests. This is called schema only usage and you will specifically want to use withPostGraphileContext to execute your request and process results:
import type { Express } from "express";
import type { Pool } from "pg";
import {
gql,
makeProcessSchemaPlugin,
postgraphile,
withPostGraphileContext,
} from "postgraphile";
import PgSimplifyInflectorPlugin from "#graphile-contrib/pg-simplify-inflector";
import type { GraphQLSchema } from "graphql";
import { graphql } from "graphql";
// Register your middlewares with express
const schemaOnlyUsageApp = (app: Express, pool: Pool) => {
let schema: GraphQLSchema;
// This plugin will execute a callback each time the PostGraphile
// GraphQl schema is rebuit.
const schemaProcessorPlugin = makeProcessSchemaPlugin((newSchema) => {
schema = newSchema;
return schema;
});
// Register the PostGraphile middleware as normal for requests on /graphql (and /graphiql)
app.use(
postgraphile(pool, "my_schema", {
simpleCollections: "omit",
dynamicJson: true,
legacyRelations: "omit",
setofFunctionsContainNulls: false,
appendPlugins: [PgSimplifyInflectorPlugin, schemaProcessorPlugin],
watchPg: true,
graphiql: true,
enhanceGraphiql: true,
showErrorStack: true,
allowExplain: true,
})
);
// custom route that will execute a predefined gql query directly against the schema
app.get("/posts", async (req, res) => {
// arbitrary gql query
const query = gql`
query posts {
posts {
edges {
node {
id
title
body
likeCount
createdAt
}
}
}
}
`;
const result = await withPostGraphileContext(
{
// Reuse your pool to avoid creating additional connections
pgPool: pool,
},
async (context) => {
// execute your query directly and get results without making
// an additional http request!
const queryResult = await graphql({
schema,
source: query.loc?.source || "",
contextValue: { ...context },
});
return queryResult;
}
);
res.send(result);
});
};
export default schemaOnlyUsageApp;

MongooseError: Operation `orders.deleteMany()` buffering timed out after 10000ms

when I run my app with npm run seeder
then I have face this error
I have checked my database connection carefully, it's ok.
also, I have checked my ordermodels file it's also ok. I have used MongoDB compass there is nothing problem. I don't know why showing buffering timed out.
MongooseError: Operation `orders.deleteMany()` buffering timed out after 10000ms
seeder.js
import mongoose from "mongoose";
import dotenv from "dotenv";
import colors from "colors";
import users from "./data/users.js";
import products from "./data/products.js";
import User from "./models/userModel.js";
import Product from "./models/productModel.js";
import Order from "./models/orderModel.js";
import connectDB from "./config/db.js";
dotenv.config();
connectDB();
const importData = async () => {
try {
await Order.deleteMany();
await Product.deleteMany();
await User.deleteMany();
const createUsers = await User.insertMany(users);
const adminUser = createUsers[0]._id;
const sampleProducts = products.map((product) => {
return { ...product, user: adminUser };
});
await Product.insertMany(sampleProducts);
console.log("Data Imported".green.inverse);
process.exit();
} catch (error) {
console.error(`${error}`.red.inverse);
process.exit(1);
}
};
const DeleteData = async () => {
try {
await Order.deleteMany();
await Product.deleteMany();
await User.deleteMany();
console.log("Data Deleted".red.inverse);
process.exit();
} catch (error) {
console.error(`${error}`.red.inverse);
process.exit(1);
}
};
if (process.argv[2] === "-d") {
DeleteData();
} else {
importData();
}
I have the same issue and I just did a research and I find that your MongoDB are trying to execute the function User.deleteMany() before the database is connected.
just put an await before connectDB();
await connectDB();
const importData = async () => {
try {
await connectDB();
await Category.collection.deleteMany({});
await Category.insertMany(categoryData);
console.log('Success ');
} catch (error) {
console.log('error in processing data ', error);
}
};
importData();
use following code for connect to mongodb
const mongoose = require('mongoose')
mongoose.connect(url, { useNewUrlParser: true, useUnifiedTopology: true, useCreateIndex: true })
after that you should removing the node_module folder and all .json files and reinstalling the mongoose and use npm update
I have faced the same issue, but I managed to resolve it.
So in package.json, in the scripts section I've added:
"data:import": "node backend/seeder",
"data:destroy": "node backend/seeder -d",
Then:
npm run data:import
After trying a couple of times, I've got Data Imported message in the console.
Note: I haven't uninstalled any node_modules.
Write useFindAndModify:true, in mongoose.connect(DB,{}) , because by default it is false and you are trying to delete something form database.
So, write it like this
mongoose.connect(DB,{
useNewUrlParser:true,
useCreateIndex:true,
useFindAndModify:true,
useUnifiedTopology: true
})
you can move connectDB() to importData() and deleteData(). and add await before connectDB(). like this:
const importData = async () => {
try {
await connectDB();
//...
}
}
and it worked for me.

TS2614: Module has no exported member 'NextHandleFunction'

Error:
node_modules/#types/body-parser/index.d.ts:14:10 - error TS2614: Module '"../../../src/connect"' has no exported member 'NextHandleFunction'. Did you mean to use 'import NextHandleFunction from "../../../src/connect"' instead? 14 import { NextHandleFunction } from 'connect';
I'm trying to use typescript into my nodejs project instead of javascript. I have followed a few tutorials and looked into some Github repositories, even though I'm getting this weird while compiling, but my server is getting started.
import * as mongoose from 'mongoose';
type TInput = {
db: string;
}
export default ({db}: TInput) => {
const connect = () => {
mongoose
.connect(
db,
{ useNewUrlParser: true }
)
.then(() => {
return console.info(`Successfully connected to ${db}`);
})
.catch(error => {
console.error('Error connecting to database: ', error);
return process.exit(1);
});
};
connect();
mongoose.connection.on('disconnected', connect);
};
try this compiler option inside tsconfig.json
"skipLibCheck": true,

Knex pool full on migration

I'm trying to get started with knex.js and I can't get migrations to work. Knex works fine for my API calls. Here's my setup:
knexfile.js
const env = process.env;
module.exports = {
client: 'mysql',
connection: {
host: env.DB_HOST,
database: env.DB_NAME,
user: env.DB_USER,
password: env.DB_PASSWORD,
port: env.PORT
},
pool: {
min: 0,
max: 50
},
migrations: {
directory: './db/migrations',
tableName: 'knex_migrations'
},
seeds: {
directory: './db/seeds'
}
};
knex.js
const config = require('../knexfile.js');
module.exports = require('knex')(config);
events.js
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
// GET api/events
router.get('/', (req, res) => {
knex('events')
.then(events => { res.send(events) }
.catch(err => { console.log(err); })
});
module.exports = router;
and then I have a file in the migrations folder with:
exports.up = function(knex) {
return knex.schema.createTable('users', function (t) {
t.increments('id').primary()
t.string('username').notNullable()
t.string('password').notNullable()
t.timestamps(false, true)
}).then(() => { console.log('created users table') })
.catch((err) => { throw err} )
.finally(() => { knex.destroy() })
};
exports.down = function(knex) {
return knex.schema.dropTableIfExists('users')
};
When I run knex migrate:latest I get TimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
I know similar questions have been asked before, but I can't seem to find any that shed light on my particular situation. I've tried adding a knex.destroy() to the end of my GET request but that doesn't seem to help (it just makes the connection unusable if I add other request handlers below).
I did try checking the knex.client.pool in a finally clause at the end of the GET request. numUsed was 0, numFree was 1, numPendingAcquires and numPendingCreates were both 0. I do find it odd that numFree was only 1 given that my knexfile specifies max 50. Any advice greatly appreciated.
Following #technogeek1995's comment, the answer turned out to be adding require('dotenv').config({path: '../.env'}); to knexfile.js (in retrospect, this part seems obvious), and running the cli from the same directory. Hope this helps someone else.

Resources