MongooseError: Operation `orders.deleteMany()` buffering timed out after 10000ms - node.js

when I run my app with npm run seeder
then I have face this error
I have checked my database connection carefully, it's ok.
also, I have checked my ordermodels file it's also ok. I have used MongoDB compass there is nothing problem. I don't know why showing buffering timed out.
MongooseError: Operation `orders.deleteMany()` buffering timed out after 10000ms
seeder.js
import mongoose from "mongoose";
import dotenv from "dotenv";
import colors from "colors";
import users from "./data/users.js";
import products from "./data/products.js";
import User from "./models/userModel.js";
import Product from "./models/productModel.js";
import Order from "./models/orderModel.js";
import connectDB from "./config/db.js";
dotenv.config();
connectDB();
const importData = async () => {
try {
await Order.deleteMany();
await Product.deleteMany();
await User.deleteMany();
const createUsers = await User.insertMany(users);
const adminUser = createUsers[0]._id;
const sampleProducts = products.map((product) => {
return { ...product, user: adminUser };
});
await Product.insertMany(sampleProducts);
console.log("Data Imported".green.inverse);
process.exit();
} catch (error) {
console.error(`${error}`.red.inverse);
process.exit(1);
}
};
const DeleteData = async () => {
try {
await Order.deleteMany();
await Product.deleteMany();
await User.deleteMany();
console.log("Data Deleted".red.inverse);
process.exit();
} catch (error) {
console.error(`${error}`.red.inverse);
process.exit(1);
}
};
if (process.argv[2] === "-d") {
DeleteData();
} else {
importData();
}

I have the same issue and I just did a research and I find that your MongoDB are trying to execute the function User.deleteMany() before the database is connected.
just put an await before connectDB();
await connectDB();

const importData = async () => {
try {
await connectDB();
await Category.collection.deleteMany({});
await Category.insertMany(categoryData);
console.log('Success ');
} catch (error) {
console.log('error in processing data ', error);
}
};
importData();

use following code for connect to mongodb
const mongoose = require('mongoose')
mongoose.connect(url, { useNewUrlParser: true, useUnifiedTopology: true, useCreateIndex: true })
after that you should removing the node_module folder and all .json files and reinstalling the mongoose and use npm update

I have faced the same issue, but I managed to resolve it.
So in package.json, in the scripts section I've added:
"data:import": "node backend/seeder",
"data:destroy": "node backend/seeder -d",
Then:
npm run data:import
After trying a couple of times, I've got Data Imported message in the console.
Note: I haven't uninstalled any node_modules.

Write useFindAndModify:true, in mongoose.connect(DB,{}) , because by default it is false and you are trying to delete something form database.
So, write it like this
mongoose.connect(DB,{
useNewUrlParser:true,
useCreateIndex:true,
useFindAndModify:true,
useUnifiedTopology: true
})

you can move connectDB() to importData() and deleteData(). and add await before connectDB(). like this:
const importData = async () => {
try {
await connectDB();
//...
}
}
and it worked for me.

Related

What is the correct way to make MongoDB Atlas connection available in middleware functions in Node.js/Express?

Environment
Node.js (18 LTS) / Express (^4.18.2)
MongoDB Native Driver (^4.12.0)
MongoDB Atlas (5.0.14)
Application Structure
.github
config
- mongodb_client.js
dist
middleware
node_modules
routes
src
views
.env
.gitignore
app.js
package.json
README.md
Connection Code
As a sanity check, this is the connection code that is provided in the MongoDB Atlas interface:
As a screenshot:
As code:
const { MongoClient, ServerApiVersion } = require('mongodb');
const uri = "mongodb+srv://admin:<password>#cluster0.******.mongodb.net/?retryWrites=true&w=majority";
const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
client.connect(err => {
const collection = client.db("test").collection("devices");
// perform actions on the collection object
client.close();
});
Desired Behaviour
The code snippet provided in the MongoDB Atlas interface performs the connection and subsequent database calls in one file.
However, I would like to:
Create a file that contains the MongoDB Atlas connection (e.g mongodb_client.js)
Export the connection so that it can be used in middleware files (e.g my_middleware_01.js)
So, in pseudo code, I imagine it would look something like this:
config / mongodb_client.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
const mongodb_client = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
// export the connection somehow
export { mongodb_client };
middleware / my_middleware_01.js
import { mongodb_client } from '../config/mongodb_client.js';
const api_myResource_get = async (req, res) => {
mongodb_client.open();
let collection = mongodb_client.db('myDatabase').collection('myCollection');
let result = await collection.findOne(query, options);
res.json({ result: result });
mongodb_client.close();
};
export { api_myResource_get };
What I've Tried
It seems I was grappling with this dynamic over a year ago and posted my solution here:
https://stackoverflow.com/a/70135909
However, I think that conventions have changed since then.
For example when instantiating the client, the current method seems to be:
const client = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
whereas previously it was something like:
await MongoClient.connect(connection_string);
I've Google searched:
how to make mongodb connection available in node.js middleware?
But all the results seem to reference this older convention and I'd like to ensure I am using best practice (and most recent conventions).
Related Questions and Resources
Passing Mongo DB Object DB to Express Middleware
What is best way to handle global connection of Mongodb in NodeJs
How do I manage MongoDB connections in a Node.js web application?
What is the difference between MongoClient and the client object which we get in the callback of MongoClient.connect() method
How to properly reuse connection to Mongodb across NodeJs application and modules
MongoDB Driver Connection Documentation
EDIT 01:
Below is one attempt which is returning the error:
TypeError: Cannot read properties of null (reading 'db')
config / mongodb_connection.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
class mongodb_connection {
static async open() {
if (this.conn) return this.conn;
this.conn = await MongoClient.connect(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
return this.conn;
}
}
mongodb_connection.conn = null;
mongodb_connection.url = connection_string;
export { mongodb_connection };
middleware / api_mongodb_get.js
import { mongodb_connection } from '../../config/mongodb_connection.js';
const api_mongodb_get = async (req, res) => {
try {
mongodb_connection.open();
const collection = mongodb_connection.conn.db('pages').collection('pages');
const result = await collection.findOne({ "my_key": "my value" });
res.json({ data: result });
mongodb_connection.close();
} catch (error) {
console.error(error);
res.status(500).send(error);
}
};
export { api_mongodb_get };
EDIT 02:
The following 'works' but I don't know if it is best practice or not.
In other words, I don't know if I am overlooking something that will cause undesired behavior.
config / mongodb_connection.js
import { MongoClient, ServerApiVersion } from 'mongodb';
const connection_string = process.env.MONGODB_CONNECTION_STRING;
const mongodb_connection = new MongoClient(connection_string, { useNewUrlParser: true, useUnifiedTopology: true, serverApi: ServerApiVersion.v1 });
export { mongodb_connection };
middleware / api_mongodb_get.js
import { mongodb_connection } from '../../config/mongodb_connection.js';
const api_mongodb_get = async (req, res) => {
try {
mongodb_connection.connect(async err => {
const collection = mongodb_connection.db('pages').collection('pages');
const result = await collection.findOne({ "my_key": "my value" });
res.json({ data: result });
mongodb_connection.close();
});
} catch (error) {
console.error(error);
res.status(500).send(error);
}
};
export { api_mongodb_get };
Insetead of using mongodb, use mongoose library to establish the connection.
Here is an example to establish the connection with mongodb cluster:
connectDb.js:
const dotenv = require('dotenv').config();
const DB_URL = process.env.DB_URL;
const mongoose = require('mongoose');
const connectDb = async () => {
try {
const connection = await mongoose.connect(DB_URL)
console.log(`Connected to database Successfully: ${connection}`)
} catch (error) {
console.log(error)
}
}
module.exports = connectDb;
and I think I don't need to mention that the DB_URL is the URL which is provided by the mondodb cluster.

TypeORM and MongoDB and Repositories: Cannot read property 'prototype' of undefined

I'm trying implement TypeORM with MongoDB using repositories. However, when I try to make use of repositories to manage the database, using the same structure as in this repository, things go a bit sideways. I'm getting the following error:
UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'prototype' of undefined
I have tried the following code:
import { Request, Response } from 'express';
import { getMongoRepository } from "typeorm";
import Task from "../models/Task";
export default class TasksController {
async listAll(request: Request, response: Response): Promise<Response> {
const tasksRepository = getMongoRepository(Task);
try {
const tasks = await tasksRepository.find();
return response.status(200).json({ "items": tasks });
} catch (err) {
return response.status(400).json({
message: err.message,
});
}
}
}
I know the error refers to implementing the .find() method. I have even managed to fetch the data, using a suggestion from this post replacing:
const tasks = await tasksRepository.find();
with
const tasks = await tasksRepository.createCursor(tasksRepository.find()).toArray();
but I still get the above mentioned error.
Anyone understands what's going on?
I have also managed to save data directly to the database through the use of the following script:
server.ts
import express from 'express';
import { createConnection } from 'typeorm'
const app = express();
const port = 3333;
createConnection();
app.use(express.json());
app.post('/tasks', (async (request, response) => {
const { item } = request.body;
task.item = item;
const task = new Task();
(await connection).mongoManager.save(task);
return response.send(task);
}))
app.listen(port, () =>
console.log(`Server running on port ${port}`)
);
TypeORM is not support mongodb v4.
https://github.com/nestjs/nest/issues/7798
You can use 3.7.0 instead.
I submitted a pull requests to resolve this. https://github.com/typeorm/typeorm/pull/8412 if anyone is looking for a workaround in the meantime.

TS2614: Module has no exported member 'NextHandleFunction'

Error:
node_modules/#types/body-parser/index.d.ts:14:10 - error TS2614: Module '"../../../src/connect"' has no exported member 'NextHandleFunction'. Did you mean to use 'import NextHandleFunction from "../../../src/connect"' instead? 14 import { NextHandleFunction } from 'connect';
I'm trying to use typescript into my nodejs project instead of javascript. I have followed a few tutorials and looked into some Github repositories, even though I'm getting this weird while compiling, but my server is getting started.
import * as mongoose from 'mongoose';
type TInput = {
db: string;
}
export default ({db}: TInput) => {
const connect = () => {
mongoose
.connect(
db,
{ useNewUrlParser: true }
)
.then(() => {
return console.info(`Successfully connected to ${db}`);
})
.catch(error => {
console.error('Error connecting to database: ', error);
return process.exit(1);
});
};
connect();
mongoose.connection.on('disconnected', connect);
};
try this compiler option inside tsconfig.json
"skipLibCheck": true,

How to mock typeorm connection

In integration tests I am using the following snippets to create connection
import {Connection, createConnection} from 'typeorm';
// #ts-ignore
import options from './../../../ormconfig.js';
export function connectDb() {
let con: Connection;
beforeAll(async () => {
con = await createConnection(options);
});
afterAll(async () => {
await con.close();
});
}
I am trying to unit test a class which calls typeorm repository in one of its method and without call that helper function connectDb() above I get the following error which is expected of course.
ConnectionNotFoundError: Connection "default" was not found.
My question is how can I mock connection. I have tried the following without any success
import typeorm, {createConnection} from 'typeorm';
// #ts-ignore
import options from "./../../../ormconfig.js";
const mockedTypeorm = typeorm as jest.Mocked<typeof typeorm>;
jest.mock('typeorm');
beforeEach(() => {
//mockedTypeorm.createConnection.mockImplementation(() => createConnection(options)); //Failed
mockedTypeorm.createConnection = jest.fn().mockImplementation(() => typeorm.Connection);
MethodRepository.prototype.changeMethod = jest.fn().mockImplementation(() => {
return true;
});
});
Running tests with that kind of mocking gives this error
TypeError: decorator is not a function
Note: if I call connectDb() in tests everything works fine. But I don't want to do that since it takes too much time as some data are inserted into db before running any test.
Some codes have been omitted for simplicity. Any help will be appreciated
After a bunch of research and experiment I've ended up with this solution. I hope it works for someone else who experienced the same issue...
it does not need any DB connection
testing service layer content, not the DB layer itself
test can cover all the case I need to test without hassle, I just need to provide the right output to related typeorm methods.
This is the method I want to test
#Injectable()
export class TemplatesService {
constructor(private readonly templatesRepository: TemplatesRepository) {}
async list(filter: ListTemplatesReqDTO) {
const qb = this.templatesRepository.createQueryBuilder("tl");
const { searchQuery, ...otherFilters } = filter;
if (filter.languages) {
qb.where("tl.language IN (:...languages)");
}
if (filter.templateTypes) {
qb.where("tl.templateType IN (:...templateTypes)");
}
if (searchQuery) {
qb.where("tl.name LIKE :searchQuery", { searchQuery: `%${searchQuery}%` });
}
if (filter.skip) {
qb.skip(filter.skip);
}
if (filter.take) {
qb.take(filter.take);
}
if (filter.sort) {
qb.orderBy(filter.sort, filter.order === "ASC" ? "ASC" : "DESC");
}
return qb.setParameters(otherFilters).getManyAndCount();
}
...
}
This is the test:
import { SinonStub, createSandbox, restore, stub } from "sinon";
import * as typeorm from "typeorm";
describe("TemplatesService", () => {
let service: TemplatesService;
let repo: TemplatesRepository;
const sandbox = createSandbox();
const connectionStub = sandbox.createStubInstance(typeorm.Connection);
const templatesRepoStub = sandbox.createStubInstance(TemplatesRepository);
const queryBuilderStub = sandbox.createStubInstance(typeorm.SelectQueryBuilder);
stub(typeorm, "createConnection").resolves((connectionStub as unknown) as typeorm.Connection);
connectionStub.getCustomRepository
.withArgs(TemplatesRepository)
.returns((templatesRepoStub as unknown) as TemplatesRepository);
beforeAll(async () => {
const builder: TestingModuleBuilder = Test.createTestingModule({
imports: [
TypeOrmModule.forRoot({
type: "postgres",
database: "test",
entities: [Template],
synchronize: true,
dropSchema: true
})
],
providers: [ApiGuard, TemplatesService, TemplatesRepository],
controllers: []
});
const module = await builder.compile();
service = module.get<TemplatesService>(TemplatesService);
repo = module.get<TemplatesRepository>(TemplatesRepository);
});
beforeEach(async () => {
// do something
});
afterEach(() => {
sandbox.restore();
restore();
});
it("Service should be defined", () => {
expect(service).toBeDefined();
});
describe("list", () => {
let fakeCreateQueryBuilder;
it("should return records", async () => {
stub(queryBuilderStub, "skip" as any).returnsThis();
stub(queryBuilderStub, "take" as any).returnsThis();
stub(queryBuilderStub, "sort" as any).returnsThis();
stub(queryBuilderStub, "setParameters" as any).returnsThis();
stub(queryBuilderStub, "getManyAndCount" as any).resolves([
templatesRepoMocksListSuccess,
templatesRepoMocksListSuccess.length
]);
fakeCreateQueryBuilder = stub(repo, "createQueryBuilder" as any).returns(queryBuilderStub);
const [items, totalCount] = await service.list({});
expect(fakeCreateQueryBuilder.calledOnce).toBe(true);
expect(fakeCreateQueryBuilder.calledOnce).toBe(true);
expect(items.length).toBeGreaterThan(0);
expect(totalCount).toBeGreaterThan(0);
});
});
});
cheers!

Knex pool full on migration

I'm trying to get started with knex.js and I can't get migrations to work. Knex works fine for my API calls. Here's my setup:
knexfile.js
const env = process.env;
module.exports = {
client: 'mysql',
connection: {
host: env.DB_HOST,
database: env.DB_NAME,
user: env.DB_USER,
password: env.DB_PASSWORD,
port: env.PORT
},
pool: {
min: 0,
max: 50
},
migrations: {
directory: './db/migrations',
tableName: 'knex_migrations'
},
seeds: {
directory: './db/seeds'
}
};
knex.js
const config = require('../knexfile.js');
module.exports = require('knex')(config);
events.js
const express = require('express');
const router = express.Router();
const knex = require('../../db/knex.js');
// GET api/events
router.get('/', (req, res) => {
knex('events')
.then(events => { res.send(events) }
.catch(err => { console.log(err); })
});
module.exports = router;
and then I have a file in the migrations folder with:
exports.up = function(knex) {
return knex.schema.createTable('users', function (t) {
t.increments('id').primary()
t.string('username').notNullable()
t.string('password').notNullable()
t.timestamps(false, true)
}).then(() => { console.log('created users table') })
.catch((err) => { throw err} )
.finally(() => { knex.destroy() })
};
exports.down = function(knex) {
return knex.schema.dropTableIfExists('users')
};
When I run knex migrate:latest I get TimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
I know similar questions have been asked before, but I can't seem to find any that shed light on my particular situation. I've tried adding a knex.destroy() to the end of my GET request but that doesn't seem to help (it just makes the connection unusable if I add other request handlers below).
I did try checking the knex.client.pool in a finally clause at the end of the GET request. numUsed was 0, numFree was 1, numPendingAcquires and numPendingCreates were both 0. I do find it odd that numFree was only 1 given that my knexfile specifies max 50. Any advice greatly appreciated.
Following #technogeek1995's comment, the answer turned out to be adding require('dotenv').config({path: '../.env'}); to knexfile.js (in retrospect, this part seems obvious), and running the cli from the same directory. Hope this helps someone else.

Resources