Nodejs Mongoose 'Operation `XXX.find()` buffering timed out after 10000ms' - node.js

index.ts is the entry point of this NodeJS program.
This is the code in index.ts:
import JobWorker from "./worker";
import { SwitchPlan } from "./jobs";
const worker = new JobWorker();
worker.addJob(SwitchPlan);
This is worker.ts:
import { CronJob } from "cron";
import mongoose from "mongoose";
import Config from "./config";
import logger from "./logger";
export default class JobWorker {
private jobs: CronJob[];
private config: {
NAME: string;
MONGO_URL: string;
};
constructor() {
this.config = Config;
this.connectDB();
this.jobs = [];
}
public async connectDB(): Promise<void> {
try {
await mongoose.connect(this.config.MONGO_URL,
{ useUnifiedTopology: true, useNewUrlParser: true, useCreateIndex: true },
);
logger.info("\nMONGODB has been connected\n");
} catch(err) {
logger.error("ERROR occurred while connecting to the database");
}
}
addJob(cronJob: CronJob) {
this.jobs.push(cronJob);
}
}
This is jobs.ts:
import moment from "moment";
import {
DatabaseOperations, Vehicle,
Plan1Doc, Plan1, VehicleDoc
} from "common-lib";
import logger from "../logger";
import { CronJob } from "cron";
const vehicleOps = new DatabaseOperations(Vehicle);
const SwitchPlan = new CronJob("25 * * * * *", async (): Promise<void> => {
const date: Date = moment(new Date()).startOf("date").toDate();
const expiringVehicles: VehicleDoc[] = vehicleOps.getAllDocuments(
{ "inspection.startTime": {
"$gte": date, "$lte": moment(date).startOf("date").add(1, "day").toDate()
}
},
{}, { pageNo: 0, limit: 0 }
).then((result: any) => {
logger.info("dsada");
}).catch((err: any) => {
logger.info("ssd");
});
});
SwitchPlan.start();
export { SwitchPlan };
I have omitted parts of code which are irrelevant to this problem. I ran this code through a debugger and there's no issue with the config. MonggoDB connected is getting printed at the start of the program. However the then block after getAllDocuments in jobs.ts is never reached and it always goes in the error block with the message, Operation vehicleinventories.find() buffering timed out after 10000ms. The getAllDocuments uses MongoDB's find() method and is working correctly because I am using this method in other projects where I have no such issues.
So far I have tried, deleting Mongoose from node_modules and reinstalling, tried connecting to MongoDB running on localhost, but the issue remains unsolved.
EDIT: DatabaseOperations class:
import { Model, Schema } from "mongoose";
class DatabaseOperations {
private dbModel: Model<any>;
constructor(dbModel: Model<any>) {
this.dbModel = dbModel;
}
getAllDocuments(
query: any,
projections: any,
options: { pageNo: number; limit: number },
sort?: any
): any {
const offset = options.limit * options.pageNo;
return this.dbModel
.find(query, projections)
.skip(offset)
.limit(options.limit)
.sort(sort ? sort : { createdAt: -1 })
.lean();
}
}

in your jobs.ts file you have the following line
SwitchToTier1Plan.start();
This line is called the moment you required the class file, hence before mongoose is connected, and all the models defined. Could this be the issue?
Another thing I noted is u are using mongoose.connect which may be wrong since mongoose.connect creates a global connection.
which means each new Worker you will be attempting to override the mongoose property with previous connection
Though i'm not sure what the implication is, but it could be because your .find could be using the old connection.
Since you are writing class, I would recommend using mongoose.createConnection which creates a new connection for each class initiation.

Related

Error: Cannot get orders: TypeError: Cannot read properties of undefined (reading 'connect')

I'm trying to build an API with express and pg. Whenever I try to access an endpoint that is related to a query to the database I get the error above.
My handler function is as follows:
import { Request, Response, Router, NextFunction } from 'express';
import { Orders } from '../models/order';
const orders = new Orders;
const index = async (_req: Request, res: Response, next: NextFunction) => {
try {
const ordersList = await orders.index();
res.json(ordersList);
} catch (err) {
next(err)
}
}
const ordersRoute = Router();
ordersRoute.get('/', index);
This handler refers to the following model:
import { Pool } from 'pg';
client = new Pool({
host: POSTGRES_HOST,
database: POSTGRES_DB,
user: POSTGRES_USER,
password: POSTGRES_PASSWORD,
port: parseInt(POSTGRES_PORT as string, 10)
export class Orders {
async index(): Promise<Order[]> {
try {
const conn = await client.connect();
const sql = 'SELECT * FROM orders';
const result = await conn.query(sql);
conn.release();
return result.rows;
} catch (err) {
throw new Error(`Cannot get orders: ${err}`);
}
}
}
anytime I try to access the endpoint I get
Error: Cannot get orders: TypeError: Cannot read properties of
undefined (reading 'connect')
in the console.
any idea how to fix ?
So how things work in your case.
All modules are read by nodejs starting from your index one by one from top to bottom.
In your script, you declare client and then export a class. In that case, your client is setup, then you export a class, and that file is completed, meaning that the only thing that remains is the exported thing. So when you try to use the exported class, you'll not have the same context as in your module.
You need to export the client too and import it where you need that class or to include client definition inside the class

Mocha, Supertest and Mongo Memory server, cannot setup hooks properly

I have a Nodejs, Express server that uses Mongodb as the Database. I am trying to write some tests but I cannot get it configured properly. I need to create a mongoose connection for each block ( .test.ts file) once and then clean up the db for the other tests. Depending on how I approach this I get two different behaviours. But first my setup.
user.controller.test.ts
import { suite, test, expect } from "../utils/index";
import expressLoader from "#/loaders/express";
import { Logger } from "winston";
import express from "express";
import request from "supertest";
import { UserAccountModel } from "#/models/UserAccount";
import setup from "../setup";
setup();
import { app } from "#/server";
let loggerMock: Logger;
describe("POST /api/user/account/", function () {
it("it should have status code 200 and create an user account", async function () {
//GIVEN
const userCreateRequest = {
email: "test#gmail.com",
userId: "testUserId",
};
//WHEN
await request(app)
.post("/api/user/account/")
.send(userCreateRequest)
.expect(200);
//SHOULD
const cnt: number = await UserAccountModel.count();
expect(cnt).to.equal(1);
});
});
And my other test post.repository.test.ts
import { suite, test, expect } from "../../utils/index";
import { PostModel } from "#/models/Posts/Post";
import PostRepository from "#/repositories/posts.repository";
import { PostType } from "#/interfaces/Posts";
import { Logger } from "winston";
#suite
class PostRepositoryTests {
private loggerMock: Logger;
private SUT: PostRepository = new PostRepository({});
#test async "Should create two posts"() {
//GIVEN
const given = {
id: "jobId123",
};
//WHEN
await this.SUT.CreatePost(given);
//SHOULD
const cnt: number = await PostModel.count();
expect(cnt).to.equal(1);
}
}
And my setup
setup.ts
import { MongoMemoryServer } from "mongodb-memory-server";
import mongoose from "mongoose";
export = () => {
let mongoServer: MongoMemoryServer;
before(function () {
console.log("Before");
return MongoMemoryServer.create().then(function (mServer) {
mongoServer = mServer;
const mongoUri = mongoServer.getUri();
return mongoose.connect(mongoUri);
});
});
after(function () {
console.log("After");
return mongoose.disconnect().then(function () {
return mongoServer.stop(true);
});
});
};
With the above setup I get
1) "before all" hook in "{root}":
MongooseError: Can't call `openUri()` on an active connection with different connection strings. Make sure you aren't calling `mongoose.connect()` multiple times. See: https://mongoosejs.com/docs/connections.html#multiple_connections
But if I don't import the setup.ts and I rename it to setup.test.ts, then it works but the data isn't cleared after each run so I actually have 10 new users created instead of 1. Every time I run it, it works and doesn't clear the data after it's finished.
Also I have a big issue where the tests hang and don't seem to finish. I am guessing that is because of the async, await in the tests or because of the hooks hanging.
What I want to happen is:
Each test should have it's own setup, the mongo memory server should be clean every time.
The tests should use async and await and not hang.
Somehow export the setup from 1) as a utility function so that I can reuse it in my code

how to capture nestjs bootstrap errors in a log file

I'm trying to capture all logs(bootstrap, app error messages, db connection error messages) into a single log file in nestjs.
As of now I'm using a custom logger to so. Below is my custom logger code
logger.ts
import * as winston from 'winston';
import * as chalk from 'chalk';
import PrettyError from 'pretty-error';
import { LoggerOptions } from 'winston';
export class LoggerService {
private readonly logger: winston.Logger;
private readonly prettyError = new PrettyError();
public static loggerOptions: LoggerOptions = {
transports: [
new winston.transports.File({
filename: 'logs/mgmtserver-main.log',
format: winston.format.json()
}),
],
};
constructor(private context: string, transport?) {
this.logger = (winston as any).createLogger(LoggerService.loggerOptions);
this.prettyError.skipNodeFiles();
this.prettyError.skipPackage('express', '#nestjs/common', '#nestjs/core');
}
get Logger(): winston.Logger {
return this.logger;
}
static configGlobal(options?: LoggerOptions) {
this.loggerOptions = options;
}
log(message: string): void {
const currentDate = new Date();
this.logger.info(message, {
timestamp: currentDate.toISOString(),
context: this.context,
});
this.formatedLog('info', message);
}
error(message: string, trace?: any): void {
const currentDate = new Date();
this.logger.error(`${message} -> (${trace || 'trace not provided !'})`, {
timestamp: currentDate.toISOString(),
context: this.context,
});
this.formatedLog('error', message, trace);
}
warn(message: string): void {
const currentDate = new Date();
this.logger.warn(message, {
timestamp: currentDate.toISOString(),
context: this.context,
});
this.formatedLog('warn', message);
}
overrideOptions(options: LoggerOptions) {
this.logger.configure(options);
}
// this method just for printing a cool log in your terminal , using chalk
private formatedLog(level: string, message: string, error?): void {
let result = '';
const color = chalk.default;
const currentDate = new Date();
const time = `${currentDate.getHours()}:${currentDate.getMinutes()}:${currentDate.getSeconds()}`;
switch (level) {
case 'info':
result = `[${color.blue('INFO')}] ${color.dim.yellow.bold.underline(time)} [${color.green(
this.context,
)}] ${message}`;
break;
case 'error':
result = `[${color.red('ERR')}] ${color.dim.yellow.bold.underline(time)} [${color.green(
this.context,
)}] ${message}`;
break;
case 'warn':
result = `[${color.yellow('WARN')}] ${color.dim.yellow.bold.underline(time)} [${color.green(
this.context,
)}] ${message}`;
break;
default:
break;
}
console.log(result);
}
}
I'm able to log application erro messages(err, warn, info) using the above logger in any file like below
import { LoggerService } from 'logger';
private readonly logger: LoggerService = new LoggerService(RegistrationService.name);
this.logger.warn('this is a warn message');
my main.ts looks like below
import { ValidationPipe, Logger } from "#nestjs/common";
import { NestFactory } from "#nestjs/core";
import { ConfigService } from '#nestjs/config';
import { AppModule } from "./app.module";
import { WinstonModule } from 'nest-winston';
import * as winston from 'winston';
import { LoggerService } from "logger";
async function bootstrap() {
const app = await NestFactory.create(AppModule, {
logger: new LoggerService('Main'), abortOnError: false
});
app.enableCors();
await app.listen(3000);
console.log(`Application is running on: ${await app.getUrl()}`);
}
bootstrap();
The issue is I'm not able to capture the Nestfactory.create. bootstrap errors in the log file. They are getting printed on the console but not to log file.
For example, the below bootstrap errors are getting printed on console but not into the log file.
[INFO] 15:12:50 [Main] Starting Nest application...
[ERR] 15:12:50 [Main] Nest cannot create the AuthorisationModule instance.
The module at index [3] of the AuthorisationModule "imports" array is undefined.
Potential causes:
- A circular dependency between modules. Use forwardRef() to avoid it. Read more: https://docs.nestjs.com/fundamentals/circular-dependency
- The module at index [3] is of type "undefined". Check your import statements and the type of the module.
Please help me. Your help is much appreciated.
That is because your logger module isn't initialized until Nest finishes initializing the App context. Your logger will capture all the errors after the your App is running but not before.
However you can make use of node's inbuilt events to log/file these exceptions
process.on('uncaughtException', err => {
console.error('There was an uncaught error', err)
process.exit(1) //mandatory (as per the Node.js docs)
})
or, for Promises,
process.on('unhandledRejection', err => {
console.error('There was an uncaught error', err)
process.exit(1) //mandatory (as per the Node.js docs)
})
https://nodejs.dev/learn/error-handling-in-nodejs#catching-uncaught-exceptions

Need to find the error with connecting subscription with schema stitching

I am using apollo-server-express for graphql back-end. I am going to process only mutations there, but I want to redirect query and subscription on hasura by means of schema stitching with introspection. Queries through apollo-server to hasura are working fine and returning the expected data.
But subscriptions are not working and I am getting this error: " Expected Iterable, but did not find one for field subscription_root.users".
And besides, server hasura is receiving events:
But apollo-server resents the answer from hasura. It is not the first day I suffer with this and I can not understand what the problem is.
In the editor hasura subscriptions work.
Link to full code
If you need any additional info, I will gladly provide it to you.
import {
introspectSchema,
makeExecutableSchema,
makeRemoteExecutableSchema,
mergeSchemas,
transformSchema,
FilterRootFields
} from 'graphql-tools';
import { HttpLink } from 'apollo-link-http';
import nodeFetch from 'node-fetch';
import { resolvers } from './resolvers';
import { hasRoleResolver } from './directives';
import { typeDefs } from './types';
import { WebSocketLink } from 'apollo-link-ws';
import { split } from 'apollo-link';
import { getMainDefinition } from 'apollo-utilities';
import { SubscriptionClient } from 'subscriptions-transport-ws';
import * as ws from 'ws';
import { OperationTypeNode } from 'graphql';
interface IDefinitionsParams {
operation?: OperationTypeNode,
kind: 'OperationDefinition' | 'FragmentDefinition'
}
const wsurl = 'ws://graphql-engine:8080/v1alpha1/graphql';
const getWsClient = function (wsurl: string) {
const client = new SubscriptionClient(wsurl, {
reconnect: true,
lazy: true
}, ws);
return client;
};
const wsLink = new WebSocketLink(getWsClient(wsurl));
const createRemoteSchema = async () => {
const httpLink = new HttpLink({
uri: 'http://graphql-engine:8080/v1alpha1/graphql',
fetch: (nodeFetch as any)
});
const link = split(
({ query }) => {
const { kind, operation }: IDefinitionsParams = getMainDefinition(query);
console.log('kind = ', kind, 'operation = ', operation);
return kind === 'OperationDefinition' && operation === 'subscription';
},
wsLink,
httpLink,
);
const remoteSchema = await introspectSchema(link);
const remoteExecutableSchema = makeRemoteExecutableSchema({
link,
schema: remoteSchema
});
const renamedSchema = transformSchema(
remoteExecutableSchema,
[
new FilterRootFields((operation, fieldName) => {
return (operation === 'Mutation') ? false : true; // && fieldName === 'password'
})
]
);
return renamedSchema;
};
export const createNewSchema = async () => {
const hasuraExecutableSchema = await createRemoteSchema();
const apolloSchema = makeExecutableSchema({
typeDefs,
resolvers,
directiveResolvers: {
hasRole: hasRoleResolver
}
});
return mergeSchemas({
schemas: [
hasuraExecutableSchema,
apolloSchema
]
});
};
Fixed by installing graphql-tools 4th version. It tutns out the editor did not even notice that I do not have this dependency and simply took the version of node_modules, which was installed by some other package. Problem was with version 3.x. Pull request is where the bug was fixed.
I had the same problem, different cause and solution.
My subscription was working well, until I introduced the 'resolve' key in
my subscription resolver:
Here is the 'Subscription' part of My resolver:
Subscription: {
mySubName: {
resolve: (payload) => {
console.log('In mySubName resolver, payload:',payload)
return payload;
},
subscribe:() => pubSub.asyncIterator(['requestsIncomplete']),
// )
},
The console.log proved the resolve() function was being called with a well structured payload (shaped the same as my Schema definiton - specifically the an object with a key named after the graphQL Subscriber, pointing to an array (array is an iterable):
In mySubName resolver, payload: { mySubName:
[ { id: 41,
...,
},
{...},
{...}
...
...
]
Even though I was returning that same unadulterated object, it caused the error expected Iterable, but did not find one for field "Subscription.mySubName"
When I commented out that resolve function all together, the subscription worked, which is further evidence that my payload was well structured, with the right key pointing to an iterable.
I must be mis-using the resolve field. From https://www.apollographql.com/docs/graphql-subscriptions/subscriptions-to-schema/
When using subscribe field, it's also possible to manipulate the event
payload before running it through the GraphQL execution engine.
Add resolve method near your subscribe and change the payload as you wish
so I am not sure how to properly use that function, specifically don't know what shape object to return from it, but using it as above breaks the subscription in the same manner you describe in your question.
I was already using graphql-tools 4.0.0, I upgraded to 4.0.8 but it made no difference.

How to use connection as standalone object with types?

Not working code just to illustrate what I'm trying to achieve
Some connection file
import { ConnectionManager } from 'typeorm';
const c = new ConnectionManager();
// user ormconfig.conf file
export const connection = c.createAndConnect();
using in some model
#Entity()
#Table("annual_incomes")
export class AnnualIncome
{
#PrimaryGeneratedColumn()
id: number;
#Column({ length: 75 })
variant: string;
#Column("int")
sort: number;
#Column()
is_active: boolean;
}
Later somewhere in the code, I want to get connection with all methods, something like:
import { connection } from 'someconnection';
import { AnnualIncome } from 'entities';
// some code here
api.get('/incomes', async(ctx) => {
ctx.body = await connection.getRepository(AnnualIncome).find();
});
Usually, I'm getting an error from tsc that .getRepository() method was not found in connection. However if I do something like:
import { connection } from 'someconnection';
import { AnnualIncome } from 'entities';
// some code here
api.get('/incomes', async(ctx) => {
ctx.body = await connection.then(async connection => {
return await connection.getRepository(AnnualIncome).find();
}
});
the above code works with definitions and tsc does not complain about not-existing methods.
I'd like to avoid an extra definition connection.then() and get plain connection with all methods defined in <Connection> type.
just use createConnection method to create your connection when you bootstrap your application. Later you can access your connection from anywhere using getConnection() method:
import { AnnualIncome } from 'entities';
import { createConnection, getConnection } from 'typeorm';
// somewhere in your app, better where you bootstrap express and other things
createConnection(); // read config from ormconfig.json or pass them here
// some code here
api.get('/incomes', async(ctx) => {
ctx.body = await getConnection().getRepository(AnnualIncome).find();
});
Also you can simply use getRepository method also avalible from anywhere:
import { AnnualIncome } from 'entities';
import { getRepository } from 'typeorm';
// some code here
api.get('/incomes', async (ctx) => {
ctx.body = await getRepository(AnnualIncome).find();
});

Resources