TypeORM create connection globally - node.js

I am using typeorm with typescript in my node Js application. I am trying to figure out the way of using the single DB connection for all functions in the class. For example, I have two functions my class and want to use the global/single connection for all functions instead of creating a connection in every function as shown below:
export class SQLDBService implements IDatabaseService{
private readonly logger = getLogger("SQLDBService");
private connection:Connection;
getConversation(conversationId: string): ConversationEntity {
let conversationEntity = new ConversationEntity();
createConnection(/*...*/).then(async connection => {
let dbObj = await connection.getRepository(ConversationEntity).findOne({
conversationId: Equal(conversationId)
});
if(dbObj)
conversationEntity = dbObj;
});
return conversationEntity;
}
pushWrapUp(conversationId: string, wrapUp: string): void {
createConnection().then(async connection => {
let conversationEntity = await connection.getRepository(ConversationEntity).findOne({
conversationId: Equal(conversationId)
});
if(conversationEntity){
conversationEntity.wrapUp = wrapUp;
conversationEntity.endTime = new Date();
await connection.manager.save(conversationEntity);
}
});
}}
Can someone point me in the right direction?

The code above doesn't efficiently use async..await because promises aren't chained, this results in poor error handling and improper control flow.
As the documentation explains,
TypeORM's Connection does not setup a database connection as it might seem, instead it setups a connection pool. <...> Connection pool setup is established once connect method of the Connection is called. connect method is called automatically if you setup your connection using createConnection function. Disconnection (closing all connections in the pool) is made when close is called. Generally, you must create connection only once in your application bootstrap, and close it after you completely finished working with the database.
createConnection is supposed to be called only once on application initialization. Since it's asynchronous, initialization routine should wait for it before using TypeORM models.
As the documentation suggests, getConnection() can be used instead of createConnection. Since the purpose is to get a repository for default connection, getRepository can be used instead:
It's:
import {getRepository} from "typeorm";
...
async getConversation(conversationId: string): ConversationEntity {
let conversationEntity = new ConversationEntity();
let dbObj = getRepository(ConversationEntity).findOne({
conversationId: Equal(conversationId)
});
if(dbObj) conversationEntity = dbObj;
return conversationEntity;
}

You should use a global connection pool which will create, hold, and take care the used connections for you. I am not familiar with node.js, so I cannot give out a name of this kind 3rd party library. But there must be some, since the connection pool is a widely accepted design pattern.

this quite minimal refactoring should do the trick
export class SQLDBService implements IDatabaseService {
private readonly logger = getLogger("SQLDBService");
private connection:Connection;
init() {
this.connection = await createConnection(/*...*/)
}
getConversation(conversationId: string): ConversationEntity {
let conversationEntity = new ConversationEntity();
let dbObj = await this.connection.getRepository(ConversationEntity).findOne({
conversationId: Equal(conversationId)
});
if(dbObj)
conversationEntity = dbObj;
return conversationEntity;
}
pushWrapUp(conversationId: string, wrapUp: string): void {
let conversationEntity = await this.connection.getRepository(ConversationEntity).findOne({
conversationId: Equal(conversationId)
});
if(conversationEntity){
conversationEntity.wrapUp = wrapUp;
conversationEntity.endTime = new Date();
await this.connection.manager.save(conversationEntity);
}
}
}
const db = new SQLDBService()
try {
await db.init()
}
catch (error) {
console.error("db connection error")
console.error(error)
console.error("db connection error")
}

Related

How to do chained node-redis transactions with singleton wrapper functions

I have a Typescript-backed Express.js project that uses a singleton Redis client.
The singleton includes wrapper functions to Redis commands needed for my application (e.g., SADD).
Singleton
Here is a snippet of my singleton Redis client service, which is relevant to my question:
/**
* Set up a singleton class instance to interface
* with the Redis database, along with helper async
* functions that provide functionality.
*/
var redis = require("redis");
// https://github.com/redis/node-redis/issues/1673
type RedisClientType = ReturnType<typeof redis.createClient>;
type RedisClientOptionsType = Parameters<typeof redis.createClient>[0];
export class Redis {
private static instance: Redis;
private static client: RedisClientType;
constructor() {
if (Redis.instance)
return Redis.instance;
Redis.instance = this;
Redis.client = null;
}
/* ... */
async initializeClient(options: RedisClientOptionsType) {
Redis.client = redis.createClient(options);
Redis.client.on('connect', function() {
Redis.instance.log('Client connected');
});
Redis.client.on('error', function(err: Error) {
Redis.instance.log(`Could not communicate with Redis client [${err}]`);
});
await Redis.client.connect();
}
async shutdownClient() {
await Redis.client.quit();
}
async multi() {
await Redis.client.multi();
}
async exec() {
await Redis.client.exec();
}
/* ... */
async sAdd(k: string, v: string) {
return await Redis.client.sAdd(k, v);
}
}
Individual calls to sAdd, sMembers, etc. work fine. So the client itself is initialized correctly, and it is able to process basic Redis calls.
What I would like to do is perform some chained transactions, e.g., from a rudimentary POST request using the singleton Redis client service, process some data from an uploaded file, and then add some key-value pairs (to start):
import { Redis } from '#/service/redis';
/* ... */
export const myPost = async (req: Request, res: Response) => {
const redis = new Redis(); // initialized client, as defined above
const k = 'my-key';
const v = 'my-value';
await redis
.multi()
.sAdd(k, v)
.exec();
}
Problem
The problem is that I get two errors with the chained await ... call.
First error
The first error is related to the await keyword, just before the multi/sAdd/exec call:
'await' expressions are only allowed within async functions and at the top levels of modules.ts(1308)
This await is within the async-ed post function. (I assume that I need to this handle the results of the underlying Promise chain.)
Second error
The second error is related to the sAdd wrapper:
Property 'sAdd' does not exist on type 'Promise<void>'.ts(2339)
I tried to add a return type to the call to multi:
async multi(): Promise<RedisClientType> {
await Redis.client.multi();
}
But this did not resolve the error with sAdd.
Question
What changes do I make to the service/singleton, which would allow calls to wrapper functions to be chained?

amqplib sendToQueue() is not sending messages to RabbitMQ

I am building a Typescript package around amqplib's Promise API to make it simpler for me to send messages between two RabbitMQ queues.
This is a method inside a class of said package that is responsible for sending a message to RabbitMQ. It's not actually sending messages, because I can't see them from CloudAMQP.
class PostOffice {
connection: any;
constructor(connection: any) {
this.connection = connection;
}
//...
// TMQMessage is an object that can be trivially serialized to JSON
// The Writer class holds the channel because I'm having trouble
// making Typescript class member updates stick after a method returns.
async send(writer: Writer, message: TMQMessage) {
//...RabbitMQ code begins here
let channel = writer.stageConsume[0]; // This channel is created elsewhere
// Queue is created here and I can see it in CloudAMQP.
await channel.assertQueue(message.to + "/stage/1", {
durable: true,
}); // `message.to` is a string indicating destination, e.g. 'test/hello'
// According to docs, the content should be converted to Buffer
await channel.sendToQueue(message.to + "/stage/1", Buffer.from(JSON.stringify(message)));
channel.close()
}
}
This is a class I made that creates connections and channels, which is being used here:
import { Writer } from "./Writer";
export class ConnectionFactory {
url: string;
amqp: any;
constructor(url: string) {
this.url = url;
this.amqp = require('amqplib');
}
async connect(timeout: number = 2000) {
let connection = await this.amqp.connect(this.url, {
// timeout for a message acknowledge.
timeout, // Make the timeout 2s per now
})
return connection;
}
async createChannels(connection: any, writer: Writer) {
//... relevant code starts here
let stageChannels = [];
stageChannels.push(await connection.createChannel())
writer.stageConsume = stageChannels;
return writer;
}
}
}
My connection factory seems to be working properly, because I can see the connections from CloudAMQP's dashboard. Also I can see the Queues that have been created (asserted in my code) from the dashboard too. However, I am unable to get amqplib to send a message out to CloudAMQP.
Here's the (async) code I'm using to call my package:
let Cf = new ConnectionFactory(url)
let connection = await Cf.connect()
let writer = new Writer();
let message: TMQMessage = {
//... message content in JSON
}
writer = await Cf.createChannels(connection, writer)
let po = new PostOffice(connection)
await po.send(writer, message)
What seems to be wrong?
Probably want to await the close method too. But in general I would recommend amqp-client.js (that also have TypeScript definitions), https://github.com/cloudamqp/amqp-client.js/

Mongoose too many connection and commands

I'm here to request help with mongo/mongoose. I use AWS lambda that accesses a mongo database and I'm having problems sometimes my connections reach the limit of 500. I'm trying to fix this problem and I did some things like this https://dzone.com/articles/how-to-use-mongodb-connection-pooling-on-aws-lambd and https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs. That basically is to use a singleton-like and set context.callbackWaitsForEmptyEventLoop = false, which indeed helped but is still, rarely, open 100 connections in less than a minute, it looks like there is some connection that is not being reused even tho our logs show that they are being reused. So I realized a weird behavior, whenever mongoatlas shows me an increased number of commands, my mongo connections increase heavily. The first chart is operations and the second is the connections.
Looking at operations, there are too many commands and just a few queries. I have no idea what are those commands, my theory is that those commands are causing the problem but I did not find anything that explained what is the difference between query and command exactly for me to know if that is a valid theory or not. Another thing is, how to choose correctly the number of pool size, we have really simple queries.
Here is our singleton class because maybe this is what we are doing wrong:
class Database {
options: [string, mongoose.ConnectionOptions];
instance?: typeof mongoose | null;
constructor(options = config) {
console.log('[DatabaseService] Created database instance...');
this.options = options;
this.instance = null;
}
async checkConnection() {
try {
if (this.instance) {
const pingResponse = await this.instance.connection.db.admin().ping();
console.log(`[DatabaseService] Connection status: ${pingResponse.ok}`);
return pingResponse.ok === 1;
}
return false;
} catch (error) {
console.log(error);
return false;
}
}
async init() {
const connectionActive = await this.checkConnection();
if (connectionActive) {
console.log(`[DatabaseService] Already connected, returning instance`);
return this.instance;
}
console.log('[DatabaseService] Previous connection was not active, creating new connection...');
this.instance = await mongoose.connect(...this.options);
const timeId = Date.now();
console.log(`Connection opened ${timeId}`);
console.time(`Connection started at ${timeId}`);
this.instance?.connection.on('close', () => {
console.timeEnd(`Connection started at ${timeId}`);
console.log(`Closing connection ${timeId}`);
});
return this.instance;
}
async getData(id: string) {
await this.init();
const response = await Model.findOne({ 'uuid': id });
return response;
}
}
I hope that is enough information. My main question is if my theory of commands causing too many connections is possible and what are exactly commands because every explanation that I found look like is the same than query.
Based on the comment written by Matt I have changed my init function and now my connections are under control.
async init() {
if (this.instance) {
console.log(`[DatabaseService] Already connected, returning instance`);
return this.instance;
}
console.log('[DatabaseService] Previous connection was not active, creating new connection...');
this.instance = await mongoose.connect(...this.options);
const timeId = Date.now();
console.log(`Connection opened ${timeId}`);
console.time(`Connection started at ${timeId}`);
this.instance?.connection.on('close', () => {
console.timeEnd(`Connection started at ${timeId}`);
console.log(`Closing connection ${timeId}`);
});
return this.instance;
}

what is the best way or pattern to create a Wrapper class to abstract the use of RabbitMQ in node js

so I'm using RabbitMQ for some Projects and i noticed that i ll use some duplicate code all the Time that's why i decided to make a Wrapper Class or Interface that have some function to use RabbitMQ direct without repeating the code all the time. i began to do this yesterday and i already had some Problems since i wanted to use OOP and Javascript can be complicated when using OOP (at least i think so)
I began with creating a class IRAbbitMQ with function init to initialize a connection and create a channel, i knew that i cant use nested classes so instead i wanted to use Factory functions, i tried to make the connection and channel a part of the class IRabbitMQ properties but i dont know why that gave me undefined when i create an instance of it
class IRabbitMQ {
constructor() {
this.init(rabbitMQServer); // rabbitMQServer for example 'localhost//5672'
}
// establish a Connection to RAbbitMQ Server
async init(host) {
try {
let connection = await amqplib.connect(host);
let channel = await connection.createChannel();
channel.prefetch(1);
console.log(' [x] Awaiting RPC requests');
this.connection = connection;
this.channel = channel;
}
catch(err) {
console.error(err);
}
}
// Close the Connection with RabbitMQ
closeConnection() {
this.connection.close();
}
log() {
console.log(this.connection);
}
EventPublisher() {
function init(IRabbit, publisherName) {
if(!IRabbit.connection) {
throw new Error('Create an Instance of IRabbitMQ to establish a Connection');
}
let ch = IRabbit.channel;
console.log(ch);
}
return {
init : init
}
}
}
var r = new IRabbitMQ();
r.log();
when i run the code the output is undefined, i dont know why since i m initializing the connection and channel properties in the init function and then called that function in the constructor so that should be initialized when i create an object of the Wrapper class. i wanted also to take some advices from you wether it is good to use classes or is there any other better way to create a Wrapper class or Interface for RabbitMQ to make it easy to use it and not have to duplicate Code.
Not really an answer, but I was able to successfully log the connection with this example code. I trimmed out other code to just focus on the .log() part that was logging a undefined.
Code is far from perfect, but works at least
const amqplib = require('amqplib');
class IRabbitMQ {
constructor() { }
async init(host) {
try {
const connection = await amqplib.connect(host);
const channel = await connection.createChannel();
channel.prefetch(1);
console.log(' [x] Awaiting RPC requests');
this.connection = connection;
this.channel = channel;
}catch(err) {
console.error(err);
}
}
log() {
console.log(this.connection);
}
}
async function createInstance(){
const instance = new IRabbitMQ();
try {
await instance.init('amqp://localhost');
}catch (e) {
throw new Error('OOPS!');
}
return instance;
}
async function runLogic() {
const r = await createInstance();
r.log();
}
runLogic().catch(console.log);
Just comment if you'd want me to give additional advice/tips, but this seems to work for me.

Scope in NodeJS / Firebase promise

I'm trying to develop a NodeJS app connecting to Firebase. I can connect successfully, but I'm unable to figure how to manage the scope in the then call.
I'm using NodeJS 6.9.2
My test implementation looks like this:
const EventEmitter = require('events');
const fb = require('firebase')
class FireGateway extends EventEmitter {
constructor() {
super();
if ( this.instance ) {
return this.instance;
}
// INIT
var fbConfig = {
apiKey: "xxxxx",
authDomain: "xxxxx.firebaseapp.com",
databaseURL: "https://xxxxx.firebaseio.com/"
};
fb.initializeApp(fbConfig)
this.instance = this;
this.testvar = "aaa";
}
login() {
fb.auth().signInWithEmailAndPassword ("email", "pwd")
.catch(function(error) {
// Handle Errors here.
}).then( function(onresolve, onreject) {
if (onresolve) {
console.log(this.testvar);
// "Cannot read property 'testvar' of undefined"
this.emit('loggedin');
// error as well
}
})
}
}
module.exports = FireGateway;
------
...
var FireGateway = require('./app/fireGateway');
this.fireGW = new FireGateway();
this.fireGW.login();
....
Any idea how can I manage it?
The callback passed to then is being called asynchronously from another context, so the this doesn't correspond to the instantiated object.
Using ES6 arrow functions you can keep your object context, since an arrow function does not create its own this context.
By the way, the syntax you are using in the then method is not correct, then accepts two callbacks with one argument each one. Check the syntax here.
The catch before the then is not necessary as well I think, it would make more sense to put it at the end.
It would be something like this:
login() {
fb.auth().signInWithEmailAndPassword("email", "pwd")
.then(
(onResolve) => {
console.log(this.testvar);
this.emit('loggedin');
},
(onReject) = > {
// error handling goes here
});
}
On the other hand, it seems login method is doing an asynchronous operation, so you might want to wait for it to finish in your code. I would make the login method return a Promise, so you can wait for it outside:
login() {
return fb.auth().signInWithEmailAndPassword("email", "pwd")
...
}

Resources