I have a script called scheduler() which runs some queries on a Postgres database and then finished. I am having a problem where the script hangs in the terminal and does not exit after.
Here is my code:
scheduler.js
const {pool} = require("./db");
const scheduler = async function() {
try {
await auctionCheck();
return pool.end().then(() => {
console.log('Pool\'s closed');
return;
})
} catch (e) {
console.error(e)
return;
}
}
return scheduler();
db.js
const {Pool} = require("pg");
const pool = new Pool({
connectionString: process.env.DATABASE_URL,
});
pool.on('error', (err, client) => {
console.error('Unexpected error on idle client', err)
process.exit(-1)
})
pool.connect();
module.exports = {pool};
When I didn't have the pool.end() call, my script didn't exit, but it does not exit with the end() call either. The "Pool's closed" log is never printed. node-pg documentation says that I should do this to close connections (I assume that the reason my script isn't finishing is due to an open db connection.) I want to basically run some functions periodically and then have the script finish, but currently it just stays live. What am I doing wrong?
It seems that the reason was that in db.js I had pool.connect(). Thus, when I did pool.end(), it couldn't close the connection because it did not have all of the clients in the pool - one being already created in db.js but never released. When I removed pool.connect() in db.js the process exited as it should.
Related
I am new to Node.js and I have the following code below:
const mysql = require('mysql');
async function insertNewRecord(record) {
const connectionUrl = 'mysql://root:mypassword#mydatabasehost:3306/mydb';
const pool = mysql.createPool(connectionUrl);
pool.on('connection', function (conn) {
console.log('*******on connection***********');
});
pool.on('error', function () {
console.log('****POOL CONNECTION ERROR***');
});
await pool.query('INSERT INTO users SET ?', record);
}
async function doProcess() {
try {
//construct record object
insertNewRecord(record)
} catch (error) {
console.log(error);
}
}
I purposely used the wrong database host name to see if the error would be caught and logged, but I saw no log. I was expecting to see POOL CONNECTION ERROR in the logs but it didn't show up. No error was caught either from the catch statement.
Am I doing something wrong? I did my own research and the pool.on lines you see in the codes is what I have tried to add, but doesn't seem to work. Am I doing error handling the correct way?
insertNewRecord(record) is an asynchronous function. If you don't use await with it node won't wait for the function to complete, and you won't see the error.
Schema operations in routes proceed without a hitch. But when I reuse the same code (just a find() call for example) and execute it in a file ('node test.js'), I get - "MongooseError: Operation refreshtokens.findOne() buffering timed out after 10000ms."
So I tried checking mongoose.connection.readyState and sure enough it equals 0 (disconnected). Why? When again, mongoose.connection is never touched in my routes and I've never had any issues there (it's only initialized in server.js).
Ultimately I'm trying to create an executable that I'll schedule, and it will need to access the database.
Nothing unusual about the code (I don't think ;) -
server.js
const express = require('express')
const app = express()
const mongoose = require('mongoose')
mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
const db = mongoose.connection
db.on('error', (error) => console.error(error))
db.once('open', () => console.log('Connected to Database'))
...
./models/refreshtoken.js
const mongoose = require('mongoose')
const refreshTokenSchema = new mongoose.Schema({
token: {
type: String,
required: true
}
})
module.exports = mongoose.model('RefreshToken', refreshTokenSchema)
Test executable (test.js)
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
(async () => {
await testGetToken();
console.log("ran testGetToken()");
})().catch(e => {
// Deal with the fact the chain failed
console.log(e)
});
async function testGetToken() {
try {
console.log("here in testGetToken() call")
console.log("pre if mongoose.connection = " + mongoose.connection.readyState);
if (mongoose.connection.readyState != 1) {
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
console.log("post if mongoose.connection = " + mongoose.connection.readyState);
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
The executable will get and print the token after I added in the if statement with await mongoose.connect but then the process doesn't close out (I have to ctrl-c twice). Why? (And as noted at top, without the await mongoose.connect, I get the MongooseError.)
Best thing to know is why does mongoose have a mind of its own and is disconnecting only in files I execute (and how to correct it). Added mongoose.connection.on('disconnected'...) event code in server.js too and it never trips.
Next best thing would be if we didn't expect any deleterious effects in proceeding as such, how to get the executable's process to close out. I can do more homework here but I tend to think it's best to resolve problems rather than ignore them (see: Best thing)? :)
Appreciate any help you can offer. Thank you.
It either has an error you aren’t catching or can’t find one. I’ve had the same problem and in my case I was misusing the findOne() an it wasn’t matching up. Try using a mongo GUI and console.log() for you results.
I think I figured this out. In a one-off script, you need to connect and close a separate mongoose connection (with mongoose.connect and mongoose.connection.close).
I thought it would leverage my existing mongoose connection, especially when I had put the test.js code in (and was executing) my route file, but even there not the case. It must be that a heroku scripting process (or what would this be called?) is separate from a web process.
I also thought if I had mongoose.connection.close() in my script, again thinking they were one in the same, then it would take that connection down and schema calls in my route would fail. Also not the case.
Updated test.js for your reference -
const RefreshToken = require('./models/refreshtoken.js');
const mongoose = require('mongoose');
// ALWAYS executes (think of this like your main)
(async () => {
await mongooseConnect()
await testGetToken();
mongoose.connection.close()
})().catch(e => {
console.log(e)
});
async function mongooseConnect() {
// This is what I was missing before. I thought it would leverage my existing mongoose connection from server.js
// but it most certainly would not when running a script
await mongoose.connect(process.env.MONGODB_URI, {useNewUrlParser: true, useUnifiedTopology: true})
}
async function testGetToken() {
// gets token from database and prints
try {
console.log("here in testGetToken() call")
const token = await RefreshToken.find()
console.log(token)
} catch (err) {
console.log("testGetToken() err = " + err)
}
}
// event code that fires when mongoose is connected
mongoose.connection.on('connected', () => {
console.log('Test script connected to database');
console.log(mongoose.connection.readyState); //logs 1
});
// event code that fires when mongoose is disconnected
mongoose.connection.on('disconnected', () => {
console.log('Test script disconnected from database');
console.log(mongoose.connection.readyState); //logs 0
});
Output when I run from my command line 'heroku run node test.js' -
Test script connected to database
1
here in testGetToken() call
[
{
_id: 61355917fedd0f00166e4e08,
token: <mytokenhere>,
__v: 0
}
]
Test script disconnected from database
0
Final key point would be I also have the same "mongoose.connection.on('disconnected'..." code in my server.js with a different log message and it doesn't show up in the server logs after running this script (that was one of my fears, that the connection.close in script would take my server mongoose connection down; it doesn't).
Sometimes it also as simple as checking if your current ip is whitelisted or not.
I can't figure out why this app keeps running. I've tried using the why-is-node-running package but I'm not perfectly sure how to read the output properly. Here's the first output of it:
There are 30 handle(s) keeping the process running
# TCPWRAP
/node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));
/node_modules/mongodb/lib/core/connection/connect.js:29 - makeConnection(family, options, cancellationToken, (err, socket) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:182 - connect(monitor.connectOptions, monitor[kCancellationToken], (err, conn) => {
/node_modules/mongodb/lib/core/sdam/monitor.js:206 - checkServer(monitor, e0 => {
/node_modules/mongodb/lib/core/sdam/monitor.js:92 - monitorServer(this);
My guess is it has something to do with MongoDB not closing properly. Although, when I removed all of the other functions between opening the client and closing it, it opened and closed perfectly.
Adding process.exit() at the end closes program properly, but I'd like to figure out why it isn't closing.
A summary of the app is that it is getting data from MongoDB, cleaning it, and then writing it into Firestore - so a lot of async actions going on, but I didn't see Firestore-related stuff pop up in the why-is-node-running logs.
const GrabStuffFromDBToCalculate = require("./helpers/GrabStuffFromDBToCalculate");
const SendToFirestore = require("./helpers/SendToFirestore");
const log = require("why-is-node-running");
const { MongoClient } = require("mongodb");
require("dotenv").config();
const main = async () => {
try {
const client = await MongoClient.connect(process.env.MONGODB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
const collection = await client.db("test").collection("testcollection");
const trip_object = await GrabStuffFromDBToCalculate(collection);
SendToFirestore(trip_object);
client.close();
log(); // "There are 30 handle(s) keeping the process running including node_modules/mongodb/lib/core/connection/connect.js:269 - socket = tls.connect(parseSslOptions(family, options));"
// process.exit() // this closes everything but I'd rather not have to use this
} catch (err) {
console.log(err);
client.close();
}
};
const runAsync = async () => {
await main(); // this exists because I'm usually running multiple main() functions
};
runAsync();
SendToFirestore code:
const firebase = require("firebase");
const firebaseConfig = require("../config");
module.exports = SendToFirestore = trip_object => {
if (!firebase.apps.length) {
firebase.initializeApp(firebaseConfig);
}
const db = firebase.firestore();
db.doc(`hello/${object._id}`).set({
objectid:object._id
});
};
GrabStuffFromDBToCalculate code (way simplified):
module.exports = GrabStuffFromDBToCalculate = async collection => {
const cursor = await collection
.aggregate([
// does a bunch of stuff here
])
.toArray();
const newObj = cursor[0];
return newObj;
};
Making my comment into an answer since it led to the missing piece.
Node does not shut down because you have an open Firestore connection. You will have to call terminate to allow the SDK to shut down and release resources:
db.terminate();
Which is relevant for allowing nodejs to shut itself down automatically.
Also, I'm not sure you understood that I was suggesting that you use await as in
await client.close()
before calling log() so you are sure that the client connection has been closed before you do the logging. client.close() is an asynchronous method so your original code would log() before that close was complete.
I am currently working on Node.js and firebase and I am currently stuck in an issue, where after writing the data in database my program does not end on a terminal, is there a way I can end this.
I guess, maybe I need to end the firebase connection but I don't have an idea how would I do that.
Here is the program,
module.exports=
{
create_firebase_entry: function ()
{
console.log("Firebase Entry")
return new Promise((resolve, reject) => {
var secondaryAppConfig =
{
credential: admin.credential.cert(serviceAccount),
databaseURL: "firebase_databse_url"
};
if (admin.apps.length===0){secondary = admin.initializeApp(secondaryAppConfig, "secondary");}
// Retrieve the database.
var db = secondary.database();
var ref = db.ref("data_records");
var usersRef = ref.child("My_Data");
usersRef.set({Data1:"Trail Data"})
})
}
}
The script opens a connection to the Firebase Realtime Database, and then performs asynchronous read and write operations over that connection. Since the script has no way to know when you're done, you will have to signal this yourself.
The most common way to do this (that I know of) is to call process.exit() when you're done. E.g.
usersRef.set({Data1:"Trail Data"})
.then(function() {
process.exit(0);
});
Also see
How to properly exit firebase-admin nodejs script when all transaction is completed
Exit from a NodeJS script when all asynchronous tasks are done
node process doesn't exit after firebase once
I agree to Frank but I think using process.exit(0); under .finally() block will provide more guarantee. Because when error occur It will never close.
usersRef.set({Data1:"Trail Data"})
.then(result => {
// ...
})
.catch(err => {
// ...
})
.finally(() => {
process.exit(0)
})
I have A Mongoose Connection and at some point in my program I need to close it.
after logging the mongoose object several times, I have found that the following workd
mongoose.connection.base.connections[1].close();
Is There A Cleaner way to do this?
To close all connections in the Mongoose connection pool:
mongoose.disconnect();
Docs here.
I used a singleton in my code.
Exporting a promise
In one file I export a promise...
connection.js
const { createConnection } = require('mongoose');
function getConnection() {
return mongoose.createConnection(...).asPromise();
}
const connPromise = getConnection();
module.exports = connPromise;
other files...
const connPromise = require('./connection.js');
async function handler() {
// Connection starts on first module import
// This gets faster when the connection is done being created
const connection = await connPromise;
// Do stuff with the connection
return {
data: '',
and: '',
stuff: '',
};
}
When everything is shutting down, like in a lambda, also in connection.js:
process.on('SIGTERM', async () => {
console.info('[runtime] SIGTERM received');
const conn = await connPromise;
await conn.close();
console.info('[runtime] closed connection');
process.exit(0);
});
So by storing the connection in a promise, I can get it everywhere. I don't export the connection itself because getting it is async.
Exporting a getter function
I could have a function like
let connection = null;
function getConnection() {
if (!connection) {
connection = await createConnection(...).asPromise();
}
return connection;
}
module.exports = getConnection;
other files...
const getConnection = require('./connection.js');
async function handler() {
// Gets faster after the first call.
// connection created on first call.
const connection = await getConnection();
// Do stuff with the connection
return {
data: '',
and: '',
stuff: '',
};
}
Then your shutdown would become
process.on('SIGTERM', async () => {
console.info('[runtime] SIGTERM received');
if (connection) {
await connection.close();
console.info('[runtime] closed connection');
}
process.exit(0);
});
Summary
Overall, I like exporting the promise, since in every lambda function, I'm always using the database.
Since the exported promise means that the connection starts as soon as the module is imported, it's ready to go sooner than if I waited till the invocation of the getConnection() in an event handler.
For having only one connection, this is a good way to get ahold of it.