I am using node.js npm module mysql for connection with a MySQL database.
I see there is a possibility to use:
either pool.getConnection(), connection.query() and connection.release()
or pool.query
I have two questions regarding the second scenario.
1) Is there any automatic reconnect in case of error, lost connection while using pool.query? (is 'handle disconnect' function needed while using pool?
2) Using the first approach I can set connection.on('error', function(){...}). How to do this in the second case? (pool.on('error'...) ?)
1) Correct me if I'm wrong, but I don't think there's a method for auto connection upon disconnect while using pool query. It's a good code design to let us know the error, and let us decide what to do upon disconnect.
2) We can use pool.getConnection((err, con) => {})
Here's the code I'm using to check for connection before running a query. Hope it helps.
connect: function ()
{
return new Promise((resolve, reject) => {
let pool = Mysql.createPool({
connectionLimit: config.mysql.connectionLimit,
host: config.mysql.host,
user: config.mysql.user,
password: config.mysql.password,
database: config.mysql.database
});
pool.getConnection((err, con) =>
{
try
{
if (con)
{
con.release();
resolve({"status":"success", "data":"MySQL connected.", "con":pool});
}
}
catch (err)
{
reject({"status":"failed", "error":`MySQL error. ${err}`});
}
resolve({"status":"failed", "error":"Error connecting to MySQL."});
});
});
}
Related
So i have this small Node JS app where i have the following script, which i invoke in my HTML index page, in order to connect to a Cloud SQL database in GCP and perform a specific query so i can pass the values to a dropdown later:
try {
pool = new Pool({
user: "postgres",
host: "/cloudsql/sfmcsms-d-970229:europe-west1:dsi-sfmc-sms-database",
database: "postgres",
password: "dsi-sfmc-sms-database",
port: "5432",
});
console.log("Connection successfull!");
pool.query("select * from ConfigParameter;", (error, results) => {
if (error) {
console.log(error);
}
qResult = results;
console.log(qResult);
//insert logic to populate dropdowns
});
} catch (err) {
console.log("Failed to start pool", err);
}
I'm still working on the logic to populate the dropdowns but for now, i'm focusing on establishing a successful connection first before i get to that. However, everytime i run the script, i seem to get this particular error:
ReferenceError: Pool is not defined
I've been looking around for some possible answers but no luck.
Before using Pool you have import first like this
const { Pool } = require('pg')
And obviously node-postgres should be installed
npm i pg
When I start my server, an error was shown like this:
Error in DB connection : Error: queryTxt ETIMEOUT yky-api-ajgvf.mongodb.net
I tried many times to work properly in order to fix this problem. I couldn't even deploy my project on heroku properly yet. I love nodejs so much but now I just want to give up continuously learning Nodejs for now.
Someone help me to solve my problem if possible. I'm the one who is learning Node.Js. It's been around 4 or 5 months. Before I decided to leran PHP, but now I'm still trying to learn nodejs.
I'm a newbie. :)
This is my config database code in my small project.
const mongoose = require('mongoose')
const log = console.log
mongoose.connect('mongodb+srv://chanlay:chanlay123#yky-api-ajgvf.mongodb.net/yky-blog-api?retryWrites=true&w=majority', {
useNewUrlParser: true
}, (err) => {
if (!err) {
log('MongoDB Connection Succeeded.')
} else {
log('Error in DB connection : ' + err)
}
});
require('../models/posts')
mongoose.connect() can look like this (docs reference):
mongoose.connect(uri, options, function(error) {
// Check error in initial connection. There is no 2nd param to the callback.
});
// Or using promises
mongoose.connect(uri, options).then(
() => { /** ready to use. The `mongoose.connect()` promise resolves to mongoose instance. */ },
err => { /** handle initial connection error */ }
);
Before demonstrating further, first of all, this block of yours is wrong:
if (!err) {
log('MongoDB Connection Succeeded.')
}
error handling doesn't work that way, this condition will never happen, as when there's no error, there is no reason to call error callback function which checks this condition.
Instead use .then() block, as a successful connection returns a Promise:
mongoose.connect(uri, options).then(
() => { console.log('MongoDB Connection Succeeded.') },
err => { console.log(err) }
);
// OR use .catch() after .then()
mongoose.connect(uri, options).then(() => console.log('MongoDB Connection Succeeded.')).catch( err => console.log(err))
Your answer: set socketTimeoutMS to 60000 or 90000.
If still didn't work, add connectTimeoutMS=90000 and reconnectTries=1000 to options too.
Declare a new variable named options (the name is arbitrary, but better follow the convention), and you put your useNewUrlParser: true also there (reference):
const options = {
useNewUrlParser: true,
socketTimeoutMS: 60000 //or 90000
}
Tip: You can also specify driver options in your connection string as parameters in the query string portion of the URI (reference):
{main-url}?socketTimeoutMS=90000, that is:
const uri = mongodb+srv://chanlay:chanlay123#yky-api-ajgvf.mongodb.net/yky-blog-api?retryWrites=true&w=majority?socketTimeoutMS=90000
but stick with options variable as it's cleaner, more readable, and easier to maintain (otherwise you would put everything in a string which makes it harder for you to read and debug).
Please let me know by leaving to my answer a comment either as a feedback or reporting the persistent error.
I have this database connection. Inside the function where the comment is located, there is a data update cycle for rest api. The data is updated, but when the data in the Oracle database is updated, the connection may fail and after that all subsequent updated data will get undefined. How can you properly connect to the database so that there are no failures?
oracledb.getConnection(
{
user: db.user,
password: db.password,
connectString: db.connectString
},
connExecute
);
function connExecute(err, connection) {
if (err) {
console.error(err.message);
return;
}
sql = `SELECT * FROM db.test`;
connection.execute(sql, {}, { outFormat: oracledb.OBJECT },
function (err, db) {
if (err) {
console.error(err.message);
connRelease(connection);
return;
}
// data update loop
connRelease(connection);
});
}
function connRelease(connection) {
connection.close(
function (err) {
if (err) {
console.error(err.message);
}
});
}
You should be using a connection pool. Connection pools have built-in logic to detect connections with issues and create new connections transparently. See this series on creating a REST API for more details: https://jsao.io/2018/03/creating-a-rest-api-with-node-js-and-oracle-database/
Keep in mind that issues can still happen, so you have to handle errors as needed for your application.
Mostly you add listener on connection object and on dissociation or failure again create connection. With minor changes you can adopt this approach and use listeners to check if connection is available if not connect again. There could be several reason that results in connection closing better handle exceptions, check if still connected and reconnect in case of error.
Or you can try this NPM this will do reconnection for you
https://www.npmjs.com/package/oracledb-autoreconnect
Ping me if you need calcification.
var dbConfig = {
host: '----',
user: '----',
password: '----',
database: '----',
port: ----
};
var connection;
function handleDisconnect() {
connection = <obj>.getConnection(dbConfig);
// Recreate the connection, since the old one cannot be reused.
connection.connect( function onConnect(err) {
// The server is either down
if (err) {
// or restarting (takes a while sometimes).
console.log('error when connecting to db:', err);
setTimeout(handleDisconnect, 10000);
// We introduce a delay before attempting to reconnect,
}
// to avoid a hot loop, and to allow our node script to
});
// process asynchronous requests in the meantime.
// If you're also serving http, display a 503 error.
connection.on('error', function onError(err) {
console.log('db error', err);
if (err.code == 'PROTOCOL_CONNECTION_LOST') {
handleDisconnect();
// lost due to either server restart, or a
} else {
// connnection idle timeout (the wait_timeout
throw err;
// server variable configures this)
}
});
}
handleDisconnect();
My stack is node, express and the pg module. I really try to understand by the documentation and some outdated tutorials. I dont know when and how to disconnect and to end a client.
For some routes I decided to use a pool. This is my code
const pool = new pg.Pool({
user: 'pooluser',host: 'localhost',database: 'mydb',password: 'pooluser',port: 5432});
pool.on('error', (err, client) => {
console.log('error ', err); process.exit(-1);
});
app.get('/', (req, res)=>{
pool.connect()
.then(client => {
return client.query('select ....')
.then(resolved => {
client.release();
console.log(resolved.rows);
})
.catch(e => {
client.release();
console.log('error', e);
})
pool.end();
})
});
In the routes of the CMS, I use client instead of pool that has different db privileges than the pool.
const client = new pg.Client({
user: 'clientuser',host: 'localhost',database: 'mydb',password: 'clientuser',port: 5432});
client.connect();
const signup = (user) => {
return new Promise((resolved, rejeted)=>{
getUser(user.email)
.then(getUserRes => {
if (!getUserRes) {
return resolved(false);
}
client.query('insert into user(username, password) values ($1,$2)',[user.username,user.password])
.then(queryRes => {
client.end();
resolved(true);
})
.catch(queryError => {
client.end();
rejeted('username already used');
});
})
.catch(getUserError => {
return rejeted('error');
});
})
};
const getUser = (username) => {
return new Promise((resolved, rejeted)=>{
client.query('select username from user WHERE username= $1',[username])
.then(res => {
client.end();
if (res.rows.length == 0) {
return resolved(true);
}
resolved(false);
})
.catch(e => {
client.end();
console.error('error ', e);
});
})
}
In this case if I get a username already used and try to re-post with another username, the query of the getUser never starts and the page hangs. If I remove the client.end(); from both functions, it will work.
I am confused, so please advice on how and when to disconnect and to completely end a pool or a client. Any hint or explanation or tutorial will be appreciated.
Thank you
First, from the pg documentation*:
const { Pool } = require('pg')
const pool = new Pool()
// the pool with emit an error on behalf of any idle clients
// it contains if a backend error or network partition happens
pool.on('error', (err, client) => {
console.error('Unexpected error on idle client', err) // your callback here
process.exit(-1)
})
// promise - checkout a client
pool.connect()
.then(client => {
return client.query('SELECT * FROM users WHERE id = $1', [1]) // your query string here
.then(res => {
client.release()
console.log(res.rows[0]) // your callback here
})
.catch(e => {
client.release()
console.log(err.stack) // your callback here
})
})
This code/construct is suficient/made to get your pool working, providing the your thing here things. If you shut down your application, the connection will hang normaly, since the pool is created well, exactly not to hang, even if it does provides a manual way of hanging,
see last section of the article.
Also look at the previous red section which says "You must always return the client..." to accept
the mandatory client.release() instruction
before accesing argument.
you scope/closure client within your callbacks.
Then, from the pg.client documentation*:
Plain text query with a promise
const { Client } = require('pg').Client
const client = new Client()
client.connect()
client.query('SELECT NOW()') // your query string here
.then(result => console.log(result)) // your callback here
.catch(e => console.error(e.stack)) // your callback here
.then(() => client.end())
seems to me the clearest syntax:
you end the client whatever the results.
you access the result before ending the client.
you don´t scope/closure the client within your callbacks
It is this sort of oposition between the two syntaxes that may be confusing at first sight, but there is no magic in there, it is implementation construction syntax.
Focus on your callbacks and queries, not on those constructs, just pick up the most elegant for your eyes and feed it with your code.
*I added the comments // your xxx here for clarity
You shouldn't disconnect the pool on every query, connection pool is supposed to be used to have "hot" connections.
I usually have a global connection on startup and the pool connection close on (if) application stop; you just have to release the connection from pool every time the query ends, as you already do, and use the same pool also in the signup function.
Sometimes I need to preserve connections, I use a wrapper to the query function that checks if the connection is active or not before perform the query, but it's just an optimization.
In case you don't want to manage open/close connections/pool or release, you could try https://github.com/vitaly-t/pg-promise, it manage all that stuff silently and it works well.
The documentation over node-postgres's github says:
pro tip: unless you need to run a transaction (which requires a single client for multiple queries) or you have some other edge case like streaming rows or using a cursor you should almost always just use pool.query. Its easy, it does the right thing ™️, and wont ever forget to return clients back to the pool after the query is done.
So for non-transactional query, calling below code is enough.
var pool = new Pool()
pool.query('select username from user WHERE username= $1',[username], function(err, res) {
console.log(res.rows[0].username)
})
By using pool.query, the library will take care of releasing the client after the query is done.
Its quite simple, a client-connection (single connection) opens up, query with it, once you are done you end it.
The pool concept is different, in the case of mysql : you have to .release() the connection back to the pool once you are done with it, but it seems that with pg is a different story:
From an issue on the github repo : Cannot use a pool after calling end on the pool #1635
"Cannot use a pool after calling end on the pool"
You can't reuse a pool after it has been closed (i.e. after calling
the .end() function). You would need to recreate the pool and discard
the old one.
The simplest way to deal with pooling in a Lambda is to not do it at
all. Have your database interactions create their own connections and
close them when they're done. You can't maintain a pool across
freeze/thaw cycles anyway as the underlying TCP sockets would be
closed.
If opening/closing the connections becomes a performance issue then
look into setting up an external pool like pgbouncer.
So I would say that your best option is to not end the pool, unless you are shutting down the server
I want my application (lets say a simple node file for now) to work as it is even if redis is not available. I'm not able to do it the correct way. This is what I've tried.
var redis = require('redis');
var redisClient = null;
var getRedisClient = function(){
if(redisClient){
return redisClient;
}
try {
redisClient = redis.createClient({connect_timeout : 5000, max_attempts : 1});
redisClient.on("error", function(err) {
console.error("Error connecting to redis", err);
redisClient = null;
});
return redisClient;
} catch(ex){
console.log("error initialising redis client " + ex);
return null;
}
};
try {
var client = getRedisClient();
console.log("done!");
} catch (ex){
console.log("Exception");
}
However, with this code my application exits if redis is not available (it shouldn't because i've not given a process.exit() command).
How can I solve this?
Checking for Successful Connection on Start
Using a promise, you could guarantee that at least initially, you were able to connect to redis without error within a specified time period:
const redis = require('redis');
const Promise = require('bluebird');
function getRedisClient(timeoutMs){
return new Promise((resolve, reject) => {
const redisClient = redis.createClient();
const timer = setTimeout(() => reject('timeout'), timeoutMs);
redisClient.on("ready", () => {
clearTimeout(timer);
resolve(redisClient);
});
redisClient.on("error", (err) => {
clearTimeout(timer);
reject(err);
});
});
};
const redisReadyTimeoutMs = 10000;
getRedisClient(redisReadyTimeoutMs)
.then(redisClient => {
// the client has connected to redis sucessfully
return doSomethingUseful();
}, error => {
console.log("Unable to connect to redis", error);
});
You Need Proper Error Handling
The redis client being non-null does NOT guarantee using it won't throw an error.
you could experience infrastructure misfortune e.g. crashed redis process, out of memory or network being down.
a bug in your code could cause an error e.g. invalid or missing arguments to a redis command.
You should be handling redis client errors as a matter of course.
DON'T null the Redis Client on Error
It won't give you much but it will force you to check for null every time you try and use it.
The redis client also has inbuilt reconnect and retry mechanisms that you'll miss out on if you null it after the first error. See the redis package docs, look for retry_strategy.
DO Wrap your redis client code with try .. catch ... or use .catch in your promise chain.
DO Make use of a retry_strategy.