I created db.js file into my node project
const environment = process.env.NODE_ENV || 'development';
const configuration = require('../knexfile')[environment]; var
database = require('knex')(configuration);
module.exports = database;
I call require('xxxxxx/db.js') where I need knex, to make sql query for Postgres database (through different files). It eats my postgres connections (reach 100 pq connections very soon). I checked it with this script code under Postgres database.
select
sum( numbackends )
from
pg_stat_database;
What is the best practice to use knex into nodejs ?
const Knex = require('knex');
const { host, user, password, database, port } = require('settings')
const knex = Knex({
client: 'pg',
connection: { host, user, password, database, port },
pool: { min: 0, max: 200 }
});
I suggest to use Knex.raw(sql, args) as much as possible. Just because Knex uses the connection pool and release. no need to worry about it.
Hope this will help you somehow.
const knex = require('knex')({
client: 'pg',
connection: {
host : '127.0.0.1',
port : 3306,
user : 'your_database_user',
password : 'your_database_password',
database : 'myapp_test'
}
});
Source - https://knexjs.org/guide/#configuration-options
Related
I am trying to migrate my google cloud app engine from Redis 3.x to 4.x. However, it appears that there have been some major changes in Redis 4.x. It appears that the client no longer autoconnect and there have been some chnages to the syntax. Here's what I have run
'use strict';
import {createClient} from 'redis';
// These are just values stored in environment variables.
const REDISHOST = process.env.REDIHOST;
const REDISPORT = process.env.REDIPORT;
const REDISAUTH = process.env.REDISAUTH;
const redisClient.createClient();
redisClient.host = REDISHOST;
redisClient.port = REDISPORT;
redisclient.auth = REDISAUTH;
redisClient.on('error', (err) => console.error(`##### REDIS ERR: ${err}.`));
await redisClient.connect();
I can tell that host, port, and auth is being set in redisClient, but when I connect, it tries to connect to localhost and fails. Any idea what I am missing here?
You need to pass the connection information in the call the createClient():
const redisClient = createClient({
socket: {
host: REDISHOST,
port: REDISPORT
},
password: REDISAUTH
})
There are lots of options for connecting. They are all detailed in the client configuration guide.
Here is my complete code for sql connection, all code I have got from stackoverflow issues.
Everywhere, I found the same code is being suggested, hence I also tried with the same.
I have some other application which uses same connection with NextJs and it works fine, however, If I try only with NodeJS code, it gives some socket hang up error (code:'ESOCKET' name:'ConnectionError').
Please make a note that TCP is already configured on remote server and its working fine with other applications.
Any help is appreciated, thank you.
const express = require('express');
const fs = require('fs');
const path = require('path');
const cheerio = require("cheerio");
const sql = require('mssql');
require('dotenv').config(); //to use the env variables
// config for your database
var config = {
user: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
server: process.env.DATABASE_HOST,
database: process.env.SOMEDB,
port: 14345, // process.env.DATABASE_PORT,
options: {
encrypt: true, // for azure
trustServerCertificate: false // change to true for local dev / self-signed certs
}
};
// make sure that any items are correctly URL encoded in the connection string
let appPool = new sql.ConnectionPool(config);
//I got error on below connect
sql.connect(config).then(function(pool) {
//It never reaches here, it directly goes to the catch block
app.locals.db = pool;
const server = app.listen(3000, function () {
const host = server.address().address
const port = server.address().port
console.log('Example app listening at http://%s:%s', host, port)
})
}).catch(function(err) {
console.error('Error creating connection pool', err)
});
I have the same issue.
Try to use mssql version 6.0.1, it works on my code, but for sure we need to figure out the problem, since we can't think to mantain forever an old version of a package.
I kept trying to find the solution with different different configuration changes.
Finally, I have made a proper config, which worked and now its connecting properly as well as returning the data from the table.
require('dotenv').config(); //to access the process.env params
const sql = require("mssql"); //mssql object
var dbConfig = {
user: "ajay",
password: "abcd123",
server: "your_remote_sql_server_path",
port: 1433,
database: "your_database_name",
options: {
database: 'your_database_name',
trustServerCertificate: true
}
};
try {
//connection config will be used here to connect to the local/remote db
sql.connect(dbConfig)
.then(async function () {
// Function to retrieve the data from table
const result = await sql.query`select top 1 * from table_name`
console.dir(result)
}).catch(function (error) {
console.dir(error);
});
} catch (error) {
console.dir(error);
}
I am not sure what was the exact issue, but as per the previous config and this one, it seems like adding database name to the options has solved the issue.
Please make sure to save all the sensitive data to the .env file. (which you can access as PROCESS.env.parametername)
For me in driver mssql#9.1.1 making encrypt=false worked
const config = {
user: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
server: process.env.DATABASE_HOST,
database: process.env.SOMEDB,
port: 14345, // process.env.DATABASE_PORT,
options: {
encrypt: false
}
};
I have a Google cloud SQL instance and I want to be able to access it from my node.js application running locally on my machine.
I have enabled Cloud SQL instance access through public IP and created a network on GCP with my local machine IP and I have tested the connection in several ways:
MySQL Workbench
terminal
From my NodeJS application running locally using mysql2 to connect
All the above mentioned ways connected successfully and I can work on my cloud SQL instance as expected, the problem is when trying to allow my NodeJS app to connect using sequelize, I can see this error
UnhandledPromiseRejectionWarning: SequelizeAccessDeniedError: Access denied for user 'root'#'localhost' (using password: YES)
this is my connection code:
host: 'XXX.XX.XX.XX',
port: "",
user: "root",
password: 'password',
database: "DB_NAME",
dialect: "mysql",
pool: {
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
const db = {};
const connection = await mysql.createConnection({ host, port, user, password });
await connection.query(`CREATE DATABASE IF NOT EXISTS \`${database}\`;`);
const sequelize = new Sequelize(database, user, password, { dialect: dialect });
db.Sequelize = Sequelize;
db.sequelize = sequelize;
// // init models and add them to the exported db object
db.orders = require("../../models/order.model.js")(sequelize, Sequelize);
db.sequelize.sync().then(function(){
console.log('DB connection sucessful.');
}, function(err){
// catch error here
});
It Seems that I should pass the host as somehow it was considering host to be localhost when it is not explicitly specified
So this:
const sequelize = new Sequelize(database, user, password, { dialect: dialect });
Should be updated to this:
const sequelize = new Sequelize(database, user, password, { host: host, dialect: dialect });
I have Node.js code to connect to a MySQL database:
var mysql = require('mysql')
var express = require('express')
var app = express()
var connection = mysql.createPool({
connectionLimit: 50,
host : 'ip',
user : 'username',
password : 'pass',
database : 'mydb'
});
app.get('/', function(req, resp) {
connection.getConnection(function(error, tempCont) {
if(!!error) {
tempCont.release();
console.log('Error');
} else {
console.log('Connected!');
tempCont.query("select * from table", function(error, rows, fields) {
tempCont.release();
if(!!error) {
console.log('Error in the query');
} else {
resp.json(rows);
}
});
}
})
})
console.log("listening requests...")
app.listen(1337);
How do I secure an IP address, username and password used for connecting to a database so that is not visible in the code or configuration file?
Install the dotenv module by: npm install --save dotenv
Create a .env file at the root folder and write down the code:
DB_CONLIMIT=50
DB_HOST=ip
DB_USER=username
DB_PASSWORD=pass
DB_DATABASE=mydb
In your JavaScript file:
var mysql = require('mysql');
var express = require('express');
var app = express();
const dotenv = require('dotenv').config();
var connection = mysql.createPool({
connectionLimit : process.env.DB_CONLIMIT,
host : process.env.DB_HOST,
user : process.env.DB_USER ,
password : process.env.DB_PASSWORD ,
database : process.env.DB_DATABASE
});
You should be configuring your systems so that your service runs as its own user with its own protected files. This offers some protection so that even if another service is compromised, the intruding user's access is isolated from other components of your system. Don't run things as root.
As for how secrets are stored and accessed, that's up to you. You can have a configuration file if you want. Another option is to use environment variables. Ultimately; however, your secrets are going to have to be stored in plaintext somewhere for your system to read and use.
Another method worth mentioning is you could possibly separate your secrets from your applications by having a dedicated secrets service. All your applications would have to know about this service and from there they could request the secrets they need for their regular operation. This has the obvious caveat of all your applications depend on the secrets service on start up - if that goes down your applications won't be able to start or restart.
I am pretty new to heroku and node. While I was trying to connect to heroku db, the following error shows up.
Error: connect ECONNREFUSED 127.0.0.1:5432
I am using connection pooling:
var pg = require('pg');
var heroconfig =process.env.DATABASE_URL || "postgres://jykyslkwkdsvhz:3ba43ff7db0c8dv9a914bac02f55ce944d8ccec31b67f858df3a858faa386c8e#ec2-54-243-214-198.compute-1.amazonaws.com:5432/dfiijlh3fbe3g9";
//var pool1 = new Pool(heroconfig);
var pool1 = new Pool(heroconfig);
app.get('/db', function(req, res){
pool1.query('SELECT * FROM test_table;',function(err, result){
if(err){
res.status(500).send(err.toString());
} else{
res.send(JSON.stringify(result.rows));
}
});
});
I tried to look at similar questions form other users but could not find solution involving pooling.
Please help.
I figured it out partially,
Storing the configuration data as object as below makes it works
var heroconfig = {
user: 'username',
database: 'database name',
password: 'some pass word',
host: 'host name',
port: 5432,
max: 10,
idleTimeoutMillis: 30000,
};
However while using the line of code mentioned in my original question, where database url is stored into the variable, it is not working:
var heroconfig =process.env.DATABASE_URL || "postgres://jykyslkwkdsvhz:3ba43ff7db0c8dv9a914bac02f55ce944d8ccec31b67f858df3a858faa386c8e#ec2-54-243-214-198.compute-1.amazonaws.com:5432/dfiijlh3fbe3g9";
I am planning to store my credentials in a different file and require it in my server file which seems to be a better approach.
I know this is late, but according to the docs in order to use a connection string, you must do this:
const { Pool, Client } = require('pg')
const connectionString = 'postgresql://dbuser:secretpassword#database.server.com:3211/mydb'
const pool = new Pool({
connectionString: connectionString,
})
See here: https://node-postgres.com/features/connecting#connection-uri
I have had such an error. After a lot of hours of research, I found out that my server deployed to Heroku was trying to access my PC PostgreSQL database. But it should have connected to the added-on PostgreSQL database in Heroku. I mean my server wasn't connecting to the database link in production mode, it was connecting to the database in development mode. I fixed it in my code like this.
db.js contents:
// focus on const environment
const environment = process.env.NODE_ENV || "development";
const knex = require("knex");
const knexfile = require("./knexfile");
const db = knex(knexfile[environment]);
module.exports = db;