i have the following code (shortened it a bit)
dbInterface.js
const sqlConnection = require('sqlite3').verbose()
const data = require('./data.js')
//open database --> uses create/readwrite per default
let db = new sqlConnection.Database('./db/test_db.db', (err) => {
if (err) {
console.error('Error connecting to database')
}
console.log('Connected to the chinook database.')
})
exports.dbFunctions = {
userPresent: userPresent,
createTable: createTable,
tablePresent: tablePresent,
dropTable: dropTable,
dropAll: dropTable,
addUser: addUser,
deleteUser: deleteUser,
showTableContent: showTableContent,
addHistory: addHistory,
clearHistory: clearHistory,
removeLastHistoryEntry: removeLastHistoryEntry,
getHistory: getHistory
}
function userPresent (id) {
// noinspection SqlResolve
const statement = 'SELECT * FROM user WHERE pk_user_id == ' + id
return new Promise((resolve, reject) => {
db.all(statement, (err, row) => {
if (err) {
console.log('reject')
reject(false)
return
}
console.log('resolve')
resolve(true)
})
})
}
When I access the exported functions on the same folder level everything works fine.
As soon as I call the functions from another level like /tests/db_tests.js
I can work with for example userPresent but the database cannot establish a connection unless I change it to:
let db = new sqlConnection.Database('../db/test_db.db', (err) => {
if (err) {
console.error('Error connecting to database')
}
console.log('Connected to the chinook database.')
})
See I changed the path location from ./db/test_db.db to ../db/test_db.db
Since my app access the dbInterface from many different locations I feel very stuck to solve this issue.
I also tried it with an index.js that is on the same level lile the dbInterface.js and contained only this
exports.dbInterface = require('./dbInterface')
But this also didn't work.
Can anyone help out please?
try absolute path
const path = require('path');
let dbPath = path.join(process.cwd(), 'db', 'test_db.db');
Related
I have some NodeJS code using the ldapjs package to query for objects. It works great when I give it a baseDN, say dc=home to start the search. What I would like to do is be able to discover the top-level object dc=home rather than coding it as a pre-configured value.
Here's what I've tried:
Replace listObjects('dc=home') (last line of program) with console.log(listObjects('')) and option scope: 'base' with scope: 'one' and/or scope: 'sub'.
I get errors with listObjects('') and scope set as either 'one' or 'sub'.
listObjects('') with a scope of 'base' does not error, but neither does it return any result.
The code is below:
#!/usr/bin/env node
import ldapjs from 'ldapjs';
const ldapURL = [ 'ldap://127.0.0.1:389' ];
const bindDN = 'uid=search,dc=home';
const bindPassword = 'P#ssw0rd';
function connect(serverURL) {
return new Promise((resolve, reject) => {
const client = ldapjs.createClient({
url: serverURL
});
client.on('connect', () => {
console.debug('Connected to:', ldapURL);
console.debug('Binding as:', bindDN);
client.bind(bindDN, bindPassword, (err) => {
if (err) {
console.debug(err.message);
reject('Bind credentials rejected.');
}
else {
resolve(client);
}
});
});
client.on('error', (err) => {
reject('Unable to connect to ' + serverURL);
});
});
}
/**
* Search LDAP and return objects.
* #baseDN {string} Where to start, like 'ou=People,dc=example,dc=com'
* #filter {string} Optional LDAP query to limit results, like '(objectClass=posixAccount)'
* #returns {promise} Array of query results.
*/
async function listObjects(baseDN, filter) {
let connection = await connect(ldapURL).catch((err) => {
console.error('LDAP server error:', err);
reject(err);
});
let opts = {
filter: filter,
scope: 'base'
};
let results = [];
return new Promise((resolve, reject) => {
connection.search(baseDN, opts, (err, res) => {
res.on('searchEntry', (entry) => {
console.debug(entry);
results.push(entry.objectName);
});
res.on('end', () => {
connection.unbind(() => {
resolve(results);
});
});
});
});
}
console.log(listObjects('dc=home'));
I'm pretty sure this sort of thing is possible. I have a GUI client (LDAPAdmin) that has a 'Fetch DNs' feature for detecting the BaseDN. What I can't figure out is how to replicate this using ldapjs.
Is there a different way to specify the dn or the scope that will let me list the top-level domain component in my directory without knowing its name?
I'm just starting to use Mongodb without mongoose (to get away from the schemas), and wanted to create a simple module with various exported functions to use in the rest of my app. I've pasted the code below.
The problem I'm having is that the databasesList.databases comes back as undefined, and I'm not sure why. There should be 2 databases on my cluster, and one collection in each database.
As a tangential question, I thought maybe I would check the collections instead (now commented out), but though I found this page (https://docs.mongodb.com/manual/reference/method/db.getCollectionNames/) the function getCollectionNames seems not to exist. Now I'm wondering if I'm using the wrong documentation and that is why my databases are coming back undefined.
const client = new MongoClient(uri)
const connection = client.connect( function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server');
}
});
module.exports = {
getDatabaseList: function() {
console.log('start ' + client);
databasesList = client.db().admin().listDatabases();
//collectionList = client.db().getCollectionNames();
//console.log("Collections: " + collectionList);
console.log("Databases: " + databasesList.databases);
//databasesList.databases.forEach(db => console.log(` - ${db.name}`));
}
}```
your code is correct Just need to change few things.
module.exports = {
getDatabaseList: async function() {
console.log('start ' + client);
databasesList = await client.db().admin().listDatabases();
//collectionList = await client.db().getCollectionNames();
//console.log("Collections: " + collectionList);
console.log("Databases: " + databasesList.databases);
databasesList.databases.forEach(db => console.log(` - ${db.name}`));
}
}
You have to declare async function and use await also.
The async and await keywords enable asynchronous, promise-based behaviour to be written in a cleaner style, avoiding the need to explicitly configure promise chains.
You can use this modular approach to build your database access code:
index.js: Run your database application code, like list database names, collection names and read from a collection.
const connect = require('./database');
const dbFunctions = require('./dbFunctions');
const start = async function() {
const connection = await connect();
console.log('Connected...');
const dbNames = await dbFunctions.getDbNames(connection);
console.log(await dbNames.databases.map(e => e.name));
const colls = await dbFunctions.getCollNames(connection, 'test');
console.log(await colls.map(e => e.name));
console.log(await dbFunctions.getDocs(connection, 'test', 'test'));
};
start();
database.js:: Create a connection object. This connection is used for all your database access code. In general, a single connection creates a connection pool and this can be used throughout a small application
const { MongoClient } = require('mongodb');
const url = 'mongodb://localhost:27017/';
const opts = { useUnifiedTopology: true };
async function connect() {
console.log('Connecting to db server...');
return await MongoClient.connect(url, opts );
}
module.exports = connect;
dbFunctions.js:: Various functions to access database details, collection details and query a specific collection.
module.exports = {
// return list of database names
getDbNames: async function(conn) {
return await conn.db().admin().listDatabases( { nameOnly: true } );
},
// return collections list as an array for a given database
getCollNames: async function(conn, db) {
return await conn.db(db).listCollections().toArray();
},
// return documents as an array for a given database and collection
getDocs: async function(conn, db, coll) {
return await conn.db(db).collection(coll).find().toArray();
}
}
I'm trying to create web services using node.js from an sql server database,in the frontend when i call those 2 webservices simultaneously it throws an error Global connection already exists. Call sql.close() first .
Any Solution ?
var express = require('express');
var router = express.Router();
var sql = require("mssql");
router.get('/Plant/:server/:user/:password/:database', function(req, res, next) {
user = req.params.user;
password = req.params.password;
server = req.params.server;
database = req.params.database;
// config for your database
var config = {
user: user,
password: password,
server: server,
database:database
};
sql.connect(config, function (err) {
// create Request object
var request = new sql.Request();
// query to the database and get the records
request.query("SELECT distinct PlantName FROM MachineryStateTable"
, function (err, recordset) {
if (err) console.log(err)
else {
for(i=0;i<recordset.recordsets.length;i++) {
res.send(recordset.recordsets[i])
}
}
sql.close();
});
});
});
router.get('/Dep/:server/:user/:password/:database/:plantname', function(req, res, next) {
user = req.params.user;
password = req.params.password;
server = req.params.server;
database = req.params.database;
plantname = req.params.plantname;
// config for your database
var config = {
user: user,
password: password,
server: server,
database:database
};
sql.connect(config, function (err) {
// create Request object
var request = new sql.Request();
// query to the database and get the records
request.query("SELECT distinct DepName FROM MachineryStateTable where PlantName= '"+plantname+"'"
, function (err, recordset) {
if (err) console.log(err)
else {
for(i=0;i<recordset.recordsets.length;i++) {
res.send(recordset.recordsets[i])
}
sql.close();
}
});
});
});
module.exports = router;
You have to create a poolConnection
try this:
new sql.ConnectionPool(config).connect().then(pool => {
return pool.request().query("SELECT * FROM MyTable")
}).then(result => {
let rows = result.recordset
res.setHeader('Access-Control-Allow-Origin', '*')
res.status(200).json(rows);
sql.close();
}).catch(err => {
res.status(500).send({ message: `${err}`})
sql.close();
});
From the documentation, close method should be used on the connection, and not on the required module,
So should be used like
var connection = new sql.Connection({
user: '...',
password: '...',
server: 'localhost',
database: '...'
});
connection.close().
Also couple of suggestions,
1. putting res.send in a loop isn't a good idea, You could reply back the entire recordsets or do operations over it, store the resultant in a variable and send that back.
2. Try using promises, instead of callbacks, it would make the flow neater
You must use ConnectionPool.
Next function returns a recordset with my query results.
async function execute2(query) {
return new Promise((resolve, reject) => {
new sql.ConnectionPool(dbConfig).connect().then(pool => {
return pool.request().query(query)
}).then(result => {
resolve(result.recordset);
sql.close();
}).catch(err => {
reject(err)
sql.close();
});
});
}
Works fine in my code!
if this problem still bother you, then change the core api.
go to node_modules\mssql\lib\base.js
at line 1723, add below code before if condition
globalConnection = null
In case someone comes here trying to find out how to use SQL Server pool connection with parameters:
var executeQuery = function(res,query,parameters){
new sql.ConnectionPool(sqlConfig).connect().then(pool =>{
// create request object
var request = new sql.Request(pool);
// Add parameters
parameters.forEach(function(p) {
request.input(p.name, p.sqltype, p.value);
});
// query to the database
request.query(query,function(err,result){
res.send(result);
sql.close();
});
})
}
Don't read their documentation, I don't think it was written by someone that actually uses the library :) Also don't pay any attention to the names of things, a 'ConnectionPool' doesn't seem to actually be a connection pool of any sort. If you try and create more than one connection from a pool, you will get an error. This is the code that I eventually got working:
const sql = require('mssql');
let pool = new sql.ConnectionPool(config); // some object that lets you connect ONCE
let cnn = await pool.connect(); // create single allowed connection on this 'pool'
let result = await cnn.request().query(query);
console.log('result:', result);
cnn.close(); // close your connection
return result;
This code can be run multiple times in parallel and seems to create multiple connections and correctly close them.
I have the following code in nodejs that uses the pg (https://github.com/brianc/node-postgres)
My code to create subscriptions for an employee is as such.
client.query(
'INSERT INTO subscriptions (subscription_guid, employer_guid, employee_guid)
values ($1,$2,$3)', [
datasetArr[0].subscription_guid,
datasetArr[0].employer_guid,
datasetArr[0].employee_guid
],
function(err, result) {
done();
if (err) {
set_response(500, err, res);
logger.error('error running query', err);
return console.error('error running query', err);
}
logger.info('subscription with created');
set_response(201);
});
As you have already noticed datasetArr is an array. I would like to create mass subscriptions for more than one employee at a time. However I would not like to loop through the array. Is there a way to do it out of the box with pg?
I did a search for the same question, but found no solution yet.
With the async library it is very simple to use the query several times, and do the necessary error handling.
May be this code variant helps.
(for inserting 10.000 small json objects to an empty database it took 6 sec).
Christoph
function insertData(item,callback) {
client.query('INSERT INTO subscriptions (subscription_guid, employer_guid, employee_guid)
values ($1,$2,$3)', [
item.subscription_guid,
item.employer_guid,
item.employee_guid
],
function(err,result) {
// return any err to async.each iterator
callback(err);
})
}
async.each(datasetArr,insertData,function(err) {
// Release the client to the pg module
done();
if (err) {
set_response(500, err, res);
logger.error('error running query', err);
return console.error('error running query', err);
}
logger.info('subscription with created');
set_response(201);
})
It looks for me that the best way is the usage PostgreSQL json functions:
client.query('INSERT INTO table (columns) ' +
'SELECT m.* FROM json_populate_recordset(null::your_custom_type, $1) AS m',
[JSON.stringify(your_json_object_array)], function(err, result) {
if(err) {
console.log(err);
} else {
console.log(result);
}
});
To do Bulk insert into Postgresql from NodeJS, the better option would be to use 'COPY' Command provided by Postgres and pg-copy-streams.
Code snippet from : https://gist.github.com/sairamkrish/477d20980611202f46a2d44648f7b14b
/*
Pseudo code - to serve as a help guide.
*/
const copyFrom = require('pg-copy-streams').from;
const Readable = require('stream').Readable;
const { Pool,Client } = require('pg');
const fs = require('fs');
const path = require('path');
const datasourcesConfigFilePath = path.join(__dirname,'..','..','server','datasources.json');
const datasources = JSON.parse(fs.readFileSync(datasourcesConfigFilePath, 'utf8'));
const pool = new Pool({
user: datasources.PG.user,
host: datasources.PG.host,
database: datasources.PG.database,
password: datasources.PG.password,
port: datasources.PG.port,
});
export const bulkInsert = (employees) => {
pool.connect().then(client=>{
let done = () => {
client.release();
}
var stream = client.query(copyFrom('COPY employee (name,age,salary) FROM STDIN'));
var rs = new Readable;
let currentIndex = 0;
rs._read = function () {
if (currentIndex === employees.length) {
rs.push(null);
} else {
let employee = employees[currentIndex];
rs.push(employee.name + '\t' + employee.age + '\t' + employee.salary + '\n');
currentIndex = currentIndex+1;
}
};
let onError = strErr => {
console.error('Something went wrong:', strErr);
done();
};
rs.on('error', onError);
stream.on('error', onError);
stream.on('end',done);
rs.pipe(stream);
});
}
Finer details explained in this link
Create your data structure as:
[ [val1,val2],[val1,val2] ...]
Then convert it into a string:
JSON.stringify([['a','b'],['c']]).replace(/\[/g,"(").replace(/\]/g,")").replace(/"/g,'\'').slice(1,-1)
append it to the query and you are done!
Agreed it has string parsing costs but its way cheaper than single inserts.
Use an ORM; eg: Objection.
Also, Increase the Connection pool size based on your db server and the number of active connection you need.
someMovie
.$relatedQuery('actors')
.insert([
{firstName: 'Jennifer', lastName: 'Lawrence'},
{firstName: 'Bradley', lastName: 'Cooper'}
])
.then(function (actors) {
console.log(actors[0].firstName);
console.log(actors[1].firstName);
});
I am using node-postgres, and at the beginning of my application I want to check whether the database exists or not. So my workflow idea is as following:
Check whether myDb is existing
If it is there, create the tables
If not, then create first the database, then tables
As you see it is a really easy process, however, the driver implementation requires to have a database name postgres://username:password#host/database to be connected, which means you need to connect to a database first.
So what I am doing now is to connect to postgres database at the beginning, making a query to create database, cathing the exception if it is already there, then closing my connection and connecting to the newly created database, then creating the tables. Here is the code:
var conStringPri = 'postgres://' + username + ':' + password + '#' + host +
'/postgres';
var conStringPost = 'postgres://' + username + ':' + password + '#' + host +
'/' + dbName;
pg.connect(conStringPri, function(err, client, done) { // connect to postgres db
if (err)
console.log('Error while connecting: ' + err);
client.query('CREATE DATABASE ' + dbName, function(err) { // create user's db
if (err)
console.log('ignoring the error'); // ignore if the db is there
client.end(); // close the connection
// create a new connection to the new db
pg.connect(conStringPost, function(err, clientOrg, done) {
// create the table
clientOrg.query('CREATE TABLE IF NOT EXISTS ' + tableName + ' ' +
'(...some sql...)';
});
});
});
As you see I am opening and closing the connection twice, and this way seems wrong to me. I'll be glad if you propose a better way, or maybe explain how did you accomplish this.
As you see it is a really easy process, however, the driver
implementation requires to have a database name
postgres://username:password#host/database to be connected, which
means you need to connect to a database first.
It's not because of the driver implementation, it's PostgreSQL itself. It's the same with any other language or driver.
A client needs to be connected to a database in order to do anything, including a CREATE DATABASE. Besides the postgres database, template1 is often used for this purpose too.
Then, since you must connect to the freshly created database to create objects inside it, there's no way to avoid opening another connection.
In short, what you're doing can't be simplified, it's already optimal.
I've just written a module for that: https://github.com/olalonde/pgtools
var pgtools = require('pgtools');
pgtools.createdb({
user: 'postgres',
password: 'some pass',
port: 5432,
host: 'localhost'
}, 'test-db', function (err, res) {
if (err) {
console.error(err);
process.exit(-1);
}
console.log(res);
});
Hopefully it can make your code a bit cleaner.
This is a bit old but I just want to share how I handled this kind of setup.
You need to call the third param from the callback which is the done from pg.connect(conn, (err, client, done) => {}). This will release the connection and bring back to pool.
async.series([
done => {
pg.connect(connPrimary, (err, client, releaseConn) => {
if (err) return done(err)
client.query(`CREATE DATABASE ${conf.database}`, (err) => {
if (err && !~err.message.indexOf('already exists')) {
return done(err)
}
client.end()
releaseConn()
done()
})
})
},
done => {
let connSecondary = `postgres://${conf.user}:${conf.password}#${conf.host}:${conf.port}/${conf.database}`
pg.connect(connSecondary, (err, client, releaseConn) => {
if (err) return done(err)
let createTableQuery = `CREATE TABLE IF NOT EXISTS test_table(_id bigint primary key, co2_field varchar(40) NOT NULL, temp_field int NOT NULL, quality_field decimal NOT NULL, reading_time_field timestamp NULL)`
client.query(createTableQuery, err => {
if (err) return done(err)
releaseConn()
done()
})
})
}
], err => {
should.ifError(err)
doneInit()
})
Here is a script I use which is essentially just executing shell commands with execa:
import execa from 'execa';
class DatabaseService {
public async setupDatabase() {
const logCmd = (cmd: execa.ExecaChildProcess) => {
cmd.stdout.on('data', (data) => {
this.logger.log(data.toString());
});
cmd.stderr.on('data', (data) => {
this.logger.error(data.toString());
});
};
const createUser = () => {
return new Promise<void>((resolve, reject) => {
const cmd = execa('createuser', [Config.databaseUser, '--superuser']);
logCmd(cmd);
let userExists = false;
cmd.stderr.on('data', (data) => {
if (
data
.toString()
.includes(`role "${Config.databaseUser}" already exists`)
) {
userExists = true;
}
});
cmd.on('exit', (code) => {
if (!userExists && code) {
reject(new Error(`Failed to create user for database: ${code}`));
} else {
resolve();
}
});
});
};
const createDatabase = () => {
return new Promise<void>((resolve, reject) => {
const cmd = execa('createdb', [Config.databaseName]);
logCmd(cmd);
let databaseExists = false;
cmd.stderr.on('data', (data) => {
if (
data
.toString()
.includes(`database "${Config.databaseName}" already exists`)
) {
databaseExists = true;
}
});
cmd.on('exit', (code) => {
if (!databaseExists && code) {
reject(new Error(`Failed to create database: ${code}`));
} else {
resolve();
}
});
});
};
await createUser();
await createDatabase();
}
}
As you can see, the script detects if the user or database already exists and will ignore errors in those events, because the intended state of Postgres will have been met, and thats all I care about when I run it.
Install
npm install --save -g pgtools
CLI Example
createdbjs my_awesome_db --user=admin --password=admin