I'm new to Node and CouchDb and I'm trying to get my hands on it.
I'm struggling to make a piece of code to work.
I would like to create a table users, insert a new user and 'at the same time' getting another user.
I'm getting this error when starting up node app.js :
antoine#ubuntu:~/projects/couchDb$ node app.js
Database users exists.
{"error":"conflict","reason":"Document update conflict."}
Leaving saveDoc
events.js:48
throw arguments[1]; // Unhandled 'error' event
^
Error: socket hang up
at createHangUpError (http.js:1107:15)
at Socket.onend (http.js:1188:27)
at TCP.onread (net.js:369:26)
And here is my very code, is there something wrong?
(When I remove the getDoc function the error goes away)
I'm using couchDB 1.0.1 and node 0.6.12
The jdoe4 and jdoe documents are already present in the users database.
var dbHost = "127.0.0.1";
var dbPort = 5984;
var dbName = 'users';
var couchdb = require('felix-couchdb');
var client = couchdb.createClient(dbPort, dbHost);
var user = {
name: {
first: 'John',
last: 'Doe'
}
}
var db = client.db(dbName);
db.exists(function(err, exists) {
if (!exists) {
db.create();
console.log('Database ' + dbName + ' created.');
} else {
console.log('Database ' + dbName + ' exists.');
}
db.saveDoc('jdoe4', user, function(err, doc) {
if( err) {
console.log(JSON.stringify(err));
} else {
console.log('Saved user.');
}
console.log('Leaving saveDoc');
});
db.getDoc('jdoe', function(err,doc) {
if( err) {
console.log(JSON.stringify(err));
} else {
console.log(JSON.stringify(doc));
}
console.log('Leaving getDoc');
});
});
It seems to be a library problem => Github Issue Socket Hangout
Look here if you dont know which library fit more your needs.
I quickly realized that felix-couchdb is not compatible with node 8 (I know you're not using version 8, but you will someday), so I switched to nano couchdb and here's the following code:
It will check if the db is created
It will insert only if the key given is unique
It will get the user with a key
var couchdb = require('nano')('http://localhost:5984')
, dbName = 'users'
, db = couchdb.use(dbName)
;
var user = {
name: {
first: 'John',
last: 'Doe'
}
}
couchdb.db.create(dbName, function(err, db_body) {
db.insert(user, 'jdoe4', function(err, doc, header) {
if( err) {
console.log('Cannot save user');
} else {
console.log('Saved user.');
}
console.log('Leaving saveDoc');
});
db.get('jdoe', function(err, doc) {
if( err) {
console.log('Cannot get user');
} else {
console.log(JSON.stringify(doc));
}
console.log('Leaving getDoc');
});
});
One thing worth noting is that, while this will get them at the "same time," it's still making 3 requests to the db, 1 to check if it exists, 1 to insert (or not), 1 to get.
Related
I am writing a web application that uses asynchronous database requests as a part of the api. Currently, I have an async express route that awaits function returns from async functions. Both of these functions return booleans and both query the database. One works correctly, however the second one does not.
Here is the MongoClient setup:
const MongoClient = require('mongodb').MongoClient;
const uri = config.uri; // Contains custom url for accessing database
const client = new MongoClient(uri, { useUnifiedTopology: true}, { useNewUrlParser: true }, { connectTimeoutMS: 30000 }, { keepAlive: 1});
where config is from a file imported as.
const config = require("./config.js");
and functions properly.
Here is the express setup:
app.post("/signup", async function(request, response) {
log("POST request at /signup");
log("BEFORE UNIQUE USER");
const isUniqueUser = await validateUniqueUser(request.body.email, request.body.password);
log(isUniqueUser);
const status = {
status: null
};
if (isUniqueUser) {
log("AFTER UNIQUE USER");
let userCreated = await createPracticeProfile(request.body.email, request.body.password);
log("user created: " + userCreated);
if (userCreated) {
status.status = "user_created";
}
response.json(status);
} else {
response.json(status);
}
console.log("********************************end");
});
The console outputs:
BEFORE UNIQUE USER
true (which it should be)
AFTER UNIQUE USER
MongoError: Topology is closed.
user created: undefined
***...***end
Here is the function for validating that a user is unique:
/* VALIDATE_UNIQUE_USER
USE: ensure user does not have existing profile
PARAMS: email (string), password (string)
RETURN: isUniqueUser (bool)
*/
async function validateUniqueUser(email, password) {
// connect to database
const database = await client.connect().catch(err => {
log("ERROR while connecting to database at: validateUniqueUser");
console.log(err);
client.close();
});
// database connection failed
if (!database) {
return false;
}
// connection successful => find user
let user;
try {
user = await database.db("guitar-practice-suite").collection("users").findOne({email: email});
} catch(err) {
log("ERROR while finding user in database at: validateUniqueUser");
console.log(err);
client.close();
return false;
} finally {
client.close();
// user not found (unique)
if (user === null || user === undefined) {
return true;
}
return false;
}
}
Here is the function for inserting the user into the collections:
/* CREATE_PRACTICE_PROFILE
USE: insert a practice profile into the database
PARAMS: email (string), password (string)
RETURN: userCreated (bool)
*/
async function createPracticeProfile(email, password) {
// hash password
let hashedPassword;
try {
hashedPassword = await new Promise((resolve, reject) => {
bcrypt.hash(password, null, null, function(err, hash) {
if (err) {
reject(err);
}
resolve(hash)
});
});
} catch(err) {
log("ERROR while hashing password at: createPracticeProfile");
console.log(err);
return false;
}
// connect to database
const database = await client.connect().catch(err => {
log("ERROR while connecting to database at: validateUniqueUser");
console.log(err);
client.close();
});
// database connection failed
if (!database) {
return false;
}
// database connection successful => insert user into database
let insertUserToUsers;
let insertUserToExercises;
let insertUserToCustomExercises;
try {
insertUserToUsers = await database.db("guitar-practice-suite").collection("users").insertOne({email: email, password: hashedPassword});
insertUserToExercises = await database.db("guitar-practice-suite").collection("exercises").insertOne({email: email});
insertUserToCustomExercises = await database.db("guitar-practice-suite").collection("custom-exercises").insertOne({email: email, exercises: []});
} catch(err) {
log("ERROR while inserting user into database at: createPracticeProfile");
console.log(err);
client.close();
return false;
} finally {
client.close();
return insertUserToUsers && insertUserToExercises && insertUserToCustomExercises;
}
}
I've found the solution to the problem, but I'm not sure I understand the reasoning.
The client.close() in the finally block of the validateUniqueUser function. It was closing the connection before the connection in the createPracticeProfile function was finished inserting the user.
When that line is taken out, the function works.
The issue is client variable needs to be reinstantiated again,
const client = new MongoClient(uri, { useUnifiedTopology: true}, { useNewUrlParser: true }, { connectTimeoutMS: 30000 }, { keepAlive: 1});
Try putting this in start of createPracticeProfile, validateUniqueUser and other functions
I was getting the error
MongoError: Topology is closed
because of the authentication problem
MongoEror: Authentication failed
In my case, the problem was with the password of my database. My password only contained numerical digits.
I changed the password to all characters and both the errors were solved.
Configure your client connection like below example
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var mongoClient = new MongoClient(new Server('localhost', 27017));
mongoClient.open(function(err, mongoClient) {
var db1 = mongoClient.db("mydb");
mongoClient.close();
});
In my case - connecting to AtlasDB using the MongoClient - I had to whitelist the IP i was accessing the cluster from
I think your mongodb service is stopped, to start it
Task Manager -> Services -> Mongodb -> RightClick -> Start
My code has been working fine for a long time and hasn't thrown this error before: MongoError: Topology is closed.
But due to the fact that my laptop was turned on for a long time and I was simultaneously developing other projects on it, while the main one was running in the terminal, mongo most likely did not close one of the connections to the database and opened another in parallel, creating some kind of collision.
In general, in my case, the usual restart of the computer helped and a similar error did not occur again.
I've searched on how to create a sqlite3 database with a callback in Node.js and have not been able to find any links. Can someone point me towards documentation or provide a 2-3 line code sample to achieve the following:
Create a sqlite3 database and catch an error if the creation fails for any reason.
Here is what I've tried:
let dbCreate = new sqlite3.Database("./user1.db", sqlite3.OPEN_CREATE, function(err){
if(!err){
logger.infoLog("Successfully created DB file: " + dbFileForUser + " for user: " + username );
} else {
logger.infoLog("Failed to create DB file: " + dbFileForUser + ". Error: " + err );
}
});
dbHandler[username] = dbCreate;
When I execute this, I get the following error:
"Failed to create DB file: ./database/user1.db. Error: Error: SQLITE_MISUSE: bad parameter or other API misuse"
This call without callback works just fine.
var customDB = new sqlite3.Database("./custom.db", sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE);
But in this, I will not know if I run into any errors while creating the Database.
Try this:
let userDB = new sqlite3.Database("./user1.db",
sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE,
(err) => {
// do your thing
});
Example.
#Irvin is correct, we can have a look at http://www.sqlitetutorial.net/sqlite-nodejs/connect/ and
check it says if you skip the 2nd parameter, it takes default value as sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE
and in this case if database does not exist new database will be created with connection.
sqlite3.OPEN_READWRITE: It is to open database connection and perform read and write operation.
sqlite3.OPEN_CREATE : It is to create database (if it does not exist) and open connection.
So here is the first way where you have to skip the 2nd parameter and close the problem without an extra effort.
const sqlite3 = require("sqlite3").verbose();
let db = new sqlite3.Database('./user1.db', (err) => {
if (err) {
console.error(err.message);
} else {
console.log('Connected to the chinook database.|');
}
});
db.close((err) => {
if (err) {
return console.error(err.message);
}
console.log('Close the database connection.');
});
And this is the 2nd way to connect with database (already answered by #Irvin).
const sqlite3 = require("sqlite3").verbose();
let db = new sqlite3.Database('./user1.db', sqlite3.OPEN_READWRITE | sqlite3.OPEN_CREATE
, (err) => {
if (err) {
console.error(err.message);
} else {
console.log('Connected to the chinook database.');
}
});
db.close((err) => {
if (err) {
return console.error(err.message);
}
console.log('Close the database connection.');
});
I'm trying to do something relatively simple and am running into a "server ...-a.mongolab.com:36648 sockets closed" error all of a sudden every time I try to do an "insert".
Reads seem to work without error, but inserts seem to get an error every time and I'm not sure if it's my code (which recently underwent minor changes), or a reliability problem with the free server I'm using at MongoLab (which recently showed itself to be down for a few minutes).
Oddly enough, the record itself seems to save okay, I just get the error back!
Can anyone see an issue with my code, or could this be something else?
var mongoClient = require('mongodb').MongoClient;
var http = require('http');
var connectionString = "...";
var pictureWallsCollectionName = 'PictureWalls';
//this is what barfs. see *** details
exports.saveWall = function (req, res) {
//reformat
var toSave = {
_id: req.body.wallId,
pictures: req.body.pictures
};
var status;
mongoClient.connect(connectionString, function (err, db) {
if (err) { return console.error(err); }
var collection = db.collection(pictureWallsCollectionName);
//*** no err yet... ***
collection.insert(
toSave,
function (error, response) {
//*********************
//*** err here! ******
//*********************
db.close();
if (error) {
console.error(error);
//bad
status = 500;
}
else {
console.log('Inserted into the ' + collection_name + ' collection');
//good
status = 200;
}
});
response.status(status).end(http.STATUS_CODES[status]);
});
}
//this seems to work pretty reliably. including it just in case it's relevant
exports.findByWallId = function (req, res) {
var id = req.params.id;
console.log('Retrieving wall: ' + id);
mongoClient.connect(connectionString, function (err, db) {
if (err) { return console.dir(err); }
var collection = db.collection(pictureWallsCollectionName);
collection.findOne(
{ _id: id },
function (err, item) {
db.close();
if (err) {
console.error(err);
//something bad happened
var status = 500;
res.status(status).end(http.STATUS_CODES[status]);
}
else {
console.log('Found wall with ID ' + id);
//reformat and send back in the response
res.send({
wallId: item._id,
pictures: item.pictures
});
}
}
);
});
};
EDIT: Part of my original issue was duplicate parameter names. See the linked question for detail.
ORIGINAL RESPONSE:
The issue ended up being that I was calling:
res.status(status).end(http.STATUS_CODES[status]);
...before the async insert was finished, so it barfed.
However, I'm not exactly sure how to issue the response in this case. See my new question here:
How Do I Properly Issue Response To Post When Waiting For Async Method To Complete?
I have the following code in nodejs that uses the pg (https://github.com/brianc/node-postgres)
My code to create subscriptions for an employee is as such.
client.query(
'INSERT INTO subscriptions (subscription_guid, employer_guid, employee_guid)
values ($1,$2,$3)', [
datasetArr[0].subscription_guid,
datasetArr[0].employer_guid,
datasetArr[0].employee_guid
],
function(err, result) {
done();
if (err) {
set_response(500, err, res);
logger.error('error running query', err);
return console.error('error running query', err);
}
logger.info('subscription with created');
set_response(201);
});
As you have already noticed datasetArr is an array. I would like to create mass subscriptions for more than one employee at a time. However I would not like to loop through the array. Is there a way to do it out of the box with pg?
I did a search for the same question, but found no solution yet.
With the async library it is very simple to use the query several times, and do the necessary error handling.
May be this code variant helps.
(for inserting 10.000 small json objects to an empty database it took 6 sec).
Christoph
function insertData(item,callback) {
client.query('INSERT INTO subscriptions (subscription_guid, employer_guid, employee_guid)
values ($1,$2,$3)', [
item.subscription_guid,
item.employer_guid,
item.employee_guid
],
function(err,result) {
// return any err to async.each iterator
callback(err);
})
}
async.each(datasetArr,insertData,function(err) {
// Release the client to the pg module
done();
if (err) {
set_response(500, err, res);
logger.error('error running query', err);
return console.error('error running query', err);
}
logger.info('subscription with created');
set_response(201);
})
It looks for me that the best way is the usage PostgreSQL json functions:
client.query('INSERT INTO table (columns) ' +
'SELECT m.* FROM json_populate_recordset(null::your_custom_type, $1) AS m',
[JSON.stringify(your_json_object_array)], function(err, result) {
if(err) {
console.log(err);
} else {
console.log(result);
}
});
To do Bulk insert into Postgresql from NodeJS, the better option would be to use 'COPY' Command provided by Postgres and pg-copy-streams.
Code snippet from : https://gist.github.com/sairamkrish/477d20980611202f46a2d44648f7b14b
/*
Pseudo code - to serve as a help guide.
*/
const copyFrom = require('pg-copy-streams').from;
const Readable = require('stream').Readable;
const { Pool,Client } = require('pg');
const fs = require('fs');
const path = require('path');
const datasourcesConfigFilePath = path.join(__dirname,'..','..','server','datasources.json');
const datasources = JSON.parse(fs.readFileSync(datasourcesConfigFilePath, 'utf8'));
const pool = new Pool({
user: datasources.PG.user,
host: datasources.PG.host,
database: datasources.PG.database,
password: datasources.PG.password,
port: datasources.PG.port,
});
export const bulkInsert = (employees) => {
pool.connect().then(client=>{
let done = () => {
client.release();
}
var stream = client.query(copyFrom('COPY employee (name,age,salary) FROM STDIN'));
var rs = new Readable;
let currentIndex = 0;
rs._read = function () {
if (currentIndex === employees.length) {
rs.push(null);
} else {
let employee = employees[currentIndex];
rs.push(employee.name + '\t' + employee.age + '\t' + employee.salary + '\n');
currentIndex = currentIndex+1;
}
};
let onError = strErr => {
console.error('Something went wrong:', strErr);
done();
};
rs.on('error', onError);
stream.on('error', onError);
stream.on('end',done);
rs.pipe(stream);
});
}
Finer details explained in this link
Create your data structure as:
[ [val1,val2],[val1,val2] ...]
Then convert it into a string:
JSON.stringify([['a','b'],['c']]).replace(/\[/g,"(").replace(/\]/g,")").replace(/"/g,'\'').slice(1,-1)
append it to the query and you are done!
Agreed it has string parsing costs but its way cheaper than single inserts.
Use an ORM; eg: Objection.
Also, Increase the Connection pool size based on your db server and the number of active connection you need.
someMovie
.$relatedQuery('actors')
.insert([
{firstName: 'Jennifer', lastName: 'Lawrence'},
{firstName: 'Bradley', lastName: 'Cooper'}
])
.then(function (actors) {
console.log(actors[0].firstName);
console.log(actors[1].firstName);
});
I am using node-postgres, and at the beginning of my application I want to check whether the database exists or not. So my workflow idea is as following:
Check whether myDb is existing
If it is there, create the tables
If not, then create first the database, then tables
As you see it is a really easy process, however, the driver implementation requires to have a database name postgres://username:password#host/database to be connected, which means you need to connect to a database first.
So what I am doing now is to connect to postgres database at the beginning, making a query to create database, cathing the exception if it is already there, then closing my connection and connecting to the newly created database, then creating the tables. Here is the code:
var conStringPri = 'postgres://' + username + ':' + password + '#' + host +
'/postgres';
var conStringPost = 'postgres://' + username + ':' + password + '#' + host +
'/' + dbName;
pg.connect(conStringPri, function(err, client, done) { // connect to postgres db
if (err)
console.log('Error while connecting: ' + err);
client.query('CREATE DATABASE ' + dbName, function(err) { // create user's db
if (err)
console.log('ignoring the error'); // ignore if the db is there
client.end(); // close the connection
// create a new connection to the new db
pg.connect(conStringPost, function(err, clientOrg, done) {
// create the table
clientOrg.query('CREATE TABLE IF NOT EXISTS ' + tableName + ' ' +
'(...some sql...)';
});
});
});
As you see I am opening and closing the connection twice, and this way seems wrong to me. I'll be glad if you propose a better way, or maybe explain how did you accomplish this.
As you see it is a really easy process, however, the driver
implementation requires to have a database name
postgres://username:password#host/database to be connected, which
means you need to connect to a database first.
It's not because of the driver implementation, it's PostgreSQL itself. It's the same with any other language or driver.
A client needs to be connected to a database in order to do anything, including a CREATE DATABASE. Besides the postgres database, template1 is often used for this purpose too.
Then, since you must connect to the freshly created database to create objects inside it, there's no way to avoid opening another connection.
In short, what you're doing can't be simplified, it's already optimal.
I've just written a module for that: https://github.com/olalonde/pgtools
var pgtools = require('pgtools');
pgtools.createdb({
user: 'postgres',
password: 'some pass',
port: 5432,
host: 'localhost'
}, 'test-db', function (err, res) {
if (err) {
console.error(err);
process.exit(-1);
}
console.log(res);
});
Hopefully it can make your code a bit cleaner.
This is a bit old but I just want to share how I handled this kind of setup.
You need to call the third param from the callback which is the done from pg.connect(conn, (err, client, done) => {}). This will release the connection and bring back to pool.
async.series([
done => {
pg.connect(connPrimary, (err, client, releaseConn) => {
if (err) return done(err)
client.query(`CREATE DATABASE ${conf.database}`, (err) => {
if (err && !~err.message.indexOf('already exists')) {
return done(err)
}
client.end()
releaseConn()
done()
})
})
},
done => {
let connSecondary = `postgres://${conf.user}:${conf.password}#${conf.host}:${conf.port}/${conf.database}`
pg.connect(connSecondary, (err, client, releaseConn) => {
if (err) return done(err)
let createTableQuery = `CREATE TABLE IF NOT EXISTS test_table(_id bigint primary key, co2_field varchar(40) NOT NULL, temp_field int NOT NULL, quality_field decimal NOT NULL, reading_time_field timestamp NULL)`
client.query(createTableQuery, err => {
if (err) return done(err)
releaseConn()
done()
})
})
}
], err => {
should.ifError(err)
doneInit()
})
Here is a script I use which is essentially just executing shell commands with execa:
import execa from 'execa';
class DatabaseService {
public async setupDatabase() {
const logCmd = (cmd: execa.ExecaChildProcess) => {
cmd.stdout.on('data', (data) => {
this.logger.log(data.toString());
});
cmd.stderr.on('data', (data) => {
this.logger.error(data.toString());
});
};
const createUser = () => {
return new Promise<void>((resolve, reject) => {
const cmd = execa('createuser', [Config.databaseUser, '--superuser']);
logCmd(cmd);
let userExists = false;
cmd.stderr.on('data', (data) => {
if (
data
.toString()
.includes(`role "${Config.databaseUser}" already exists`)
) {
userExists = true;
}
});
cmd.on('exit', (code) => {
if (!userExists && code) {
reject(new Error(`Failed to create user for database: ${code}`));
} else {
resolve();
}
});
});
};
const createDatabase = () => {
return new Promise<void>((resolve, reject) => {
const cmd = execa('createdb', [Config.databaseName]);
logCmd(cmd);
let databaseExists = false;
cmd.stderr.on('data', (data) => {
if (
data
.toString()
.includes(`database "${Config.databaseName}" already exists`)
) {
databaseExists = true;
}
});
cmd.on('exit', (code) => {
if (!databaseExists && code) {
reject(new Error(`Failed to create database: ${code}`));
} else {
resolve();
}
});
});
};
await createUser();
await createDatabase();
}
}
As you can see, the script detects if the user or database already exists and will ignore errors in those events, because the intended state of Postgres will have been met, and thats all I care about when I run it.
Install
npm install --save -g pgtools
CLI Example
createdbjs my_awesome_db --user=admin --password=admin