I have 4 separate tables in the same database. Would it be better to use mysql2.createConnection() or mysql2.createPool to bulk insert into each table? I'd like to run the inserts asynchronously.
The code will be executing the inserts from an AWS Lambda and connections are done through RDS Proxy which handles connection pooling for all connections to the Aurora MySql database instance.
const mysql2 = require('mysql2');
const connection = mysql2.createConnection({
host : 'example.org',
user : 'bob',
password : 'secret'
});
or
mysql2.createPool
const mysql2 = require('mysql2');
const pool = mysql2.createPool({
connectionLimit : 10,
host : 'example.org',
user : 'bob',
password : 'secret'
});
If you would like to run the inserts asynchronously, you will want createPool.
Because in with createConnection, there is only 1 connection and all queries executed on that connection are queued, and that is not really asynchronous. (Async from node.js perspective, but the queries are executed sequentially)
Related
I'm trying to use Redis in a nextJs API which has a custom express server. I created a util file for Redis in order to connect to the Redis labs, the same way I did with mongodb. The problem is that when I try to use the client.Hset the console keeps sending me errors that econ refused, but I also made sure that when Redis is connected it console logs a string. So the console, logs both the error of econ refused, and the log connected to redis. This is how my util file looks like:
/* ------ connecting to redis ------ */
const redis = require('redis');
const { promisify } = require('util');
const client = redis.createClient({
host: process.env.REDIS_HOSTNAME,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASSWORD
});
client.on('connect', ()=>{
console.log('connected to redis')
})
/* promisifying redis in order to use async functionality */
const redisHget = promisify(client.hget).bind(client)
const redisHset = promisify(client.hset).bind(client)
module.export = {redisHget, redisHset}
It seems like every time I call the API, it reconnects to Redis. may I know if there is a way I can use redis the same way I use MongoDB by just creating and importing collections so that it doesn't keep reconnecting to the redis labs server?
Lee Robinson (DevRel # Vercel), wrote a nice article about using Redis in Next.js. https://leerob.io/blog/serverless-redis-nextjs.
Complete Example here: https://github.com/vercel/next.js/blob/canary/examples/with-redis/README.md
Snippets:
# env.local
REDIS_URL=redis://:password#endpoint:port
We'll use ioredis, a Node.js Redis client with async/await support, to connect to Redis.
// lib/redis.js
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);
export default redis;
Using it, either in pages/ or pages/api/,
import redis from '#/lib/redis';
const value = JSON.parse(await redis.hget('feedback', id));
I use mysql2 module in my NodeJS project.
I understand the concept of database pooling in mysql2 module
(https://www.npmjs.com/package/mysql2).
Before using pool, I used regular connection with mysql2.createConnection() function,
but after some time I got 'PROTOCOL_CONNECTION_LOST' error.
My code is:
db.js:
const mysql = require('mysql2');
const pool = mysql.createPool({
host: 'sql server',
user: 'username',
database: 'database',
password: 'pass',
port: 3306,
connectionLimit: 10,
queueLimit: 0
});
module.exports = pool;
And how I using it:
const db = require('db');
db.query(...);
The query() and execute() functions on pool instance automatically call release() function of pool instance, and that is very good (because I don't need to write that command manually after any query).
But I need to understand: if I work like that (with pool, instead of without pool), there is a guarentee
'PROTOCOL_CONNECTION_LOST' will not be thrown? Should I handle that or the pooling mechanism does it automatically?
I ask because I saw on the internet some code that, for example, re-create the connection.
With pool, I need to re-create the connection sometime?
Thanks a lot!
I have been unable to connect to MySQL database using Node from Lambda function. The error I receive is Task timed out after 4.00 seconds.
Does anyone have any solutions?
Here is an overview of my state:
The AWS RDS database is a MySQL database. It is not confined to the VPC (I am able to connect using host/user/password from MySQLWorkbench).
The execution role of my Lambda function is set to have Lambda as a trusted entity and given AdministratorAccess.
On my local machine, I installed the mysql module, zipped my index.js and node_modules folder, and uploaded to my Lambda function.
I have tried putting the createConnection and connect function inside the handler. I have tried putting my query inside the callback function of the connection function. I have tried increasing the timeout time to 10 seconds.
My code:
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'amazon-string.rds.amazonaws.com',
user : 'myusername',
password : 'mypassword'
});
connection.connect();
exports.handler = (event, context, callback) => {
connection.query("SELECT * FROM table", function(err, rows, fields) {
console.log("rows: " + rows);
callback(null);
});
};
Increase the timeout to one minute. It could be due to the coldstart of the lambda function.
Only your first call should take time, consecutive calls should be very fast, since you are reusing the same connection.
Also, By having higher timeout, does not mean you will be charged for that timeout, you will be charged only for the time the Lambda runs.
Also to speed up the coldstart time you can webpack your scripts,
http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/webpack.html
There is one more issue noticed,
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'amazon-string.rds.amazonaws.com',
user : 'myusername',
password : 'mypassword'
});
connection.connect();
exports.handler = (event, context) => {
connection.query("SELECT * FROM table", function(err, rows, fields) {
console.log("rows: " + rows);
context.succeed('Success');
});
};
Hope it helps.
Since you're using RDS, go check out it's security group configuration. By default RDS's security group will allow inbound connections from your own IP and your default security group on your default VPC. However Lambda, by default, runs under no VPC, and thus is not able to establish a connection to RDS.
Either change your RDS instance to allow all IP addresses, or execute your Lambda function under a VPC that your RDS instance can access, and allow access to the security group.
I have a replica set on MongoDB Atlas and this is my mongo shell connection string which connects perfectly:
$ mongo "mongodb://MY_SERVER-shard-00-00-clv3h.mongodb.net:27017,MY_SERVER-shard-00-01-clv3h.mongodb.net:27017,MY_SERVER-shard-00-02-clv3h.mongodb.net:27017/MY_DATABASE?replicaSet=MY_REPLICASET-NAME-shard-0" --ssl --username MY_USERNAME --password MY_PASSWORD --authenticationDatabase MY_ADMIN_DATABASE
How Can I convert it to use in mongoose? How Can I build my uri and options variable?
I tried the following without success:
// connection string using mongoose:
var uri = 'mongodb://MY_USER:MY_PASSWORD#' +
'MY_SERVER-shard-00-00-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-01-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-02-clv3h.mongodb.net:27017/MY_DATABASE';
var options = {
replset: {
ssl: true,
authSource: 'MY_ADMIN_DATABASE',
rs_name: 'MY_REPLICASET_NAME-shard-0'
}
};
mongoose.connect(uri, options);
var db = mongoose.connection;
I've tried including user: and pass: on options, removing MY_USER:MY_PASSWORD# from uri, change rs_name to replicaSet, every unsuccessful attempt. It seems that mongoose is not considering the authSource option.
Using the mongojs, it works fine with the following code:
// connection string using mongojs:
var uri = 'mongodb://MY_USER:MY_PASSWORD#' +
'MY_SERVER-shard-00-00-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-01-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-02-clv3h.mongodb.net:27017/MY_DATABASE';
var options = {
ssl: true,
authSource: 'MY_ADMIN_DATABASE',
replicaSet: 'MY_REPLICASET_NAME-shard-0'
};
var db = mongojs(uri,'', options);
But, I need to use mongoose because the ODM in my project.
How can I build my uri and options variable using mongoose?
ON MONGODB 3.4.x
I resolved this issue putting the 'options' value directly in 'uri' string, according to documentation (http://mongoosejs.com/docs/connections.html) on 'Replica Set Connections' section.
// connection string using mongoose:
var uri = 'mongodb://MY_USER:MY_PASSWORD#' +
'MY_SERVER-shard-00-00-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-01-clv3h.mongodb.net:27017,' +
'MY_SERVER-shard-00-02-clv3h.mongodb.net:27017/MY_DATABASE' +
'ssl=true&replicaSet=MY_REPLICASET_NAME-shard-0&authSource=MY_ADMIN_DATABASE';
mongoose.connect(uri);
var db = mongoose.connection;
Now, it is working fine!
NOTICE WITH MONGODB 3.6
On MongoDB Atlas using the version 3.6.x, the connection string changed to use a DNS server making the link shorter.
mongodb+srv://MY_USER:MY_PASSWORD#MY_SERVER.mongodb.net/MY_DATABASE
...if you use this connection string in your application, this will connect with success but it will be able to read and write only with atlas users with higher privilegies access (atlasAdmin, readWriteAnyDatabase...).
To you work with an specific user with privilege only to readWrite your database, you will need to keep the same connection string used in MongoDB 3.4 because the mongoose not recognized the DNS option (mongodb+srv).
P.S. all the new resources from MongoDB 3.6.x will continue working normally!
Add username and password to database connection
mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
Standard Connection String Format
Am working with node.js and mongodb and am quite new to the concepts, when tried to insert a huge data on to the mongodb the connection made with the mongodb on the local machine lost and the insertion stops and i found that the connection is made to insert the data to the db is 5 as it is the default connection, can please some one help me to increase the default connection from 5 or any suggestions to insert a huge data on to the mongodb using node.js. Am using mongoclient to establish the connection to the db.
The default pool size for connecting mongodb is 5 which is set as a default value. To increase the pool size use
var MongoClient = require('mongodb').MongoClient;
var mongoServer = require('mongodb').Server;
var serverOptions = {
'auto_reconnect': true,
'poolSize': 5
};
var mongoClient = new MongoClient(new mongoServer('localhost', 27017, serverOptions));
//Where in the pool size can be increased by increasing the poolSize value
//eg: 'poolSize':100 will increase the connections into 100 provided the available connections are more.