I have an idea to use both NoSQL (Google datastore) and SQL (google SQL) together in my nodejs project hosted on Google cloud platform. The reason for this is because I want to keep the statistics, massive amount of data, away from the mySQL db. It will create to many writes, a lot cheaper and faster to use noSQL for this purpose.
I cannot use noSQL for the entire site, the project has already been created with mySQL.
Is this possible? If so please share any guidelines to do this
using sails.js (waterline) you can define as many connections as you want. And then in the model you can select the connection that you want to use.
It is very cool, 2 models, using 2 different connections (mysql - gdatastore) can even have relations between one and the other one.
basically in your model you have to set the connection key { connection: 'mysqldb'}
here you can find the documentation: https://github.com/balderdashy/waterline-docs/blob/master/models/configuration.md
#model
module.exports.connections = {
localMysql: {
adapter: 'sails-mysql',
user: 'root',
host: 'localhost',
database: 'someDbase'
},
remoteMysql: {
adapter: 'sails-mysql',
user: 'remoteUser',
password: 'remotePassword',
host: 'http://remote-mysql-host.com',
database: 'remoteDbase'
},
localMongo:{
adapter: 'sails-mongo',
......
}
};
Related
Recently I've been learning google cloud sql it took a little but I was able to connect my cloud sql auth proxy to my postgres client. However I'm not sure how to query or make post request to my cloud sql. Originally I was just doing
const Pool = require("pg").Pool;
const pool = new Pool({
user: "postgres",
password: "****",
host: "localhost",
port: 5432,
database: "somedb"
});
I'm not sure how to convert this over to try and query the cloud sql db. I did try converting it and got.
const Pool = require("pg").Pool;
const pool = new Pool({
user: "postgres",
password: "****",
host: "[cloud sql ip]",
port: 5432,
database: "[pg/gc db]"
});
I end up getting the error [pg_hba.conf rejects connection for host "[ipv4 ip]", user "postgres", database "[pg/gc db]", no encryption]. I know that the documentation has a code sample but I don't understand it and cant really find any resources on explaining it.
Edit: I am uploading files to a bucket in cloud storage which I was successfully able to do. I plan on mapping out all these files onto a webpage. However I would like to filter them by certain features so I am making a second request after I store the file. My second request will store attributes into a database that I can then relate to the files for filtering.
If you're running the auth proxy from your local machine where you're running your application, then the code will be the same from your application's code perspective. You'll still connect to localhost (although you may need to connect to 127.0.0.1 depending on how you have hosts set up on the machine).
The database field will depend on how you've set up the database in Cloud SQL, but it should be the same as your local database. E.g. if you created a database named "somedb" in Cloud SQL, you don't have to change anything to connect to it in Cloud SQL. The proxy running locally will make everything behave as if you're running the database locally from the application's perspective.
Edit: This particular answer wasn't the issue they were having, but in the comments it came up that both the Proxy and SSL-only was being used, which is (generally) less recommended as it doubles up the SSL/TLS usage because the Proxy also uses generated SSL certificates to connect to Cloud SQL so the database-level SSL connectivity is a redundancy that's likely not needed. There are some edge cases where you may want both, but broadly speaking one or the other is recommended.
I've defined 2 connections in the /config/connections.js file:
monolithMysql: {
user: 'store_app',
database: 'store',
dialect: 'mysql',
options: {
dialect: 'mysql',
host: 'dockerhost',
port: 3306,
logging: console.log
}
},
postgres: {
user: 'user_app',
database: 'user_authentication',
dialect: 'postgres',
options: {
dialect: 'postgres',
host: 'dockerhost',
port: 8201,
logging: console.log
}
}
and in the different models I've put the property connection, so that they're distinguished, as follows:
module.exports = {
options: {
connection: 'monolithMysql', // or 'postgres' in other models.
...
In /config/models.js I set as default connection the monolithMysql one. But that should be overridden by the connection property, if specified, in the models. If I comment out or do not specify the connection property in /config/models.js then Sequelize hook fails to load.
Nevertheless, when trying to query models that have postgres as connection, it still queries them in the MySQL DB, and fails... If I set postgres as default connection, then it will always query in that DB, no matter what the local connection property of the different models says.
Any suggestions how to setup 2 connections at the same time?
Update: found out that it initializes only 1 instance of Sequelize - an instance with the default connection, specified in /config/models.js
Eventually, I ended up writing an upgrade to sails-hook-sequelize to support multiple database connection instances simultaneously.
The pull-request hasn't been reviewed at this point (13 April '16), but it has passed all tests and CI checks, so no worries. (the owner of the repo seems to be neglecting PR reviews in the last couple of months).
You can find the pull-request here.
In order to require the module, along with the multiple db connections upgrade, in your package.json file, just write:
"sails-hook-sequelize": "git#github.com:galioy/sails-hook-sequelize.git#f0c0c5d72ee97ac5504e6f3a0825e37c3587909a"
The configuration is the basic one, as you would do it normally for Sequelize:
add connections configurations to /config/connections.js (see OP)
for each model that you want to use secondary connection - just add
the name of the connection to options.connection property in the model itself (see
OP).
For detailed explanation, see the main comment of the pull-request.
Is it possible to use Sequelize with Redshift? If not, what are the alternatives? I need an ORM for Node.js with built-in support for transactions, hence Sails.js is not an option. I've also looked at Bookshelf, but could not find any support for Redshift either.
I've been able to get Sequelize to at least connect to Redshift (and make a simple SELECT query) with these options:
var Sequelize = require('sequelize');
Sequelize.HSTORE.types.postgres.oids.push('dummy'); // avoid auto-detection and typarray/pg_type error
module.exports = new Sequelize(process.env.REDSHIFT_DATABASE, process.env.REDSHIFT_USER, process.env.REDSHIFT_PASSWORD, {
host: process.env.REDSHIFT_HOST,
port: process.env.REDSHIFT_PORT,
dialect: 'postgres',
pool: false,
keepDefaultTimezone: true, // avoid SET TIMEZONE
databaseVersion: '8.0.2' // avoid SHOW SERVER_VERSION
});
Sequelize is not compatible with Redshift. Though Redshift is written on top of Postgres, it is a columnar DB and major core functions are rewritten.
While trying to connect to it gives an error 'Set Time Zone is not supported'
The following thread shows a few people overriding the time zone error but facing other issues subsequently. 'Using Node 'pg' library to connect to Amazon Redshift
if Redshift is the mandatory you may use the node-jdbc package to connect with Redshift
https://github.com/CraZySacX/node-jdbc
of if ORM is mandatory, you should may try moving your data store to pure Postgres
Redshift is based on top of postgres 8.0.2 (http://docs.aws.amazon.com/redshift/latest/dg/c_redshift-and-postgres-sql.html), so both postgres and sequelize should be able to connect to it.
I don't have any personal experience with it, but the redshift documentation suggests that you can connect to it using regular JDBC / ODBC drivers, so I would be surprised if the node drivers don't work
At this point in time I dont think Sequelize is the best option to connect to Redshift. I recommend that you use node-postgres module directly.
That being said it might be the case that you might already have a project that uses Sequelize and want to reuse it in those cases you can do the following.
const Sequelize = require('sequelize');
// sequelize config options documented at
// https://sequelize.org/master/class/lib/sequelize.js~Sequelize.html#instance-constructor-constructor
const config = {
dialect: 'postgres',
host: 'localhost',
port: 5439,
database: '',
username: '',
password: '',
dialectOptions: {
// either set to ssl to true or use the config options documented at
// https://nodejs.org/api/tls.html#tls_new_tls_tlssocket_socket_options
ssl: true
},
standardConformingStrings: false,
clientMinMessages: false
};
const redshift = new Sequelize(config);
const queryOpts = {type: sequelize.QueryTypes.SELECT, raw:true};
redshift.query('SELECT 1', queryOpts)
.then(console.log)
.catch(console.log);
This will generate the following deprecation warning as Redshift is based on postgres 8.x so consider using pg module directly.
(node:5177) [SEQUELIZE0006] DeprecationWarning: This database engine version is not supported, please update your database server. More information https://github.com/sequelize/sequelize/blob/master/ENGINE.md
I am a newbie in sails.js framework(and for node.js) and I have created myself an API from the address bar using sails.js features. my question is how do I transfer this data to mongodb so I can see it visually.
I have tried to follow this guide: https://www.npmjs.org/package/sails-mongo
but its for 0.9 sails version(current version is 0.10) so that a lot of files that I needed to change has been modified and gone/renamed. I couldn't find an updated tutorial as well so if anyone can please write down how can I implement mongoDB to my sails project that would be wonderful.
In config/connections.js you need to configure your mongoDB connection. Then go to config/models.js, uncomment the connection property and set the name of your mongoDB connection.
In config/connection.js, You can configure your MongoDb connection like this.
module.exports.connections = {
MongoConnection: {
adapter: 'sails-mongo',
host: 'localhost',
port: 27017,
// user: 'username',
// password: 'password',
database: 'DB_Name'
}
};
You need to install sails-mongo package using npm in your working folder.
Then in config/models.js, Add the name of your mongoDb connection like this:
module.exports.models = {
connection: 'MongoConnection',
migrate: 'alter'
};
Thats it, you are ready to go.
As we are building a data feed REST service with node.js and MongoDB/Express, it works very well when the query result is small. But it will hang the server when the client query a large dataset, such as 1m rows (already using gzip to compression). Is this caused by node.js single thread design?
I would like to consulting you about any idea to handle this.
Any comments are welcome:)
Following are the code about the service (with JayData OData Server Module)
app.use('/d.svc', $data.ODataServer({
type: TYPE,
CORS: true,
database: 'odata',
responseLimit: -1,
checkPermission: function (access, user, entitySets, callback) {
logger.info('Check Access Permission for User');// + JSON.stringify(user));
if (access & $data.Access.Create) {
if (user == 'admin') callback.success();
else callback.error('Auth failed');
} else callback.success();
},
provider: {
name: 'mongoDB',
databaseName: 'odata',
address: settings.host,
port: settings.port,
username: USER,
password: PASSWORD
}
}));
Thank you very much.
Luke.
You should never transmit this amount of data in one query. Imagine that you have to keep the results in the memory on the client-side. No matter what technology you use, paged downdload is recommended. In case of JayData, you can implement it based on this blogpost - Synchronized Online/Offline data applications, part 2: Syncing large tables and tables with foreign relations