Recently I've been learning google cloud sql it took a little but I was able to connect my cloud sql auth proxy to my postgres client. However I'm not sure how to query or make post request to my cloud sql. Originally I was just doing
const Pool = require("pg").Pool;
const pool = new Pool({
user: "postgres",
password: "****",
host: "localhost",
port: 5432,
database: "somedb"
});
I'm not sure how to convert this over to try and query the cloud sql db. I did try converting it and got.
const Pool = require("pg").Pool;
const pool = new Pool({
user: "postgres",
password: "****",
host: "[cloud sql ip]",
port: 5432,
database: "[pg/gc db]"
});
I end up getting the error [pg_hba.conf rejects connection for host "[ipv4 ip]", user "postgres", database "[pg/gc db]", no encryption]. I know that the documentation has a code sample but I don't understand it and cant really find any resources on explaining it.
Edit: I am uploading files to a bucket in cloud storage which I was successfully able to do. I plan on mapping out all these files onto a webpage. However I would like to filter them by certain features so I am making a second request after I store the file. My second request will store attributes into a database that I can then relate to the files for filtering.
If you're running the auth proxy from your local machine where you're running your application, then the code will be the same from your application's code perspective. You'll still connect to localhost (although you may need to connect to 127.0.0.1 depending on how you have hosts set up on the machine).
The database field will depend on how you've set up the database in Cloud SQL, but it should be the same as your local database. E.g. if you created a database named "somedb" in Cloud SQL, you don't have to change anything to connect to it in Cloud SQL. The proxy running locally will make everything behave as if you're running the database locally from the application's perspective.
Edit: This particular answer wasn't the issue they were having, but in the comments it came up that both the Proxy and SSL-only was being used, which is (generally) less recommended as it doubles up the SSL/TLS usage because the Proxy also uses generated SSL certificates to connect to Cloud SQL so the database-level SSL connectivity is a redundancy that's likely not needed. There are some edge cases where you may want both, but broadly speaking one or the other is recommended.
Related
I am using an app on GCP with Node.js with Postgresql (Cloud SQL, lowest tier i.e. 25 connections) using the 'pg' package ("pg": "^8.7.3",). I am quite new with this configuration so there may be some very basic errors here.
I configure my pg_client like this
// CLOUD SQL POSTGRESQL DATABASE
const { Client, Pool } = require('pg')
const pg_client = new Pool({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DB,
password: process.env.PG_PWD,
port: 5432,
})
and then, in order to copy the data from a nosql-database with some 50.000+ items I go through them pretty much like this. I know the code doesn't make perfect sense but this is how the SQL calls are being made:
fiftyThoussandOldItems.forEach(async (item) => {
let nameId = await pg_client.query("SELECT id from some1000items where name='John'")
pg_client.query("INSERT into items (id, name, url) VALUES (nameId, 1,2)"
})
This does however quickly render sorry, too many clients already :: proc.c:362 and error: remaining connection slots are reserved for non-replication superuser connections.
I have done similar runs before without experiencing this issue (but then with about 1000 items).
As far as I understand, I do Not need to do a pg_client.connect() and pg_client.release() (or is it .end()) any longer, according to a SO-answer I unfortunately can't find any longer. Is this really correct? (When I tried to before, I ended up with a lot of other issues that causes other types of problems)
So, my questions are:
What am I doing wrong? Do I need to use pg_client.connect() before every SQL-call and then pg_client.release() after every SQL-call? Or is it pg_client.end()?
Is there a way to have this automatically handled? It doesn't seem very DRY and bug prone.
I am attempting to link an app engine service to an instance of SQL Server on the Google Cloud Platform using this documentation
My service is a nodejs application and I am using the mssql library to connect to my database.
The documentation only describes using TCP/IP (an ip address):
const pool = new sql.ConnectionPool({
user: '...',
password: '...',
server: 'localhost',
database: '...'
})
If I try to use localhost or 127.0.0.1 for my server, the connection fails:
I need help with one of the following:
How can I connect to the SQL instance using TCP/IP using the mssql library?
What is the configuration option for the connection pool using unix domain socket?
I haven't actually done this one yet, so I'm guessing a little bit, but try one of these:
Pass the unix socket to the server config tag.
Pass localhost to the server tag and the unix socket value to the port tag.
I want to connect the databases which are for the different servers (like dev, staging, live, production). There are different databases and credential for all the server. When a server will be called it will access data only that particular database.
In my code, I am using node js with sequelize for the database connection to build the API in GraphQL.How can I do this? I have already tried a lot. Please suggest to me. If need code I can post it here.
The best way to do this is to keep a seperate .env file on each server. and load the credentials form .env file itself by using dotenv module.
you can load credenials by process.env.CREDENTIALS_KEY
Example
DB_USERNAME=username
DB_PASSWPRD=password
DB_DATABASE=db_name
DB_HOST=localhost
Load in your js/ts file like:
"USERNAME": process.env.DB_USERNAME,
"PASSWORD": process.env.DB_PASSWPRD,
"DATABASE": process.env.DB_DATABASE,
"HOST": process.env.DB_HOST,
I'm trying to connect Nodejs to PostgreSQL database, for that I'm using node-postgres.
var pool = new Pool({
user: username,
password: password,
host: server
database: database,
max: 25
});
module.exports = {
execute_query: function (query2) {
//usage of query
pool.query('query2', function(err, result){
return (result);
});
}
};
Then in my application, the function execute_query is called in different places in the application.
Locally it works but I wonder how the pool is managed, is this enough configuration to manage concurrent users if my application is used by different people ?
Do I need to do anything else to ensure that I have clients in the pool ?
Or should I use the old way of managing clients with a hard code ?
I read the documentation of node-postgres and it says that pool.query is the simplist way but it doesnt say how it manages the connections...
Do you have any information ?
Thank you
is this enough configuration to manage concurrent users if my
application is used by different people ?
This is a very broad question and depends on more than one thing. Let me still give this a shot
Number of connection in the pool is the number of active connections your server will maintain with the db. Each connection as a cost as postgres maintains this as a separate process. So just focussing on the connection pool is not enough.
pgtune gives you a good recommendation on your postgresql.conf setting based on your hardware.
If you want to test out your application, you can test using jMeter or any other load testing tool to see how your application will perform under certain load.
Some good resources to read on the topic stack overflow answer, postgres wiki
I am learning NodeJS by building a JWT server. Basically I want to authorize users against credentials in a PostgreSQL database. I am considering node-postgres, passport, pg to connect with PostgreSQL but I have not found anyway to store my connection values encrypted. Ideally I would store them in a properties file so I can change them per environment.
Most examples I see do something like:
var pg = require('pg');
var conString = "postgres://YourUserName:YourPassword#localhost:5432/YourDatabase";
Can someone help show me how to encrypt and use my credentials so I don't have to hard code the plain values in my source?
There seem to exist npm packages for this already. See https://www.npmjs.com/package/secure-conf. Seems to fulfill your needs.
Please note, that you should also secure your Connection to the DB using SSL. See SSL for PostgreSQL connection nodejs for a Solution.
This should help.
if you use sequelize to connect postgres
const sequelize = new Sequelize("DB", usrname, password, {
host: "/var/run/postgresql",
dialect: "postgres",
});
NB: get the host string from your pgsl db might be different //