Set postgres database url on travis-ci for testing node.js application - node.js

I have written some tests for my node.js application and the tests are running locally using a Postgresql test database.
When I run my test script, npm run test, the environment is set to test and when this happens, the database connection string is set for the test database and my queries in the application are now done on the test database. Like so:
let connectionString;
if (process.env.NODE_ENV === 'test') {
connectionString =`postgresql://${process.env.DB_USER}:${process.env.DB_PASS}#${process.env.DB_HOST}:${process.env.DB_PORT}/${process.env.TEST_DB_NAME}`;
} else {
connectionString =`postgresql://${process.env.DB_USER}:${process.env.DB_PASS}#${process.env.DB_HOST}:${process.env.DB_PORT}/${process.env.DB_NAME}`;
}
This way my tests are being run on the test database.
On travis however, I know I am going to need to further configure its own database. On the travisCI docs, I read about how I could set up a PostgreSQL database here, but this doesn't help me because how do I get the full database URL as above? On travisCI, what am I to use as my database hostname or port and how do I set this value inside my code?
How do I set the database connection string and access it in my code?
Thanks for any ideas.

Found this while searching of how to do the exact same thing, and this didn't answer my question, so in case anyone else comes across it, I think I have solved it.
zerosand1s kind of answered it, but there seems to be nowhere online which actually says what the hostname or port should be (maybe I'm being dumb and this is obvious but :shrug:)
As per the psql docs:
Defaults to the value of the PGPORT environment variable or, if not set, to the port specified at compile time, usually 5432.
So I guessed port would be 5432 (you could probably just use PGPORT as your variable).
Also elsewhere on travis, it says local host will bind to 127.0.0.1 for other databases (eg. mongo) so I took a guess on that too.
Travis docs tell us the user as postgres and password is blank.
I was using the whole string to connect, where as you split it up, so as such I set my entire connection string as (you can extract each component):
postgres://postgres#127.0.0.1:5432/testing_db
I did this on the travis dashboard settings.
Amazingly all of this worked :tada:
Hope this helps.

As per docs, you can create a database using before_scriptlike so
before_script:
- psql -c "CREATE DATABASE testing_db;" -U postgres
then you can add your database credentials (along with database name and port etc) to travis environment variables like so
travis encrypt DB_USER=TEST_DB_USER --add env.matrix
You can find more on travis environment variables here. You can also add environment variables on travis dashboard under your repository settings.

Related

How i can restoremy database psql heroku to other server?

How i can backup data my database psql on Heroku to my local computer and i want to move to another sever. can heroku backup a database my server to my local computer?
If you app needs some sql database, you should have access to the connection parameters like the url, user and password. There is no chance for you app to connect to a sql database without these parameters
Reviewing the teledrive docs, I found the variable which contains that values:
DATABASE_URL
format: postgresql://[user]:[password]#[host]:[port][/dbname][?paramspec]
Is similar to java and other languages: https://devcenter.heroku.com/articles/connecting-to-relational-databases-on-heroku-with-java#using-the-jdbc_database_url-in-a-spring-boot-app
You can get these values usually in the environment variables configuration in heroku
Something like this:
Your duty is to find these values.
import & export using some IDE
If you have the database connection parameters and the database has public access, you could use any database ide . I recommend you the dbeaver because is free and powerful. You could use it to connect to any database in the galaxy.
Here are the steps to export and restore a database.
https://community.pyramidanalytics.com/t/h7hk07w/how-to-backup-and-restore-a-postgresql-database-via-dbeaver
Basically you should:
connect to your heroku postgress,
export the data
connect to another postgres
import the data
import & export using the shell
This is more fast than an IDE. Check this to understand it: https://simplebackups.com/blog/postgresql-pgdump-and-pgrestore-guide-examples/

Unable to properly define MongoDB connection string as an Travis CI environmental variable

I'm working on my own project using Node.js and MongoDB as a database (using Mongoose).
When I developed it locally, I defined my environmental variables in a .env file and used 'env-cmd'.
Now I'm trying to create a testing workflow using Travis-CI without using a .env file so I defined my environmental variables on the repository setting but I keep getting an error that the connection string is invalid when trying to build:
It is worth mentioning that:
I escaped every special character when defining the env variable of the connection string.
I tried passing both process.env.MONGODB_URL with and without the ES6 backticks format to the mongoose.connect() method.
The connection string is valid (I tried running my test suites locally with the .env file and it worked)
The connection string is to the MongoDB Atlas database, not a local one.
I'm running the tests from as a docker container.
I can't figure out what is the problem, and if any additional information is needed to tackle this problem I will happily provide.
Thanks in advance!

Configuring Azure PostgreSQL in Gitlab EE

I am searching for some help in how to configure Azure PostgreSQL DB in a Docker Swarm based Gitlab instance.
Initially, I followed the documentation in https://docs.gitlab.com/13.6/ee/administration/postgresql/external.html. Yet I came to find out that the default provided user is in the form of username, whereas Azure requires it to be in the form of username#hostname. I tried passing the username in the gitlab.rb file (gitlab_rails['db_username'] = 'username#hostname') but it still failed, even after replacing the # with the %40 as URI encoded.
After some extensive searching, I found this documentation - https://docs.gitlab.com/13.6/ee/administration/environment_variables.html, which suggests using the DATABASE_URL environment variable to set the full connection string in the form postgresql://username:password#hostname:port/dbname, which I did and it did solve the issue for Gitlab itself communicating with Azure PostgreSQL (in this case I replaced the username with username%40hostname, according to Azure requirements).
Allas, the success was short lived since then I came to find out that neither Puma and Sidekiq can connect to the database, always throwing the following error:
==> /var/log/gitlab/sidekiq/current <==
could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
After some searching, I found that gitlab-ctl is generating the following file when starting the Gitlab instance:
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.
production:
adapter: postgresql
encoding: unicode
collation:
database: <database>
username: "<username>"
password:
host: "/var/opt/gitlab/postgresql"
port: 5432
socket:
sslmode:
sslcompression: 0
sslrootcert:
sslca:
load_balancing: {"hosts":[]}
prepared_statements: false
statement_limit: 1000
connect_timeout:
variables:
statement_timeout:
(database and username where removed)
Pretty much it ignores the DATABASE_URL env variable and assumes the now non-existing configuration parameters in gitlab.rb.
So, right now, I'm a bit out of options and was wondering if anyone has had a similar issue and, if so, how where you able to overcome this.
Any help is appreciated.
Thanks in advance.
TL/DR: Pass the username#hostname string directly into the gitlab_rails['db_username'] in double quotes. The documentation for connecting to an Azure PostgreSQL in the official Gitlab page is not correct.
So, after some searching and going deep into the Gitlab configuration, I came to find out that the issue is very specific and related with the usage of docker secrets.
In my gitlab.rb configuration file, in the database configuration part, I'm using the following:
### GitLab database settings
###! Docs: https://docs.gitlab.com/omnibus/settings/database.html
###! **Only needed if you use an external database.**
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = File.read('/run/secrets/postgresql_database')
gitlab_rails['db_username'] = File.read('/run/secrets/postgresql_user')
gitlab_rails['db_password'] = File.read('/run/secrets/postgresql_password')
gitlab_rails['db_host'] = File.read('/run/secrets/postgresql_host')
gitlab_rails['db_port'] = File.read('/run/secrets/postgresql_port')
gitlab_rails['db_sslmode'] = 'require'
Now, this exact configuration was used previously for testing purposes and worked (but without the usage of Azure PostgreSQL database). And I'm passing the correct secrets to docker and I've confirmed that the secrets in fact, do exist.
(Sidenote: Also, I've established that Gitlab uses the method ActiveRecord::Base.establish_connection from the Ruby ActiveRecord::Base library in order to connect to the database)
Yet, when using the username#hostname configuration for the user and passing that into the postgresql_user secret, suddenly the ActiveRecord::Base.establish_connection method assumes that the #hostname is the actual hostname to where I want to connect to. And I've confirmed that the secret is being generated correctly inside the docker container
Now, it gets even stranger because if I pass the username#hostname string directly to the gitlab.rb file - gitlab_rails['db_username'] parameter - in double quotes, it suddenly starts connecting without complaining.
So, in short, if using an Azure PostgreSQL database for a dockerized Gitlab instance and using secrets to pass the configuration to the gitlab.rb file, don't pass the username#hostame through a secret, but put it directly in the gitlab.rb file.
I don't know if this is a specific issue of Ruby or of Gitlab (I'm not a Ruby developer), but I did try converting the File.read output to a String, to a symbol, used the File.open('filepath', &:readline) and other shenanigans, but nothing worked. So, if anyone out there would care to add their reason for this, please feel free to do so.
Also, the tutorial provided by Azure - https://learn.microsoft.com/pt-pt/azure/postgresql/connect-ruby - doesn't work with Gitlab, since it complains about the %40.
Hope this can help anyone out there.

Node-Red mongodb3 connect DB using URL from environment variable

I'm running Node-Red embedded in Express application. Also using 'dotenv' to load environment variables.
For storage using MongoDB with 'node-red-contrib-mongodb3'.
Everything works as expected. But, I have different environments and different MongoDB for each environments.
I want to connect to MongoDB from configuration (.env file or environment file).
Something like, in MongoDB config node URL input box golbal.get('env').MONGODB_DEV_URL or msg.MONGODB_URL
Tried looking for an option in the documentation of 'mongodb3' and google, still no luck. Any help or direction will be appreciated.
From the Node-RED docs
Any node property can be set with an environment variable by setting
its value to a string of the form ${ENV_VAR}. When the runtime loads
the flows, it will substitute the value of that environment variable
before passing it to the node.
This only works if it replaces the entire property - it cannot be used
to substitute just part of the value. For example, it is not possible
to use CLIENT-${HOST}.

Access remote MongoDB database?

I've mostly worked with PHP/MySQL but I've now been handed a Node.js/MongoDB project on Github.
Having gone through a Mongo tutorial, I feel I understand the concept to a reasonable extent now, but I am still unsure how to do the most basic thing - view the Mongo database associated with the project.
In the config file, I found the following:
module.exports = {
database: {
url: 'localhost:27017/app_name'
},
But seeing how I'm on a remote machine, how do I reach the database? Do I need to ask the previous dev for the DB so I can set it up locally?
Searching the code for the word mongo it only appears in packages.json so that's not a lot of help.
localhost:27017
means the DB is in the local machine in which your are developing your app.
In your case, you have MONGO DB installed in your local machine and run the project.
Otherwise if you have a centralized DB then you have to configure that IP here as follows:
Also configure your mongodb.config of your remote DB to accept the connection from your local machine by changing the "BIND IP"
module.exports = {
database: {
url: 'yourRemoteIP:27017/app_name'
},
Use a GUI tool, i would recommend MongoChef. All you need to do is when you start the server, just open this GUI and connect to DB. GAME ON!! You will be able to see your collections and you are ready to go. You can also run mongo query direct on the shell. It is a helpful tool for playing with your local DB.
PS I am considering you want to see your local DB.
You can access your DB through terminal as well, all upto you.

Resources