I need to update a domain name in the static file when my app promoted to production. I've added this line to my Procfile:
release: ./update-domain.sh
Here is the content:
if [ -n "$DOMAIN" ]; then
sed -i 's#$DOMAIN#'"$DOMAIN"'#' ./assets/config.json
fi
But it seems like Heroku doesn't have access to the env variables from bash. Is that true?
I'm not sure if that is related, but in Heroku Exec documentation I found this:
The SSH session created by Heroku Exec will not have the config vars set as environment variables (i.e., env in a session will not list config vars set by heroku config:set).
Thank you.
Related
I'm trying to run some commands on my NodeJS app that need to be run via SSH (Sequelize seeding for instance), however when I do so, I noticed that the expected env vars were missing.
If I run eb printenv on my local machine I see the expected environment variables that were set in my EB Dashboard
If I SSH in, and run printenv, all of those variables I expect are missing.
So what happens, is when I run my seeds, I get an error:
node_modules/.bin/sequelize db:seed:all
ERROR: connect ECONNREFUSED 127.0.0.1:3306
I noticed that the port was wrong, it should be 5432. I checked to see if my environment variables were set with printenv and they are not there. This leads me to suspect that the proper env variables are not loaded in my ssh session, and NodeJS is falling back to the default values I provided in my config.
I found some solutions online, like running the following to load the env variables:
/opt/python/current/env
But the python directory doesn't exist. All that's inside /opt/ is elasticbeanstalk and aws directories.
I had some success so I could at least see that the env variables exist somewhere on the server by running the following:
sudo /opt/elasticbeanstalk/bin/get-config environment --output YAML
But simply running this command does not fix the problem. All it does is output the expected env variables to the screen. That's something! At least I know they are definitely there! But the env variables are still not there when I run printenv
So how do I fix this problem? Sequelize and NodeJS are clearly not seeing the env variables either, and it's trying to use the fallback default values that are set in my config file.
I know my answer is late, but I had the same problem and after some attempts with bash script I found a way to store it in your env vars.
you can simply run the following command:
export env=`/opt/elasticbeanstalk/bin/get-config environment -k <your-variable-name>`
now you will be able to easily access this variable:
echo $your-variable-name
afterward, you can utilize the env var to do what ever you like. in my case, I use it to decide which version of my code to build in a file called build-script.sh and its content is as follows:
# get env variable to know in which environment this code is running in
export env=`/opt/elasticbeanstalk/bin/get-config environment -k environment`
# building the code based on the current environment
if [ $env = "production" ]
then
echo "building for production"
npm --prefix /var/app/current run build-prod
else
echo "building for non production"
npm --prefix /var/app/current run build-prod
fi
hope this helps anyone facing the same issue 🤟🏻
I'm using NPM's scripts.postinstall to build our application using GulpJS when we push it to Heroku:
"postinstall": "./node_modules/.bin/gulp build --require coffee-script/register"
I want to be able to set an environment variable at this time with the value of the commit hash. This will then be referenced in files as a cachebuster, e.g.:
<script type="text/javascript" src="/index.{{ process.env.commit }}.js"></script>
I can use git-rev to easily get this value asynchronously but I want it stored on an environment level. How can I set a custom Node environment variable?
If I'm going about this the wrong way and there's a better solution please let me know (I can only find methods to set NODE_ENV and not the process.env object itself).
You cannot do this via a postinstall script: the heroku command won't be available in the build environment. You can create your own lightweight deployment script that runs the command on your machine when a successful build occurs however:
#!/bin/sh
git push heroku master && heroku config:set COMMIT=$(git rev-parse --short HEAD)
To note, I got confirmation of this from Heroku support directly. This was their response:
There isn't an installation of heroku toolbelt on the slug compiler. If you need to set an environment variable as part of the push then you'll have to do it on your machine rather than the Heroku compiler.
Solution I've found is to use the heroku config system to set custom environment variables:
git = require "git-rev"
exec = require "exec-sync"
gulp.task "commit", ->
git.long (hash) ->
exec "heroku config:set COMMIT=#{ hash }"
Needs further testing but for now it works (also, because the command is run on the server there's no need to use heroku config:push).
You can now add this metadata with this labs feature from heroku !
heroku labs:enable runtime-dyno-metadata -a <app name>
Other than PORT, MONGO_URL, where can we find the documentation for all the environmental variables that's exposed by Meteor?
In the Meteor docs, I can only find this line
PORT=3000 MONGO_URL=mongodb://localhost:27017/myapp node bundle/main.js
Is there a variable that combines all the JS files and minifies them?
I've been using grasp to answer your question. Here is the command line I used:
grasp -reo --no-line-number --no-color --no-filename process.env.__ . 2> /dev/null | sort | uniq | cut -c 17-
And here is the result on the devel branch:
_
ADMIN_APP
APP_CONFIG
APPDATA
AUTOUPDATE_VERSION
BIND_IP
COMP_CWORD
COMP_LINE
COMP_POINT
COMPUTERNAME
ComSpec
DDP_DEFAULT_CONNECTION_URL
DEBUG
DEBUG_MIME
DEBUG_NOPT
DESTDIR
DISABLE_WEBSOCKETS
EDITOR
GALAXY
GALAXY_APP
GALAXY_JOB
HOME
HOSTNAME
http_proxy
HTTP_PROXY
https_proxy
HTTPS_PROXY
JOBS
LAST_START
MAIL_URL
MAKE
METEOR_SETTINGS
MONGO_URL
NODE_BINDINGS_ARROW
NODE_BINDINGS_COMPILED_DIR
NODE_DEBUG
NODE_ENV
NODE_NDEBUG
NOPT_DEBUG
npm_config_proxy
path
Path
PATH
PATHEXT
PORT
PREFIX
PROMPT
PS1
PWD
PYTHON
ROOT_URL
ROUTE
SERVER_ID
SHELL
SUDO_GID
SUDO_UID
SystemDrive
TEMP
TMP
TMPDIR
ULTRAWORLD_DDP_ENDPOINT
USE_JSESSIONID
USER
USERDOMAIN
USERNAME
USERPROFILE
VISUAL
windir
Most of thoses environment variables are not documented yet.
About minifing all js and css files, you can use the --production parameter:
meteor run --production
To get the documentation use the --help parameter:
meteor run --help
The easiest way to discover what exists "under the covers" of any open source project is to look at its source code. A simple grep "process.env" * at the top of the meteor source tree will show you every instance of Meteor's use of the process environment.
Alternately, you can visit Meteor's github and search for 'process.env'. Doing this should display thirty matches.
I have a few apps on Heroku. All of them use GruntJS to build assets and deploy to S3. One app has been working fine for quite some time.
The other apps have a problem where I can not read my config variables from the Gruntfile. When I use Heroku's toolbelt to view my setup, I see:
$ heroku config --app mydevapp --account personal
=== mydevapp Config Vars
AWS_ACCESS_KEY: #########
AWS_BUCKET: #########
AWS_SECRET_KEY: #########
BUILDPACK_URL: https://github.com/ddollar/heroku-buildpack-multi.git
DATABASE_URL: #########
PAPERTRAIL_API_TOKEN: #########
TEST: test
Which is great. However none of these variables are available to me from Grunt. When I console.log(process.env) from the Gruntfile, I see:
{ GEM_HOME: '/tmp/build_e26d1d60-d447-40d8-b09b-02d3758a6027/.gem/ruby/1.9.1',
SHELL: '/bin/bash',
SSH_CLIENT: '10.207.46.127 55868 50678',
GROUP: 'production',
DEPLOY: 'production',
STACK: 'cedar',
SHLVL: '3',
HOME: '/app',
CPPPATH: '/tmp/node-node-hP8q/include',
_: '/tmp/build_e26d1d60-d447-40d8-b09b-02d3758a6027/node_modules/grunt-cli/bin/grunt'
}
There are some other vars in there, but I'm not sure what is safe to show. I don't see ANY of my config vars listed.
I have no idea what the difference is between my working app, and the two apps that don't have config vars in the process.env variable.
I've read that using Grunt in this manner isn't really the best idea, but it is what we have setup. Of course that could change if need be.
Any ideas? Is there anything I need to clarify?
Heroku does not expose the config variables to the build stage by default. It you want this, you'll have to enable the user-env-compile lab by issuing:
heroku labs:enable user-env-compile -a myapp
Docs: https://devcenter.heroku.com/articles/labs-user-env-compile
2017 Update:
Its seems you should use the ENV_DIR argument.
According to doc:
ENV_DIR is a directory that contains a file for each of the
application’s configuration variables. Config vars are made available
as environment variables during execution of commands specified in the
Procfile, as well as when running one-off processes.
Check the bin/compile section for further details
Here is a snippet to extract the config vars in the build phase:
export_env_dir() {
env_dir=$1
whitelist_regex=${2:-''}
blacklist_regex=${3:-'^(PATH|GIT_DIR|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH)$'}
if [ -d "$env_dir" ]; then
for e in $(ls $env_dir); do
echo "$e" | grep -E "$whitelist_regex" | grep -qvE "$blacklist_regex" &&
export "$e=$(cat $env_dir/$e)"
:
done
fi
}
For me its seems an overkill - happy to hear your thoughts.
I am trying to deploy my app to Heroku however I rely on using some private git repos as modules. I do this for code reuse between projects, e.g. I have a custom logger I use in multiple apps.
"logger":"git+ssh://git#bitbucket.org..............#master"
The problem is Heroku obviously does not have ssh access to this code. I can't find anything on this problem. Ideally Heroku have a public key I can can just add to the modules.
Basic auth
GitHub has support for basic auth:
"dependencies" : {
"my-module" : "git+https://my_username:my_password#github.com/my_github_account/my_repo.git"
}
As does BitBucket:
"dependencies" : {
"my-module": "git+https://my_username:my_password#bitbucket.org/my_bitbucket_account/my_repo.git"
}
But having plain passwords in your package.json is probably not desired.
Personal access tokens (GitHub)
To make this answer more up-to-date, I would now suggest using a personal access token on GitHub instead of username/password combo.
You should now use:
"dependencies" : {
"my-module" : "git+https://<username>:<token>#github.com/my_github_account/my_repo.git"
}
For Github you can generate a new token here:
https://github.com/settings/tokens
App passwords (Bitbucket)
App passwords are primarily intended as a way to provide compatibility with apps that don't support two-factor authentication, and you can use them for this purpose as well. First, create an app password, then specify your dependency like this:
"dependencies" : {
"my-module": "git+https://<username>:<app-password>#bitbucket.org/my_bitbucket_account/my_repo.git"
}
[Deprecated] API key for teams (Bitbucket)
For BitBucket you can generate an API Key on the Manage Team page and then use this URL:
"dependencies" : {
"my-module" : "git+https://<teamname>:<api-key>#bitbucket.org/team_name/repo_name.git"
}
Update 2016-03-26
The method described no longer works if you are using npm3, since npm3 fetches all modules described in package.json before running the preinstall script. This has been confirmed as a bug.
The official node.js Heroku buildpack now includes heroku-prebuild and heroku-postbuild, which will be run before and after npm install respectively. You should use these scripts instead of preinstall and postinstall in all cases, to support both npm2 and npm3.
In other words, your package.json should resemble:
"scripts": {
"heroku-prebuild": "bash preinstall.sh",
"heroku-postbuild": "bash postinstall.sh"
}
I've come up with an alternative to Michael's answer, retaining the (IMO) favourable requirement of keeping your credentials out of source control, whilst not requiring a custom buildpack. This was borne out of frustration that the buildpack linked by Michael is rather out of date.
The solution is to setup and tear down the SSH environment in npm's preinstall and postinstall scripts, instead of in the buildpack.
Follow these instructions:
Create two scripts in your repo, let's call them preinstall.sh and postinstall.sh.
Make them executable (chmod +x *.sh).
Add the following to preinstall.sh:
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod 400 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host github.com\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
fi
Add the following to postinstall.sh:
#!/bin/bash
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Cleaning up SSH config" >&1
echo "" >&1
# Now that npm has finished running, we shouldn't need the ssh key/config anymore.
# Remove the files that we created.
rm -f ~/.ssh/config
rm -f ~/.ssh/deploy_key
# Clear that sensitive key data from the environment
export GIT_SSH_KEY=0
fi
Add the following to your package.json:
"scripts": {
"preinstall": "bash preinstall.sh",
"postinstall": "bash postinstall.sh"
}
Generate a private/public key pair using ssh-agent.
Add the public key as a deploy key on Github.
Create a base64 encoded version of your private key, and set it as the Heroku config var GIT_SSH_KEY.
Commit and push your app to Github.
When Heroku builds your app, before npm installs your dependencies, the preinstall.sh script is run. This creates a private key file from the decoded contents of the GIT_SSH_KEY environment variable, and creates an SSH config file to tell SSH to use this file when connecting to github.com. (If you are connecting to Bitbucket instead, then update the Host entry in preinstall.sh to bitbucket.org). npm then installs the modules using this SSH config. After installation, the private key is removed and the config is wiped.
This allows Heroku to pull down your private modules via SSH, while keeping the private key out of the codebase. If your private key becomes compromised, since it is just one half of a deploy key, you can revoke the public key in GitHub and regenerate the keypair.
As an aside, since GitHub deploy keys have read/write permissions, if you are hosting the module in a GitHub organization, you can instead create a read-only team and assign a 'deploy' user to it. The deploy user can then be configured with the public half of the keypair. This adds an extra layer of security to your module.
It's a REALLY bad idea to have plain text passwords in your git repo, using an access token is better, but you will still want to be super careful.
"my_module": "git+https://ACCESS_TOKEN:x-oauth-basic#github.com/me/my_module.git"
I created a custom nodeJS buildpack that will allow you to specify an SSH key that is registered with ssh-agent and used by npm when dynos are first setup. It seamlessly allows you to specify your module as an ssh url in your package.json like shown:
"private_module": "git+ssh://git#github.com:me/my_module.git"
To setup your app to use your private key:
Generate a key: ssh-keygen -t rsa -C "your_email#example.com" (Enter no passphrase. The buildpack does not support keys with passphrases)
Add the public key to github: pbcopy < ~/.ssh/id_rsa.pub (in OS X) and paste the results into the github admin
Add the private key to your heroku app's config: cat id_rsa | base64 | pbcopy, then heroku config:set GIT_SSH_KEY=<paste_here> --app your-app-name
Setup your app to use the buildpack as described in the heroku nodeJS buildpack README included in the project. In summary the simplest way is to set a special config value with heroku config:set to the github url of the repository containing the desired buildpack. I'd recommend forking my version and linking to your own github fork, as I'm not promising to not change my buildpack.
My custom buildpack can be found here: https://github.com/thirdiron/heroku-buildpack-nodejs and it works for my system. Comments and pull requests are more than welcome.
Based on the answer from #fiznool I created a buildpack to solve this problem using a custom ssh key stored as an environment variable. As the buildpack is technology agnostic, it can be used to download dependencies using any tool like composer for php, bundler for ruby, npm for javascript, etc: https://github.com/simon0191/custom-ssh-key-buildpack
Add the buildpack to your app:
$ heroku buildpacks:add --index 1 https://github.com/simon0191/custom-ssh-key-buildpack
Generate a new SSH key without passphrase (lets say you named it deploy_key)
Add the public key to your private repository account. For example:
Github: https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
Bitbucket: https://confluence.atlassian.com/bitbucket/add-an-ssh-key-to-an-account-302811853.html
Encode the private key as a base64 string and add it as the CUSTOM_SSH_KEY environment variable of the heroku app.
Make a comma separated list of the hosts for which the ssh key should be used and add it as the CUSTOM_SSH_KEY_HOSTS environment variable of the heroku app.
# MacOS
$ heroku config:set CUSTOM_SSH_KEY=$(base64 --input ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
# Ubuntu
$ heroku config:set CUSTOM_SSH_KEY=$(base64 ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
Deploy your app and enjoy :)
I was able to setup resolving of Github private repositories in Heroku build via Personal access tokens.
Generate Github access token here: https://github.com/settings/tokens
Set access token as Heroku config var: heroku config:set GITHUB_TOKEN=<paste_here> --app your-app-name or via Heroku Dashboard
Add heroku-prebuild.sh script:
#!/bin/bash
if [ "$GITHUB_TOKEN" != "" ]; then
echo "Detected GITHUB_TOKEN. Setting git config to use the security token" >&1
git config --global url."https://${GITHUB_TOKEN}#github.com/".insteadOf git#github.com:
fi
add the prebuild script to package.json:
"scripts": {
"heroku-prebuild": "bash heroku-prebuild.sh"
}
For local environment we can also use git config ... or we can add the access token to ~/.netrc file:
machine github.com
login PASTE_GITHUB_USERNAME_HERE
password PASTE_GITHUB_TOKEN_HERE
and installing private github repos should work.
npm install OWNER/REPO --save will appear in package.json as: "REPO": "github:OWNER/REPO"
and resolving private repos in Heroku build should also work.
optionally you can setup a postbuild script to unset the GITHUB_TOKEN.
This answer is good https://stackoverflow.com/a/29677091/6135922, but I changed a little bit preinstall script. Hope this will help someone.
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
echo "Preinstall"
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod o-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host bitbucket.org\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" HostName bitbucket.org\n" \
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
echo "eval `ssh-agent -s`"
eval `ssh-agent -s`
echo "ssh-add -l"
ssh-add -l
echo "ssh-add ~/.ssh/deploy_key"
ssh-add ~/.ssh/deploy_key
# uncomment to check that everything works just fine
# ssh -v git#bitbucket.org
fi
You can use in package.json private repository with authentication example below:
https://usernamegit:passwordgit#github.com/reponame/web/tarball/branchname
In short it is not possible. The best solution to this problem I came up with is to use the new git subtree's. At the time of writing they are not in the official git source and so needs to be installed manual but they will be included in v1.7.11. At the moment it is available on homebrew and apt-get. it is then a case of doing
git subtree add -P /node_modules/someprivatemodue git#github.......someprivatemodule {master|tag|commit}
this bulks out the repo size but an update is easy by doing the command above with gitsubtree pull.
I have done this before with modules from github. Npm currently accepts the name of the package or a link to a tar.gz file which contains the package.
For example if you want to use express.js directly from Github (grab the link via the download section) you could do:
"dependencies" : {
"express" : "https://github.com/visionmedia/express/tarball/2.5.9"
}
So you need to find a way to access you repository as a tar.gz file via http(s).