'Git: gpg failed to sign the data' in visual studio code - linux

After a fresh Linux install I'm trying to set up my environment and I keep getting the Git: gpg failed to sign the data error upon committing changes locally. I'm using Visual Studio Code, proprietary, not opensource version.
.gitconfig:
[user]
name = djweaver-dev
email = djweaver#djweaver.dev
signingkey = 37A0xxxx...
[core]
excludesfile = /home/dweaver/.gitignore_global
[commit]
gpgSign = true
yikes. furthermore I can't find a way to copy the output log nor can I find where that log is so here is a pic:
Steps I have taken so far:
generated new key (RSA 4096) in gnugp
added signing key to global .gitconfig
set "git.enableCommitSigning": true in Visual Studio Code settings
cloned my repo from github
Typically when I commits in the past I would get a dialog box requesting GPG authentication upon commit. I do not get this now, just the error dialog.
UPDATE: Okay now I'm really confused. I restarted vscode (not the first time I've done this in this process) and voilà, it works. Only thing I can think of is maybe I biffed the directory somehow? Either way, it works now.
UPDATE: Oddly, I'm back to this same issue almost a month later after a fresh arch install. I've tried everything that I've been able to find on this site, and nothing works.
I've tried adding export GPG_TTY=$(tty) to .bash_profile, and also .bashrc
Git log:
Looking for git in: git
Using git 2.26.2 from git
> git rev-parse --show-toplevel
> git rev-parse --git-dir
Open repository: /home/dw/dev/website
> git status -z -u
> git symbolic-ref --short HEAD
> git rev-parse master
> git rev-parse --symbolic-full-name master#{u}
> git rev-list --left-right master...refs/remotes/origin/master
> git for-each-ref --format %(refname) %(objectname) --sort -committerdate
> git remote --verbose
Failed to watch ref '/home/dw/dev/website/.git/refs/remotes/origin/master', is most likely packed.
Error: ENOENT: no such file or directory, watch '/home/dw/dev/website/.git/refs/remotes/origin/master'
at FSWatcher.start (internal/fs/watchers.js:165:26)
at Object.watch (fs.js:1270:11)
at Object.t.watch (/usr/lib/code/extensions/git/dist/main.js:1:604919)
at T.updateTransientWatchers (/usr/lib/code/extensions/git/dist/main.js:1:83965)
at e.fire (/usr/lib/code/out/vs/workbench/services/extensions/node/extensionHostProcess.js:46:87)
at e.updateModelState (/usr/lib/code/extensions/git/dist/main.js:1:103179)
> git config --get commit.template
> git check-ignore -v -z --stdin
> git check-ignore -v -z --stdin
> git commit --quiet --allow-empty-message --file - -S
error: gpg failed to sign the data
fatal: failed to write commit object
> git config --get-all user.name
> git config --get-all user.email
Same config as last time, user.name and user.email both match each key I've been trying it with... user.signingkey matches. Not sure where else to go with this one, as I've tried it across newly initialized local repos as well as repos that I've pulled from github both with official MS vscode (AUR) and OSS version, in the vscode terminal emulator as well as gnome terminal with same results so it has to be either a git thing or a gnugp thing.
What I have noticed is that after committing without signing, it will work immediately after: I get prompted for my key passphrase the first time, then it works on subsequent commits until a seemingly random number of minutes later, it just doesn't work anymore and the process has to be repeated.
There were a few macos users posting about having a stalled gpg-agent running in the background and it fixed it for them, however, I am seeing:
[dw#dwLinux website]$ gpg-agent
gpg-agent[2870]: gpg-agent running and available
Whats interesting also is that by doing echo "test" | gpg --clearsign I get the same results: it works for a short period of time, then I can't sign anymore.
UPDATE
Okay so day number 2 of trying to fix this. To rule out the gpg-agent theory as described here I followed the instructions on how to reload gpg-agent using the $ gpg-connect-agent reloadagent /bye command demonstrated on the Arch Linux Wiki
This had no effect
So being that I can reproduce this problem across vscode official, oss code, and vscodium, as well as bash, I thought maybe this was a permissions related issue, as so many problems with linux typically are. I added my user to all kinds of groups, including root, and this also had no effect so I think I can safely rule out the following:
VS Code
GnuGP
gpg-agent
Linux permissions
So my next focus was the config files themselves, but as has been stated before the credentials match the key in .gitconfig and my .bash_profile has been correctly configured with export GPG_TTY=$(tty).
An interesting note on this from the official GnuPG docs shows a syntax discrepency between their way, and the way you are instructed to append this to .bash_profile on the GitHub docs here
From GnuPG: "The far most common reason for this is that the environment variable GPG_TTY has not been set correctly. Make sure that it has been set to a real tty device and not just to ‘/dev/tty’; i.e. ‘GPG_TTY=tty’ is plainly wrong; what you want is ‘GPG_TTY=tty’ — note the back ticks. Also make sure that this environment variable gets exported, that is you should follow up the setting with an ‘export GPG_TTY’"
As I understood $(whatever) in bash was to execute a command, but for safe measure I've appended .bash_profile using both ways and neither solved the issue.
One last thing
In this post the user talks about gpg-agent authentication not being available when daemonized and gpg access is being initiated by another application (such as an IDE like VSCode), which explains how I could temporarily sign commits after committing a random file or doing echo "test" | gpg --clearsign and being authenticated... but alas like most other 'solutions' to this topic, they reveal that all they had to do in the end was add export GPG_TTY=$(tty) to their .bash_profile, which I have already tried.
Where to go from here?
I still can't explain why it worked on my previous install, and frankly, not a whole lot has changed afaik. I typically do fresh installs often and keep a pretty minimal arch linux build with lts kernel each time w/base-devel and nodejs/python/git/vscode/firefox/discord is pretty much my entire workflow. I'm all out of ideas.

first make sure to add
export GPG_TTY=$(tty)
in your .bashrc
Apparently VSCode doesn't ask for the passphrase and that's why it gives an error.
I don't know the reason.
My personal solution do a console commit first or run the following line
echo "test" | gpg --clearsign
Edit
In order to avoid typing the passphrase on every commit, you can make GPG remember it for 8 hours or until the next reboot:
mkdir -p ~/.gnupg
echo "default-cache-ttl 28800" >> ~/.gnupg/gpg-agent.conf
GitHub Guide

Maybe git cannot find gpg? That was my problem with working with VSCode and using Remote-Containers to create development containers. Try running this in the Terminal within VSCode (in the container)
git config --global --unset gpg.program
git config --global --add gpg.program /usr/bin/gpg
or wherever your gpg is located. You can find out by typing
which gpg
If that works then you can put it in your Dockerfile for your development container.

I had the same issue a few days ago while using VS Code with WSL. The problem is that VS Code doesn't load the .profile file (and all the environment variables in it) correctly (try to run this command but it doesn't get the correct result: echo $GPG_TTY). Fortunately, setting the "-l" option for shell args in VS Code preferences worked for me. This ensures that the .profile (or .zprofile) file is successfully loaded.
I added these lines to settings.json:
"terminal.integrated.shellArgs.linux": [
"-l"
]
Make sure to add export GPG_TTY=$(tty) in your .profile file and restart your terminal and VS Code.
Update: Since VSCode is deprecating the shellArgs oprion, use
the following snippet as an alternative.
"terminal.integrated.profiles.linux": {
"bash": {
"path": "bash",
"args": ["-l"],
"icon": "terminal-bash"
},
"zsh": {
"path": "zsh",
"args": ["-l"],
},
"fish": {
"path": "fish",
"args": ["-l"],
},
"tmux": {
"path": "tmux",
"args": ["-l"],
"icon": "terminal-tmux"
},
"pwsh": {
"path": "pwsh",
"args": ["-l"],
"icon": "terminal-powershell"
}
},
"terminal.integrated.defaultProfile.linux": "bash"
-l option is added to all terminal profiles above,
delete unused profiles and set your default profile at your wish.

I have same issue, and I have resolved it.
Background
macOS
GPG Suite to generate GPG key
pinentry-mac
How I solve this problem
I saw this answer, and followed it.
Get keys
gpg2 --list-keys
Result
/Users/xxuser/.gnupg/pubring.kbx
---------------------------------
pub dsa2048 2010-08-19 [SC] [expires: 2024-05-11]
85E38F69046BSDFB07B76D78F0500D026C4
uid [ unknown] GPGTools Team <team#gpgtools.org>
uid [ unknown] [jpeg image of size 6329]
sub rsa4096 2014-04-08 [S] [expires: 2024-05-11]
sub rsa4096 2020-05-11 [E] [expires: 2024-05-11]
pub rsa4096 2020-05-04 [SC] [expires: 2024-05-03]
B97E9964ACAD1928300D37CC8A9E3745558E41AF
uid [ unknown] GPGTools Support <support#gpgtools.org>
sub rsa4096 2020-05-04 [E] [expires: 2024-05-03]
pub rsa4096 2021-07-29 [SC] [expires: 2025-07-29]
926E268C01892E8A2FCCD2A101CEB6267272A9A5
uid [ultimate] xxuser <x#xxgolo.com>
sub rsa4096 2021-07-29 [E] [expires: 2025-07-29]
Since x#xxgolo.com is the email that I create secret for, 926E268C01892E8A2FCCD2A101CEB6267272A9A5 is the key code I need.
Let git user this key.
git config --global user.signingkey 926E268C01892E8A2FCCD2A101CEB6267272A9A5
Now it should work.
git commit -S -m "This is a signed commit"
note If you need it to work with Github, you need to add your public GPG key to Github, following this Guide.

Make sure echo "test" | gpg --clearsign runs successfully first before trying the below.
Very helpful to check what git commit is doing under the hood. Run the following commit with GIT_TRACE=1 as follow
GIT_TRACE=1 git commit -S -m "MESSAGE"
This will show what user name, email and key does git uses when committing.
In my case, I found that git was picking up the wrong user's and key details for signing the commit. I mainly intended to use the local config of the repo rather than the global and adding the following to the local git config (located at "REPO_PATH/.git/config") got signing the commit to work both in Terminal and VSCode
[user]
name = USER NAME
email = USER EMAIL
signingKey = SIGNING KEY
It can also be set with the following:
git config --local user.name "USER NAME"
git config --local user.email "USER EMAIL"
git config --local user.signingkey "USIGNING KEY"

I'm not sure if this is too late, but... I did find an immediate solution.
To see what user.name and user.email you have, run:
git config -l
You may notice two entries for user.name. You may have made the same mistake as me! I put my actual name in there instead of GitHub username, and there ended up being two entries of user.name! I just changed the global user.name back to my github username, like so...
git config --global user.name "ghusername"
Next, git commit, and it should work:
git commit -m "<YOUR MESSAGE>"
Let me know if this works for you, I want to know if it's the same problem.

Related

unsure how to connect rstudio with gitlab?

I'm trying to set up RStudio with gitlab. I have never used gitlab before, so i'm following set up instructions found online.
I created my gitlab account and project without any issues. I then followed the instructions found here for connecting RStudio with gitlab:
https://www.enricodata.com/post/setup-gitlab-with-rstudio/
The instructions say:
"In Rstudio go to Terminal and paste the following:
git config - - global user.email “YOUR EMAIL”
git config - - global user.name “YOUR NAME”
git config remote.origin.url git#gitlab.com:Allerious/code project.git"
When i execute the first two lines of code it outputs this in the console:
I'm unsure if this means it has done what it's supposed to have done.
Note: I did change “YOUR EMAIL” and “YOUR NAME” when running the code.
When i execute the third line of code it outputs this:
I'm thinking i should be changing the directory path to something else, but i'm unsure how to do this and what it should be changed to.
Here's another screen shot:
Maybe i should be changing the directory to one of these?

How to create a GitLab merge request via command line

We are working on integrating GitLab (enterprise edition) in our tooling, but one thing that is still on our wishlist is to create a merge request in GitLab via a command line (or batchfile or similar, for that matter). We would like to integrate this in our tooling. Searching here and on the web lead me to believe that this is not possible with native GitLab, but that we need additional tooling for that.
Am I correct? And what kind of tooling would I want to use for this?
As of GitLab 11.10, if you're using git 2.10 or newer, you can automatically create a merge request from the command line like this:
git push -o merge_request.create
More information can be found in the docs.
It's not natively supported, but it's not hard to throw together. The gitlab API has support for opening MR: https://github.com/gitlabhq/gitlabhq/blob/master/doc/api/merge_requests.md#create-mr
You can use following utility.
Disclosure : I developed it.
https://github.com/vishwanatharondekar/gitlab-cli
You can create merge request using this.
Some of the features it has are.
Base branch is optional. If base branch is not provided. Current branch is used as base branch.
target branch is optional. If target branch is not provided, default branch of the repo in gitlab will be used.
Created pull request page will be opened automatically after successful creation.
If title is not supported with -m option value. It will be taken from in place editor opened. First line is taken as title.
In the editor opened third line onwards takes as description.
Comma separated list of labels can be provided with its option.
Supports CI.
Repository specific configs can be given.
squash option is available.
remove source branch option is available.
If you push your branch before this command (git push -o merge_request.create) it will not work. Git will response with Everything up-to-date and merge request will not be created (gitlab 12.3).
When I tried to remove my branch from a server (do not remove your local branch!!!) then it worked for me in this form.
git push --set-upstream origin your-branch-name -o merge_request.create
In addition to answering of #AhmadSherif, You can use merge_request.target=<branch_name> for declaring the target branch.
sample usage:
git push -o merge_request.create -o merge_request.target=develop origin feature
Simple This:
According to the Gitlab documents, you can define an alias for this command to simpler usage.
git config --global alias.mwps "push -o merge_request.create -o
merge_request.target=master -o merge_request.merge_when_pipeline_succeeds"
I made a shell function which opens up the GitLab MR web page with desired parameters.
Based on the directory with the git repo you are currently in, it:
Finds the correct URL to your repo.
Sets the source branch to the branch you're currently on.
As a optional first argument you can provide the target branch. Otherwise, GitLab defaults to your default branch, which is typically master.
gmr() {
# A quick way to open a GitLab merge request URL for the current git branch
# you're on. The optional first argument is the target branch.
repo_path=$(git remote get-url origin --push | sed 's/^.*://g' | sed 's/.git$//g')
current_branch=$(git rev-parse --abbrev-ref HEAD)
if [[ -n $1 ]]; then
target_branch="&merge_request[target_branch]=$1"
else
target_branch=""
fi
xdg-open "https://gitlab.com/$repo_path/merge_requests/new?merge_request[source_branch]=$current_branch$target_branch"
}
You can set more default values in the URL, like removing the source branch after merge:
&merge_request[force_remove_source_branch]=true
Or assignee to someone:
&merge_request[assignee_ids][]=12345
Or add a reviewer:
&merge_request[reviewer_ids][]=54321
You can easily find the possible query string parameters by searching the source of the GitLab MR webpage for merge_request[.
As of now, GitLab sadly does not support this, however I recently saw it on their issue tracker. It appears one can expect a 'native tool' in the upcoming months.
GitLab tweeted out about numa08/git-gitlab some time ago, so I guess this would be worth a try.
In our build script we just pop up the browser with the correct URL and let the developer write his comments in the form hit save to create the merge request. You get this url with the correct parameters by creating a merge request manually and copying the url of the form.
#!/bin/bash
set -e
set -o pipefail
BRANCH=${2}
....
git push -f origin-gitlab $BRANCH
open "https://gitlab.com/**username**/**project-name**/merge_requests/new?merge_request%5Bsource_branch%5D=$BRANCH&merge_request%5Bsource_project_id%5D=99999&merge_request%5Btarget_branch%5D=master&merge_request%5Btarget_project_id%5D=99999"
You can write a local git alias to open a Gitlab Merge Request creation page in the default browser for the currently checked-out branch.
[alias]
lab = "!start https://gitlab.com/path/to/repo/-/merge_requests/new?merge_request%5Bsource_branch%5D=\"$(git rev-parse --abbrev-ref HEAD)\""
(this is a very simple alias for windows; I guess there are equivalent replacements for "start" on linux and fancier aliases that work with github and bitbucket too)
As well as being able to immediately see&modify the details of the MR, the advantage of this over using the merge_request.create push option is that you don't need your local branch to be behind the remote for it to work.
You might additionally want to store the alias in the repo itself.
I use https://github.com/mdsb100/cli-gitlab
I am creating the MR from inside of a gitlab CI docker container based on alpine linux, so I include the install command in before-script (that could also be included in your image). All commands in the following .gitlab-ci.yml file, are also relevant for normal command line usage (as long as you have the cli-gitlab npm installed).
variables:
TARGET_BRANCH: 'live'
GITLAB_URL: 'https://your.gitlab.net'
GITLAB_TOKEN: $PRIVATE_TOKEN #created in user profile & added in project settings
before-script:
-apk update && apk add nodejs && npm install cli-gitlab -g
script:
- gitlab url $GITLAB_URL && gitlab token $GITLAB_TOKEN
- 'echo "gitlab addMergeRequest $CI_PROJECT_ID $CI_COMMIT_REF_NAME \"$TARGET_BRANCH\" 13 `date +%Y%m%d%H%M%S`"'
- 'gitlab addMergeRequest $CI_PROJECT_ID $CI_COMMIT_REF_NAME "$TARGET_BRANCH" 13 `date +%Y%m%d%H%M%S` 2> ./mr.json'
- cat ./mr.json
This will echo true if the merge request already exists, and echo the json result of the new MR if it succeeds to create one (also saving to a mr.json file).
Since GitLab 15.7 (Dec. 2022), the GitLab CLI glab is officially integrated to GitLab.
Introducing the GitLab CLI
The command line is one of the most important tools in a software engineer’s toolkit and the majority of their process and work revolve around tools available there. They customize their CLI with styles and extend it through applications to ensure maximum efficiency while performing tasks. The CLI is the backbone of scripts and workflows developers depend on to complete their work.
To support more developers where they’re already working, we’ve adopted the open source project glab, which will form the foundation of GitLab’s native CLI experience.
The GitLab CLI brings GitLab together with Git and your code, with no application or tab switching required.
You can read about our adoption of glab, our partnership with 1Password, and how to contribute to the project in our blog post.
A special thank you to Clement Sam for creating glab and trusting us with its future.
That means you can create a MR with glab mr create:
glab mr create -a username -t "fix annoying bug"

GIT: Can't Push (Strange Config Issue)

I'm on a fresh install of Linux Mint.
I'm getting the following error when trying to push from any repository:
error: Malformed value for push.default: simple
error: Must be one of nothing, matching, tracking or current.
fatal: bad config file line 8 in /home/leng/.gitconfig
fatal: Could not read from remote repository.
This is very odd, because I definitely have a version that supports the simple push behavior.
The output of git --version is git version 1.8.3.2.
The contents of ~/.gitconfig:
[user]
name = My Name
email = MyEmail#website.com
[color]
ui = true
[push]
default = simple
Here's where it gets creepy.
If I change the behavior to matching (or to nothing, tracking, or current, for that matter), then attempt to push, I get the same exact error message. How is that possible? Is it caching the config somehow? I've even tried rebooting. I've even tried purging GIT completely from the system (and deleting ~/.gitconfig) then reinstalling it.
If I delete the [push] section completely from the .gitconfig file (or if I delete the file entirely), then try to push, then I get this:
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:
git config --global push.default matching
To squelch this message and adopt the new behavior now, use:
git config --global push.default simple
See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)
error: Malformed value for push.default: simple
error: Must be one of nothing, matching, tracking or current.
fatal: bad config file line 8 in /home/leng/.gitconfig
fatal: Could not read from remote repository.
...so it appears to be both acknowledging that I haven't chosen a pushing behavior, but then also saying that I've chosen an unsupported behavior. What on earth is going on here?
I even get the error if I delete ~/.gitconfig completely.
Can anyone help me out with this witchcraft?
Thanks!
EDIT:
Here is a .git/config file requested:
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = ssh://{my remote repo}
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
Okay, so I fixed it, but the method is absolute witchcraft.
I tried to isolate the problem by purging GIT, deleting the config file, reinstalling GIT, then creating a local bare repository, then cloning it, then attempting to push from there. Pretty much like this:
sudo apt-get purge git-core
rm -f ~/.gitconfig
sudo apt-get install git-core
cd /git
mkdir foo
cd foo
git init --bare
cd /var/www
git clone /git/foo
cd foo
touch blah.txt
git add -A
git config --global user.name "Name"
git config --global user.email "user#email.com"
git commit -m "Blah"
git push
...same exact error message, no change there. (Still some serious witchcraft.)
Then, I deleted one of my repositories that doesn't have a local origin (it connects to its origin via SSH) and cloned the repository anew after deleting it (with a fresh git clone ssh://... command).
I got an error from the clone command:
remote: Malformed value for push.default: simple
remote: Must be one of nothing, matching, tracking or current.
Ah ha! Now it says remote instead of error. So the remote doesn't support this behavior. (That doesn't explain why the error persists on local-only repositories with local origins, then, though.)
So I then SSH'ed into the remote server and updated the git-core there to the latest version, re-attempted to clone the repository from my local machine, and it worked.
Now, I can finally git push. Insanely, this also fixed it so I can git push from the entirely local /var/www/foo to the also entirely local /git/foo (the local origin bare repository). SSH'ing into this remote server and updating it somehow - WITCHCRAFT - fixed my local machine's error.
Why on earth the entirely local repos care about an entirely different machine's GIT version is... beyond me. How utterly, utterly insane.
I had the same error message on git push.
For me it turned out that the remote user's git was an older version (1.7.2.5),
and I had recently updated the remote ~/.gitconfig to include:
[push]
default = simple
The solution was to remove this setting from the remote's configuration.
Since it seems other people are having this issue, and I found a solution HERE, I thought I'd post the solution that worked for me.
IN SHORT:
The solution I found was at this page. Evidently the best solution is to upgrade to a newer version of Git (if possible). That was not an option for me, however. From a local machine, I typed the following command:
git config -–global push.default upstream
This got rid of the Malformed value for push.default: simple error I had been getting. I'm not entirely sure what upstream does, however.
MY CONTEXT (for comparison): I had an empty (bare) repository on a remote computer, and I had a few repositories on a couple "local" workstations. I pull from the remote repository, do some work, and then push my work to the remote repository. Pushing/pulling was accomplished via SSH. Most of the time, while working on a local machine, pushing/pulling would result in the error described above.
In short, before the fix, I had the following ~/.gitconfig file on the remote machine:
[user]
name = Foo Bar
email = FooBarPerson#email.com
[diff]
external = /Users/foobar/bin/git-diff-cmd.sh
[color]
diff = auto
status = auto
branch = auto
[push]
default = simple
After entering in the above command, my ~/.gitconfig file on the remote machine changed to:
[user]
name = Foo Bar
email = FooBarPerson#email.com
[diff]
external = /Users/foobar/bin/git-diff-cmd.sh
[color]
diff = auto
status = auto
branch = auto
[push]
default = upstream
Version information:
Remote machine (repository location): 1.9.4
My laptop: 1.8.5.2 (Apple Git-48)
Other computer I work on: 1.7.7.4
Here's another site that may be useful to some people:
http://www.lorrin.org/blog/2011/10/03/argumentless-git-pull-and-git-push/comment-page-1/

Mint - Stored SVN password was changed and now unable to commit

I have a copy of a repo on my localhost with a saved username / password for the SVN repo.
The problem is that I changed my svn password (and would like to keep it that way) but every time I try to svn commit, it is asking for my GNOME keyring password (which I enter correctly). This is odd in the first place because I never had it ask me this before.
Then, after entering my password to the keyring, I get this error message:
svn: OPTIONS of 'PATH_TO_CHANGED_FILES': authorization failed: Could not authenticate to server: rejected Basic challenge (REPO_URL)
This is happening on 2 repos that I have but a 3rd one is just fine.
When I disable authentication on the server, everything commits just fine and if I try to update / commit from another server, it also works just fine.
I tried adding the following lines to my ~/.subversion/servers:
store-passwords = no
store-plaintext-passwords = no
And I also tried adding the following lines to my ~/.subversion/config:
store-passwords = no
store-auth-creds = no
But those config file changes do nothing.
Is there a way for my localhost svn to forget the username and passwords I have entered for these repos (they were saved before) so I can get back to everything?
I was able to solve this by deleting the keyring file for MATE. It is a bit of a brute way of doing it but it worked. You can delete the keyring file for MATE with the following command:
rm ~/.config/mate/keyrings/*.keyring
I don't know, how to remove stored data from keyring (old pass for repo), but you can try to replace it.
Use in console any SVN command, which will require authentication with additional options
Global options:
--username ARG : specify a username ARG
--password ARG : specify a password ARG
and test repo-communication in usual way after it
About keyring pass-request:
Check settings in ~/.subversion/config, [auth] section for password-stores =
Check settings in ~/.subversion/servers, [global] section for
store-passwords =
store-plaintext-passwords =

NPM private git module on Heroku

I am trying to deploy my app to Heroku however I rely on using some private git repos as modules. I do this for code reuse between projects, e.g. I have a custom logger I use in multiple apps.
"logger":"git+ssh://git#bitbucket.org..............#master"
The problem is Heroku obviously does not have ssh access to this code. I can't find anything on this problem. Ideally Heroku have a public key I can can just add to the modules.
Basic auth
GitHub has support for basic auth:
"dependencies" : {
"my-module" : "git+https://my_username:my_password#github.com/my_github_account/my_repo.git"
}
As does BitBucket:
"dependencies" : {
"my-module": "git+https://my_username:my_password#bitbucket.org/my_bitbucket_account/my_repo.git"
}
But having plain passwords in your package.json is probably not desired.
Personal access tokens (GitHub)
To make this answer more up-to-date, I would now suggest using a personal access token on GitHub instead of username/password combo.
You should now use:
"dependencies" : {
"my-module" : "git+https://<username>:<token>#github.com/my_github_account/my_repo.git"
}
For Github you can generate a new token here:
https://github.com/settings/tokens
App passwords (Bitbucket)
App passwords are primarily intended as a way to provide compatibility with apps that don't support two-factor authentication, and you can use them for this purpose as well. First, create an app password, then specify your dependency like this:
"dependencies" : {
"my-module": "git+https://<username>:<app-password>#bitbucket.org/my_bitbucket_account/my_repo.git"
}
[Deprecated] API key for teams (Bitbucket)
For BitBucket you can generate an API Key on the Manage Team page and then use this URL:
"dependencies" : {
"my-module" : "git+https://<teamname>:<api-key>#bitbucket.org/team_name/repo_name.git"
}
Update 2016-03-26
The method described no longer works if you are using npm3, since npm3 fetches all modules described in package.json before running the preinstall script. This has been confirmed as a bug.
The official node.js Heroku buildpack now includes heroku-prebuild and heroku-postbuild, which will be run before and after npm install respectively. You should use these scripts instead of preinstall and postinstall in all cases, to support both npm2 and npm3.
In other words, your package.json should resemble:
"scripts": {
"heroku-prebuild": "bash preinstall.sh",
"heroku-postbuild": "bash postinstall.sh"
}
I've come up with an alternative to Michael's answer, retaining the (IMO) favourable requirement of keeping your credentials out of source control, whilst not requiring a custom buildpack. This was borne out of frustration that the buildpack linked by Michael is rather out of date.
The solution is to setup and tear down the SSH environment in npm's preinstall and postinstall scripts, instead of in the buildpack.
Follow these instructions:
Create two scripts in your repo, let's call them preinstall.sh and postinstall.sh.
Make them executable (chmod +x *.sh).
Add the following to preinstall.sh:
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod 400 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host github.com\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
fi
Add the following to postinstall.sh:
#!/bin/bash
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Cleaning up SSH config" >&1
echo "" >&1
# Now that npm has finished running, we shouldn't need the ssh key/config anymore.
# Remove the files that we created.
rm -f ~/.ssh/config
rm -f ~/.ssh/deploy_key
# Clear that sensitive key data from the environment
export GIT_SSH_KEY=0
fi
Add the following to your package.json:
"scripts": {
"preinstall": "bash preinstall.sh",
"postinstall": "bash postinstall.sh"
}
Generate a private/public key pair using ssh-agent.
Add the public key as a deploy key on Github.
Create a base64 encoded version of your private key, and set it as the Heroku config var GIT_SSH_KEY.
Commit and push your app to Github.
When Heroku builds your app, before npm installs your dependencies, the preinstall.sh script is run. This creates a private key file from the decoded contents of the GIT_SSH_KEY environment variable, and creates an SSH config file to tell SSH to use this file when connecting to github.com. (If you are connecting to Bitbucket instead, then update the Host entry in preinstall.sh to bitbucket.org). npm then installs the modules using this SSH config. After installation, the private key is removed and the config is wiped.
This allows Heroku to pull down your private modules via SSH, while keeping the private key out of the codebase. If your private key becomes compromised, since it is just one half of a deploy key, you can revoke the public key in GitHub and regenerate the keypair.
As an aside, since GitHub deploy keys have read/write permissions, if you are hosting the module in a GitHub organization, you can instead create a read-only team and assign a 'deploy' user to it. The deploy user can then be configured with the public half of the keypair. This adds an extra layer of security to your module.
It's a REALLY bad idea to have plain text passwords in your git repo, using an access token is better, but you will still want to be super careful.
"my_module": "git+https://ACCESS_TOKEN:x-oauth-basic#github.com/me/my_module.git"
I created a custom nodeJS buildpack that will allow you to specify an SSH key that is registered with ssh-agent and used by npm when dynos are first setup. It seamlessly allows you to specify your module as an ssh url in your package.json like shown:
"private_module": "git+ssh://git#github.com:me/my_module.git"
To setup your app to use your private key:
Generate a key: ssh-keygen -t rsa -C "your_email#example.com" (Enter no passphrase. The buildpack does not support keys with passphrases)
Add the public key to github: pbcopy < ~/.ssh/id_rsa.pub (in OS X) and paste the results into the github admin
Add the private key to your heroku app's config: cat id_rsa | base64 | pbcopy, then heroku config:set GIT_SSH_KEY=<paste_here> --app your-app-name
Setup your app to use the buildpack as described in the heroku nodeJS buildpack README included in the project. In summary the simplest way is to set a special config value with heroku config:set to the github url of the repository containing the desired buildpack. I'd recommend forking my version and linking to your own github fork, as I'm not promising to not change my buildpack.
My custom buildpack can be found here: https://github.com/thirdiron/heroku-buildpack-nodejs and it works for my system. Comments and pull requests are more than welcome.
Based on the answer from #fiznool I created a buildpack to solve this problem using a custom ssh key stored as an environment variable. As the buildpack is technology agnostic, it can be used to download dependencies using any tool like composer for php, bundler for ruby, npm for javascript, etc: https://github.com/simon0191/custom-ssh-key-buildpack
Add the buildpack to your app:
$ heroku buildpacks:add --index 1 https://github.com/simon0191/custom-ssh-key-buildpack
Generate a new SSH key without passphrase (lets say you named it deploy_key)
Add the public key to your private repository account. For example:
Github: https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
Bitbucket: https://confluence.atlassian.com/bitbucket/add-an-ssh-key-to-an-account-302811853.html
Encode the private key as a base64 string and add it as the CUSTOM_SSH_KEY environment variable of the heroku app.
Make a comma separated list of the hosts for which the ssh key should be used and add it as the CUSTOM_SSH_KEY_HOSTS environment variable of the heroku app.
# MacOS
$ heroku config:set CUSTOM_SSH_KEY=$(base64 --input ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
# Ubuntu
$ heroku config:set CUSTOM_SSH_KEY=$(base64 ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
Deploy your app and enjoy :)
I was able to setup resolving of Github private repositories in Heroku build via Personal access tokens.
Generate Github access token here: https://github.com/settings/tokens
Set access token as Heroku config var: heroku config:set GITHUB_TOKEN=<paste_here> --app your-app-name or via Heroku Dashboard
Add heroku-prebuild.sh script:
#!/bin/bash
if [ "$GITHUB_TOKEN" != "" ]; then
echo "Detected GITHUB_TOKEN. Setting git config to use the security token" >&1
git config --global url."https://${GITHUB_TOKEN}#github.com/".insteadOf git#github.com:
fi
add the prebuild script to package.json:
"scripts": {
"heroku-prebuild": "bash heroku-prebuild.sh"
}
For local environment we can also use git config ... or we can add the access token to ~/.netrc file:
machine github.com
login PASTE_GITHUB_USERNAME_HERE
password PASTE_GITHUB_TOKEN_HERE
and installing private github repos should work.
npm install OWNER/REPO --save will appear in package.json as: "REPO": "github:OWNER/REPO"
and resolving private repos in Heroku build should also work.
optionally you can setup a postbuild script to unset the GITHUB_TOKEN.
This answer is good https://stackoverflow.com/a/29677091/6135922, but I changed a little bit preinstall script. Hope this will help someone.
#!/bin/bash
# Generates an SSH config file for connections if a config var exists.
echo "Preinstall"
if [ "$GIT_SSH_KEY" != "" ]; then
echo "Detected SSH key for git. Adding SSH config" >&1
echo "" >&1
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Load the private key into a file.
echo $GIT_SSH_KEY | base64 --decode > ~/.ssh/deploy_key
# Change the permissions on the file to
# be read-only for this user.
chmod o-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/deploy_key
# Setup the ssh config file.
echo -e "Host bitbucket.org\n"\
" IdentityFile ~/.ssh/deploy_key\n"\
" HostName bitbucket.org\n" \
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
> ~/.ssh/config
echo "eval `ssh-agent -s`"
eval `ssh-agent -s`
echo "ssh-add -l"
ssh-add -l
echo "ssh-add ~/.ssh/deploy_key"
ssh-add ~/.ssh/deploy_key
# uncomment to check that everything works just fine
# ssh -v git#bitbucket.org
fi
You can use in package.json private repository with authentication example below:
https://usernamegit:passwordgit#github.com/reponame/web/tarball/branchname
In short it is not possible. The best solution to this problem I came up with is to use the new git subtree's. At the time of writing they are not in the official git source and so needs to be installed manual but they will be included in v1.7.11. At the moment it is available on homebrew and apt-get. it is then a case of doing
git subtree add -P /node_modules/someprivatemodue git#github.......someprivatemodule {master|tag|commit}
this bulks out the repo size but an update is easy by doing the command above with gitsubtree pull.
I have done this before with modules from github. Npm currently accepts the name of the package or a link to a tar.gz file which contains the package.
For example if you want to use express.js directly from Github (grab the link via the download section) you could do:
"dependencies" : {
"express" : "https://github.com/visionmedia/express/tarball/2.5.9"
}
So you need to find a way to access you repository as a tar.gz file via http(s).

Resources