keytar.node causing issues in Buildkite pipeline - linux

I'm trying to create a CI/CD pipeline using buildkite where the build steps runs on a Virtual Machine (Ubuntu). However, an npm package called keytar which is a dependency of tedious library is causing issues in the pipeline.
When running one of the steps from using the buildkite-agent, it tries to clean up an existing repository. And during the command git clean -fxdq it fails to remove keytar.node file due to a Permission denied Error.
Here's a full log of the step:
> Running global environment hook
> Setting up plugins
> Preparing working directory
$ cd /var/lib/buildkite-agent/builds/build-server-1/someorg/some-repo
# Host "bitbucket.org" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"
$ git remote set-url origin git#bitbucket.org:someorg/some-repo.git
$ git clean -fxdq
warning: failed to remove node_modules/keytar/build/Release/keytar.node: Permission denied
# Removing /var/lib/buildkite-agent/builds/build-server-1/someorg/some-repo
⚠️ Warning: Checkout failed! Error running `/usr/bin/git clean -fxdq`: exit status 1 (Attempt 1/3 Retrying in 2s)
# Host "bitbucket.org" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"
$ git clone -v -- git#bitbucket.org:someorg/some-repo.git .
fatal: destination path '.' already exists and is not an empty directory.
# Removing /var/lib/buildkite-agent/builds/build-server-1/someorg/some-repo
⚠️ Warning: Checkout failed! Error running `/usr/bin/git clone -v -- git#bitbucket.org:someorg/some-repo.git .`: exit status 128 (Attempt 2/3 Retrying in 2s)
# Host "bitbucket.org" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"
$ git clone -v -- git#bitbucket.org:someorg/some-repo.git .
fatal: destination path '.' already exists and is not an empty directory.
# Removing /var/lib/buildkite-agent/builds/build-server-1/someorg/some-repo
⚠️ Warning: Checkout failed! Error running `/usr/bin/git clone -v -- git#bitbucket.org:someorg/some-repo.git .`: exit status 128 (Attempt 3/3 Retrying in 2s)
# Skipping artifact upload, no checkout
🚨 Error: Error running `/usr/bin/git clone -v -- git#bitbucket.org:someorg/some-repo.git .`: exit status 128
Upon examining the permissions on the folder /some-repo using the ls -l command the permissions seem to be drwxr-xr-x and the owner is buildkite-agent. I'm stumped as how this file does not get deleted while all other files get deleted without any issue.

Related

Is there a solution for the odd error of bitbake?

When I used yocto to build my first linux system and after 'bitbake imx-image-multimedia' was excuted, I faced the odd error:
ERROR: gnu-config-native-20190501+gitAUTOINC+b98424c249-r0 do_unpack: Unpack failure for URL: 'git://git.savannah.gnu.org/config.git'. No up to date source found: clone directory not available or not up to date: /home/admin/Linux/Yocto/fsl/downloads//git2/git.savannah.gnu.org.config.git; shallow clone not enabled
ERROR: Logfile of failure stored in: /home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/temp/log.do_unpack.73483
ERROR: Task (virtual:native:/home/admin/Linux/Yocto/fsl/sources/poky/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_unpack) failed with exit code '1'
Curious about the logfile, I opened /home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/temp/log.do_unpack.73483 and I see:
DEBUG: Executing python function do_unpack
DEBUG: Executing python function base_do_unpack
DEBUG: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; export SSH_AUTH_SOCK="/run/user/0/vscode-ssh-auth-sock-7925763"; export PATH="/home/admin/Linux/Yocto/fsl/sources/poky/scripts/native-intercept:/home/admin/Linux/Yocto/fsl/sources/poky/scripts:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/usr/bin/x86_64-linux:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/usr/bin:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/usr/sbin:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/usr/bin:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/sbin:/home/admin/Linux/Yocto/fsl/build/tmp/work/x86_64-linux/gnu-config-native/20190501+gitAUTOINC+b98424c249-r0/recipe-sysroot-native/bin:/home/admin/Linux/Yocto/fsl/sources/poky/bitbake/bin:/home/admin/Linux/Yocto/fsl/build/tmp/hosttools"; export HOME="/root"; git -c core.fsyncobjectfiles=0 branch --contains b98424c249119b79d3f709e26eb86f2fd4d5e5f3 --list master 2> /dev/null | wc -l' in /home/admin/Linux/Yocto/fsl/downloads//git2/git.savannah.gnu.org.config.git
ERROR: Unpack failure for URL: 'git://git.savannah.gnu.org/config.git'. No up to date source found: clone directory not available or not up to date: /home/admin/Linux/Yocto/fsl/downloads//git2/git.savannah.gnu.org.config.git; shallow clone not enabled
DEBUG: Python function base_do_unpack finished
DEBUG: Python function do_unpack finished
What is 'ERROR: Unpack failure for URL: git://git.savannah.gnu.org/config.git'. No up to date source found: clone directory not available or not up to date: /home/admin/Linux/Yocto/fsl/downloads//git2/git.savannah.gnu.org.config.git; shallow clone not enabled means?
What can I do to fix it? Thanks!
See first if this is similar to this thread
Currently, for my Yocto builds (for nxp and other boards) I used to share a same "downloads" DL_DIR to avoid unnecessary fetch operations.
I tried to use an empty DL_DIR...and it worked fine.
After investigating, I found out there is something wrecked in the "git2" sub-directory of DL_DIR.
I don't know what exactly.
So if you have a custom DL_DIR with a lot of stuff, try to rename your "git2" subdir as "git2.bak".
Check also if the // seen in /home/admin/Linux/Yocto/fsl/downloads//git2/git.savannah.gnu.org.config.git means an empty environment variable, meaning check if there should be an intermediate folder between downloads/ and /git2.
I've recently encountered this problem. I had an empty do_fetch function
do_fetch(){
:
}
. Just by removing it, the git repo was cloned properly.

Jenkins git fetch returned status code 143

I'm facing the problem with Jenkins with fetching git repo on Linux. I attach the stack trace. I have SSH gitlab connection configured with a private key and a passphrase. The configuration is made in Credentials/System/Global credentials (unrestricted). The path to the private key is /home/jenkins/.jenkins/secrets/my_private_key.pub. Owner and a group: jenkins:jenkins, where jenkins is the user making the installation. Permissions: 600.
The private key has been checked with the passphrase using ssh from the command line. The git fetch command used by jenkins has also been checked with command line. Both works.
Repository URL format is like:
git#github.com:org-name/project.git (short notation for ssh protocol)
Environment:
* CentOS 7 64 bit
* openjdk version "1.8.0_131" 64 bit
* Jenkins ver. 2.67
* Git client plugin 2.4.6Git plugin 3.3.0 (3.3.1 is not helping too)
* GIT server Plugin 1.7
* GitLab Plugin 1.4.5 (1.4.6 is not helping too)
* SCM API Plugin 2.1.1
* Credentials Plugin 2.1.14
* Jenkins is run directly
* Jenkins was installed with yum installer
* Browser: Google Chrome 59
* GitLab Community Edition 9.3.5
* git version 1.8.3.1
Stack trace:
started by user Jenkins Admin
Building in workspace /home/jenkins/.jenkins/workspace/my_project
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url git#gitlab.repo:XYZ/docs/project.git #
timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
> git clean -fdx # timeout=10
Fetching upstream changes from git#gitlab.repo:XYZ/docs/project.git
> git --version # timeout=10
using GIT_SSH to set credentials
> git fetch --no-tags --progress git#gitlab.repo:XYZ/docs/project.git
+refs/heads/*:refs/remotes/origin/* --depth=1
ERROR: Timeout after 10 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git#gitlab.repo:XYZ/docs/project.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:809)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1076)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1107)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1212)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:560)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:485)
at hudson.model.Run.execute(Run.java:1735)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:415)
Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --progress git#gitlab.repo:XYZ/docs/project.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code 143:
stdout:
stderr:
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1903)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1622)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:71)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:348)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:807)
... 11 more
ERROR: null
Finished: FAILURE
I found a solution. Running Jenkins as a service with systemctl resolved the problem, instead of running Jenkins with java -jar jenkins.war.
To give more details:
According to the documentation, Jenkins is started by
java -jar jenkins.war
And it was visible as running in the system when I checked it with:
ps aux | grep jenkins
However, on CentOS 7 the suggested way of running Jenkins is to run it as a daemon with systemctl command.
When I checked it with
sudo systemctl status jenkins.service
it was inactive.
So I started Jenkins with
sudo systemctl start jenkins.service
and it worked.

gitlab: git-http-push failed, return code 22

First of all my gitlab connection was broken by error 500 and 502. The server was unattended for a while. So i decided to reinstall:
#gitlab-ctl cleanse
#apt-get purge gitlab-ce
Installation:
https://about.gitlab.com/downloads/#ubuntu1404
After the gitlab was reachable and i could create a project and clone it.
After i tried to push a new file to my repo and got following message:
git add .
git commit -m "inital"
git push --set-upstream origin master
error: Cannot access URL http://<username>#gitlab.domain.de:<port>/group/project.git/, return code 22
fatal: git-http-push failed
error: failed to push some refs to 'http://<username>#gitlab.domain.de:<port>/group/project.git/'
The gitlab-check gives this information:
#gitlab-rake gitlab:check
everything was green, besides violett ones:
Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Why i can clone but not push? I searched in several log-files, but got nothing useful.

How I can config the fluentd Proxy for install plugins?

I can't install a plugin at fluentd. i've set the proxy-variable at the System (rhel) but it have no influence.
$ td-agent --gemfile grok-plugin/Gemfile
fatal: Not a git repository (or any of the parent directories): .git
Fetching source index from https://rubygems.org/
Retrying source fetch due to error (2/3): Bundler::HTTPError Could not fetch specs from https://rubygems.org/
Retrying source fetch due to error (3/3): Bundler::HTTPError Could not fetch specs from https://rubygems.org/
Could not fetch specs from https://rubygems.org/
$ echo $HTTP_PROXY
http://proxy.de:port
$ echo $HTTPS_PROXY
http://proxy.de:port
$ echo $https_proxy
http://proxy.de:port
$ echo $http_proxy
http://proxy.de:port
Please can you help me to set the proxy for the gem-install?
I find the solution. I have an other Problem and it gave me an approach. In rhel6 you must write the proxy without http://, then it works. (Write the proxy without protocol, is only the solution for this problem)

Why does git clone hang when calling from a shell script which SSH's to a remote server?

I am writing a small app for those in the company who don't like command line and to help with productivity. The app runs a shell script which gets input from the user, uses it to SSH to a remote staging server and git clone the repo to that remote server. SSH keys are all good for the staging server and on Bitbucket. I can manually SSH into the server and git clone without any issue. I have also managed to get it to work going to a live server for initial setup. All other commands sent to the server are working.
The code used for this particular part is:
ssh $SUSER#$SHOST "git clone --depth 1 git#bitbucket.org:team/$SITEADDRESS.git --branch staging $CURRDIR"
I've also tried to run this manually from terminal with the following code:
ssh user#server "GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone -v --depth 1 git#bitbucket.org:team/repo.git --branch staging /path/to/folder"
This is the output I get:
trace: built-in: git 'clone' '-v' '--depth' '1' 'git#bitbucket.org:team/repo.git' '--branch' 'staging' '/path/to/folder'
Cloning into '/path/to/folder'...
trace: run_command: 'ssh' 'git#bitbucket.org' 'git-upload-pack '\''team/repo.git'\'''
trace: run_command: '--shallow-file' '/path/to/folder/.git/shallow.lock' 'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 557 on ubuntu'
trace: exec: 'git' '--shallow-file' '/path/to/folder/.git/shallow.lock' 'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 557 on ubuntu'
trace: built-in: git 'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 557 on ubuntu'
At this point it just hangs without errors instead of moving on to remote: Counting objects: 4535, done. and cloning the repo.
I've been trying to get this working for quite a few hours now and have been unable to find any info on similar problems. Using Git v1.9.1

Resources