I run the GitLab from docker-compose. I had repositories with content and everything had been working as expected but now it says "The repository for this project does not exist." and I cannot get back my existing repositories and the content of them.
In /etc/gitlab/gitlab.rb file the git_data_dirs points to /var/opt/gitlab/git-data as default. The user/group set to git for /var/opt/gitlab/git-data/repositories recursively.
Content of /var/opt/gitlab/git-data/repositories:
ll /var/opt/gitlab/git-data/repositories
total 84
drwxrws--- 3 git git 4096 Dec 22 01:32 +gitaly/
drwxrws--- 4 git git 4096 Dec 22 09:28 ./
drwx------ 3 git git 4096 Aug 31 2020 ../
-rwxrwx--- 1 git git 64 Dec 22 09:28 .gitaly-metadata*
-rwxrwx--- 1 git git 80 Dec 22 03:15 .gitaly-metadata.locked*
drwxrws--- 16 git git 4096 Apr 19 2021 #hashed/
-rwxrwx--- 1 git git 57607 Dec 22 03:15 __$$RECOVERY_README$$__.html
Content of /var/opt/gitlab/git-data/repositories/#hashed: (It seems my Git repos are available)
ll /var/opt/gitlab/git-data/repositories/#hashed
total 64
drwxrws--- 16 git git 4096 Apr 19 2021 ./
drwxrws--- 4 git git 4096 Dec 22 09:28 ../
drwxrws--- 3 git git 4096 Oct 21 2020 19/
drwxrws--- 3 git git 4096 Oct 21 2020 2c/
drwxrws--- 3 git git 4096 Dec 4 2020 3f/
drwxrws--- 3 git git 4096 Oct 28 2020 4a/
drwxrws--- 3 git git 4096 Sep 20 2020 4b/
drwxrws--- 3 git git 4096 Sep 20 2020 4e/
drwxrws--- 3 git git 4096 Nov 5 2020 4f/
drwxrws--- 4 git git 4096 Nov 11 2020 6b/
drwxrws--- 3 git git 4096 Oct 21 2020 79/
drwxrws--- 3 git git 4096 Dec 11 2020 85/
drwxrws--- 3 git git 4096 Aug 28 2020 d4/
drwxrws--- 3 git git 4096 Apr 19 2021 e6/
drwxrws--- 3 git git 4096 Oct 12 2020 e7/
drwxrws--- 3 git git 4096 Sep 20 2020 ef/
The repos are even not accessible from my local PC:
>>> git fetch
Username for 'https://my_host.com': milan.balazs
Password for 'https://milan.balazs#my_host.com':
remote: A repository for this project does not exist yet.
fatal: repository 'https://my_host.com/systemdevelopers/tools.git/' not found
I have search and tried many "solution"/work-around but I couldn't solve it.
Tried:
gitlab-rake cache:clear
gitlab-rake gitlab:check
gitlab-ctl reconfigure
Restart Docker Service
Restart Docker Swarm
Restart Docker Daemon
The given error on UI:
My docker-compose.yml:
version: "3.8"
services:
gitlab:
image: ${ENVIRONMENT_HOST}:5000/environment/gitlab
ports:
- "80:80"
- "443:443"
- "60000:22"
hostname: "${ENVIRONMENT_HOST}"
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://${ENVIRONMENT_HOST}'
gitlab_rails['gitlab_shell_ssh_port'] = 60000
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/fullchain.pem'
nginx['ssl_certificate_key'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/privkey.pem'
gitlab_shell['custom_hooks_dir'] = '/home/git/gitlab-shell/hooks'
volumes:
- gitlab_config:/etc/gitlab
- gitlab_logs:/var/log/gitlab
- gitlab_data:/var/opt/gitlab
- /etc/letsencrypt:/etc/letsencrypt
- ./gitlab/server_hooks/:/home/git/gitlab-shell/hooks
networks:
- gitlab_ci_net
gitlab-runner-1:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_1:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-2:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_2:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-3:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_3:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
volumes:
gitlab_config: {}
gitlab_logs: {}
gitlab_data: {}
gitlab_runner_config_1: {}
gitlab_runner_config_2: {}
gitlab_runner_config_3: {}
networks:
gitlab_ci_net: {}
image_registry_net: {}
The ${ENVIRONMENT_HOST}:5000/environment/gitlab Dockerfile:
FROM gitlab/gitlab-ce:13.6.3-ce.0
# Install python3 for server hooks
RUN apt-get update && apt-get install -y python3
# Copy server hooks
COPY --chown=git:git ./server_hooks /home/git/gitlab-shell/hooks
# Give permission to server hooks
RUN find . -name *.py -exec chmod +x {} \
The gitlab-ctl reconfigure command has been ran successfully.
Some strange log parts:
I don't know the following error messages from log folder are related or not but I share them, perhaps they can help in debugging. The used command to find messages root#env:/var# grep -rnw 'log/' -e 'error'
In my understanding the gitaly is not able to recognize the Git repository (I can see this message for more Git repos).
log/gitlab/gitaly/#4000000061c31f1028b9701c.u:437:
{
"correlation_id":"EyMTcQIIRP9",
"error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git'",
"grpc.code":"NotFound",
"grpc.meta.auth_version":"v2",
"grpc.meta.client_name":"gitlab-web",
"grpc.meta.deadline_type":"regular",
"grpc.method":"FindDefaultBranchName",
"grpc.request.deadline":"2021-12-22T12:32:41Z",
"grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName",
"grpc.request.glProjectPath":"systemdevelopers/database",
"grpc.request.glRepository":"project-6",
"grpc.request.repoPath":"#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git",
"grpc.request.repoStorage":"default",
"grpc.request.topLevelGroup":"#hashed",
"grpc.service":"gitaly.RefService",
"grpc.start_time":"2021-12-22T12:32:31Z",
"grpc.time_ms":0.252,
"level":"info",
"msg":"finished unary call with code NotFound",
"peer.address":"#",
"pid":266,
"span.kind":"server",
"system":"grpc",
"time":"2021-12-22T12:32:31.361Z"
}
Other strange message (I see more times in my logs):
log/gitlab/gitlab-rails/application_json.log:42:
{
"severity":"ERROR",
"time":"2021-12-22T12:07:41.806Z",
"correlation_id":null,
"message":" \n!! RUNTIME IDENTIFICATION FAILED: Failed to identify runtime for process 52717 (bin/rails)\n Runtime based configuration settings may not work properly.\n If you continue to see this error, please file an issue via\n https://gitlab.com/gitlab-org/gitlab/issues/new\n"
}
Some errors from gitlab-workhorse:
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11373:
{
"command":[
"exiftool",
"-all=",
"--IPTC:all",
"--XMP-iptcExt:all",
"-tagsFromFile",
"#",
"-ResolutionUnit",
"-XResolution",
"-YResolution",
"-YCbCrSubSampling",
"-YCbCrPositioning",
"-BitsPerSample",
"-ImageHeight",
"-ImageWidth",
"-ImageSize",
"-Copyright",
"-CopyrightNotice",
"-Orientation",
"-"
],
"correlation_id":"ODWhqqnLNH2",
"error":"exit status 1",
"level":"info",
"msg":"exiftool command failed",
"stderr":" % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0Error: Writing of this type of file is not supported - -\n",
"time":"2021-12-22T12:44:50Z"
}
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11374:
{
"correlation_id":"ODWhqqnLNH2",
"error":"error while removing EXIF",
"level":"error",
"method":"POST",
"msg":"error",
"time":"2021-12-22T12:44:50Z",
"uri":"/uploads/user"
}
log/gitlab/gitlab-workhorse/#4000000061c343b71091c0ac.u:4:
{
"correlation_id":"OKWMhVphvr5",
"duration_ms":0,
"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused",
"level":"error",
"method":"GET",
"msg":"error",
"time":"2021-12-22T12:51:32Z",
"uri":"/help"
}
Results of GitLab environment info:
root#env:/# gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.1
Redis Version: 5.0.9
Git Version: 2.29.0
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.6.3
Revision: 857c6c6a6a9
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.9
URL: https://my_host.com
HTTP Clone URL: https://my_host.com/some-group/some-project.git
SSH Clone URL: ssh://git#my_host.com:60000/some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.13.0
Repository storage paths:
- default: /storage/data/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check:
root#env:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
11/1 ... yes
5/2 ... yes
11/3 ... yes
11/4 ... yes
11/5 ... yes
11/6 ... yes
11/7 ... yes
11/8 ... yes
11/9 ... yes
11/10 ... yes
11/11 ... yes
11/12 ... yes
11/13 ... yes
11/14 ... yes
11/15 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 14
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
I have already written a bug ticket for GitLab but I haven't got any feedback yet: https://gitlab.com/gitlab-org/gitlab/-/issues/349149
I would be very grateful if somebody could provide some hits how I can recover my system.
I am got an issue when running the whole pipeline of ChIP-seq using profile singularity on my local PC (window but subsystem Linux)
Error executing process > 'output_documentation'
Caused by:
Failed to pull singularity image
command: singularity pull --name nfcore-chipseq-1.2.2.img.pulling.1630098407814 docker://nfcore/chipseq:1.2.2 > /dev/null
status : 255
message:
INFO: Using cached SIF image
FATAL: While making image from oci registry: error copying image out of cache: could not open temporary file for copy: failed to change permission of ./tmp-copy-2575820807: chmod ./tmp-copy-2575820807: operation not permitted
I'm using singularity 3.8.2
I also have specified NXF_SINGULARITY_CACHEDIR to a hard drive instead of /home/.singularity
I also checked the folder to make sure all the file can be accessed
total 0
drwxrwxrwx 1 root root 4096 Aug 28 05:06 .
drwxrwxrwx 1 root root 4096 Aug 28 04:47 ..
-rwxrwxrwx 1 root root 0 Aug 28 04:53 tmp-copy-2299332276
-rwxrwxrwx 1 root root 0 Aug 28 05:06 tmp-copy-2575820807
I am running Jenkins as a docker container, and have installed the NodeJS plugin and followed thoroughly the setup instructions. When I try to run a script using node, I get the following error:
/tmp/jenkins9123978873441132802.sh: line 1: node: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I checked the docker volume, the node bin is where it should be and is executable is there and it works fine when I run it from my host server:
user#server:/data/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin$ ./node --version
v9.2.0
I modified my build script to explore a bit further the problem:
echo $PATH
cd /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
ls -all
./node --version
node --version
npm --version
and look how strange this is:
Building in workspace /var/jenkins_home/workspace/release
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Adding all registry entries
copy managed file [Main config] to file:/var/jenkins_home/workspace/release#tmp/config69012336710357692tmp
[release] $ /bin/sh -xe /tmp/jenkins6243047436861395796.sh
+ echo /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
/var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
+ cd /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
+ ls -all
total 34112
drwxr-xr-x 2 jenkins jenkins 4096 Nov 20 16:16 .
drwxr-xr-x 6 jenkins jenkins 4096 Nov 20 16:16 ..
-rwxrwxrwx 1 jenkins jenkins 34921762 Nov 14 20:33 node
lrwxrwxrwx 1 jenkins jenkins 38 Nov 20 16:16 npm -> ../lib/node_modules/npm/bin/npm-cli.js
lrwxrwxrwx 1 jenkins jenkins 38 Nov 20 16:16 npx -> ../lib/node_modules/npm/bin/npx-cli.js
+ ./node --version
/tmp/jenkins6243047436861395796.sh: line 1: ./node: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
The node executable is present, and it's executable (+x). The path is correctly set, but the build still fails.
This is because the path to the node binary
/data/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
Does not exist on your shell path.
You should edit Jenkins' variables to adjust your PATH.
I have a main Git bare repository in my Dropbox folder and I'm using a headless Debian server which runs Jenkins to build my projects.
Until now I've been using this approach to build my projects: I created a bare git repository on my headless server and configured the Jenkins job to pull the sources from this repository with the following URL: /var/cache/git/project. So when I want Jenkins to build my changes, I have to push them to this repository. This works fine, but it's not very convenient because I have to manually push to two repositories (Dropbox and Debian) and also when I want to add a new project, I need to create a new repository on the Debian server.
So I decided to configure the Jenkins jobs to pull directly from the Dropbox repositories. I've set up Dropbox on my Debian server at /home/jardo/Dropbox.
But when I try to configure the Jenkins job to pull drom URL /home/jardo/Dropbox/git/project.git, I get the following error:
Failed to connect to repository : Command "git ls-remote -h /home/jardo/Dropbox/git/project.git HEAD" returned status code 128:
stdout:
stderr: fatal: '/home/jardo/Dropbox/git/project.git' does not appear to be a git repository
fatal: The remote end hung up unexpectedly
How come Jenkins is able to clone from /var/cache/git/project, but not from my Dropbox folder? Dropbox is completely synced and the repository content looks ok. I've set permissions on the whole repository recursively to 777 and that didn't help.
Here is the output of ls -l on my Dropbox repository:
-rwxrwxrwx 1 jardo jardo 104 Dec 15 21:10 config
-rwxrwxrwx 1 jardo jardo 73 Dec 15 21:10 description
-rwxrwxrwx 1 jardo jardo 23 Dec 15 21:10 HEAD
drwxrwxrwx 2 jardo jardo 4096 Dec 24 13:16 hooks
drwxrwxrwx 2 jardo jardo 4096 Dec 24 13:01 info
drwxrwxrwx 150 jardo jardo 4096 Dec 24 13:01 objects
drwxrwxrwx 4 jardo jardo 4096 Dec 24 12:59 refs
It turned out that user Jenkins could not access the GIT repository, which was owned by user Jardo, even when I tried to set its permissions recursively to 777. The solution was to setup Dropbox also for user Jenkins.
I created my project and added my ssh-key (both over webinterface), but when I try to push something to the repo, I get this error: "Could not read from the repository". This might be due to the fact, that the .git-project-file does not exist in the /home/git/repositories/[group]/ folder.
The gitlab-check returns:
[Group] / [Project] ... can't create, repository is empty
Error when pushing to repository
git remote add origin git#myserver.com:[group]/[project].git
git push -u origin master
fatal: '/home/git/repositories/[group]/[project].git' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
What might have caused the error
ls -ld /home/git/repositories/[group]/[project].git
File not found.
Why was the file not created?
Infos about my setup
ls -ld /home/ /home/git/
drwxr-xr-x 3 root root 4096 Aug 28 15:04 /home/
drwxr-xr-x 8 git git 4096 Aug 31 11:33 /home/git/
ll /home/git
drwxr-xr-x 8 git git 4096 Aug 31 11:33 ./
drwxr-xr-x 3 root root 4096 Aug 28 15:04 ../
-rw-r--r-- 1 git git 220 Aug 28 15:04 .bash_logout
-rw-r--r-- 1 git git 3637 Aug 28 15:04 .bashrc
drwx------ 2 git git 4096 Aug 29 19:52 .cache/
-rw-r--r-- 1 git git 88 Aug 31 11:33 .gitconfig
drwxr-xr-x 16 git git 4096 Aug 29 15:32 gitlab/
drwxr-xr-x 2 git git 4096 Aug 28 15:09 gitlab-satellites/
drwxr-xr-x 8 git git 4096 Aug 29 19:51 gitlab-shell/
-rw------- 1 git git 13 Aug 29 15:36 .mysql_history
-rw-r--r-- 1 git git 675 Aug 28 15:04 .profile
drwxrws--- 5 git git 4096 Aug 29 16:15 repositories/
drwx------ 2 git git 4096 Aug 31 11:47 .ssh/
-rw------- 1 git git 3453 Aug 29 16:03 .viminfo
sudo -u git -H bundle exec rake gitlab:env:info
RAILS_ENV=production
System information
System: Ubuntu 13.04
Current User: git
Using RVM: no
Ruby Version: 2.0.0p247
Gem Version: 2.0.3
Bundler Version:1.3.5
Rake Version: 10.1.0
GitLab information
Version: 6.0.0
Revision: 6c1c284
Directory: /home/git/gitlab
DB Adapter: mysql2
URL: http://myserver.com
HTTP Clone URL: http://myserver.com/some-project.git
SSH Clone URL: git#myserver.com:some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 1.7.0
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/bin/git
sudo -u git -H bundle exec rake gitlab:check
RAILS_ENV=production
Checking Environment ...
Git configured for git user? ... yes
Has python2? ... yes
python2 is supported version? ... yes
Checking Environment ... Finished
Checking GitLab Shell ...
GitLab Shell version >= 1.7.0 ? ... OK (1.7.0)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by git:git? ... yes
Repo base access is drwxrws---? ... yes
post-receive hook up-to-date? ... yes
post-receive hooks in repos are links: ...
[Group] / [Project] ... repository is empty
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Checking Sidekiq ... Finished
Checking GitLab ...
Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have satellites? ...
[Group] / [Project] ... can't create, repository is empty
Redis version >= 2.0.0? ... yes
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (1.8.1)
Checking GitLab ... Finished
ssh -T git#myserver.com
Welcome to GitLab, Christian Hammer!
cat /home/git/.ssh/authorized_keys
command="/home/git/gitlab-shell/bin/gitlab-shell key-2",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa [key] [email]
(Re)starting sidekiq solved the problem:
sudo -Hu git RAILS_ENV=production bundle exec rake sidekiq:start
From the information you have given, it is not able to create. Please check what exists inside of repositories/[group] directory.
If your repository is a bare repository (one which does not have working directory), then it is a convention to use a name like [project.git].
I had the same problem. But for me restarting sidekiq did not solve it. So i leave a small solution for the same error message. I manage the server gitlab is running on, so i created one key for remote ssh access (gitlab-server.pub) and another one (gitlab-repo.pub) for the repository. In the gitlab interface i uploaded gitlab-repo.pub but i forgot git to tell which one to use so after updating ~/.ssh/config like this everything works