Received error in GitLab: "The repository for this project does not exist." - gitlab

Question

I run the GitLab from docker-compose. I had repositories with content and everything had been working as expected but now it says "The repository for this project does not exist." and I cannot get back my existing repositories and the content of them.
In /etc/gitlab/gitlab.rb file the git_data_dirs points to /var/opt/gitlab/git-data as default. The user/group set to git for /var/opt/gitlab/git-data/repositories recursively.
Content of /var/opt/gitlab/git-data/repositories:
ll /var/opt/gitlab/git-data/repositories
total 84
drwxrws--- 3 git git 4096 Dec 22 01:32 +gitaly/
drwxrws--- 4 git git 4096 Dec 22 09:28 ./
drwx------ 3 git git 4096 Aug 31 2020 ../
-rwxrwx--- 1 git git 64 Dec 22 09:28 .gitaly-metadata*
-rwxrwx--- 1 git git 80 Dec 22 03:15 .gitaly-metadata.locked*
drwxrws--- 16 git git 4096 Apr 19 2021 #hashed/
-rwxrwx--- 1 git git 57607 Dec 22 03:15 __$$RECOVERY_README$$__.html
Content of /var/opt/gitlab/git-data/repositories/#hashed: (It seems my Git repos are available)
ll /var/opt/gitlab/git-data/repositories/#hashed
total 64
drwxrws--- 16 git git 4096 Apr 19 2021 ./
drwxrws--- 4 git git 4096 Dec 22 09:28 ../
drwxrws--- 3 git git 4096 Oct 21 2020 19/
drwxrws--- 3 git git 4096 Oct 21 2020 2c/
drwxrws--- 3 git git 4096 Dec 4 2020 3f/
drwxrws--- 3 git git 4096 Oct 28 2020 4a/
drwxrws--- 3 git git 4096 Sep 20 2020 4b/
drwxrws--- 3 git git 4096 Sep 20 2020 4e/
drwxrws--- 3 git git 4096 Nov 5 2020 4f/
drwxrws--- 4 git git 4096 Nov 11 2020 6b/
drwxrws--- 3 git git 4096 Oct 21 2020 79/
drwxrws--- 3 git git 4096 Dec 11 2020 85/
drwxrws--- 3 git git 4096 Aug 28 2020 d4/
drwxrws--- 3 git git 4096 Apr 19 2021 e6/
drwxrws--- 3 git git 4096 Oct 12 2020 e7/
drwxrws--- 3 git git 4096 Sep 20 2020 ef/
The repos are even not accessible from my local PC:
>>> git fetch
Username for 'https://my_host.com': milan.balazs
Password for 'https://milan.balazs#my_host.com':
remote: A repository for this project does not exist yet.
fatal: repository 'https://my_host.com/systemdevelopers/tools.git/' not found
I have search and tried many "solution"/work-around but I couldn't solve it.
Tried:
gitlab-rake cache:clear
gitlab-rake gitlab:check
gitlab-ctl reconfigure
Restart Docker Service
Restart Docker Swarm
Restart Docker Daemon
The given error on UI:
My docker-compose.yml:
version: "3.8"
services:
gitlab:
image: ${ENVIRONMENT_HOST}:5000/environment/gitlab
ports:
- "80:80"
- "443:443"
- "60000:22"
hostname: "${ENVIRONMENT_HOST}"
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://${ENVIRONMENT_HOST}'
gitlab_rails['gitlab_shell_ssh_port'] = 60000
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/fullchain.pem'
nginx['ssl_certificate_key'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/privkey.pem'
gitlab_shell['custom_hooks_dir'] = '/home/git/gitlab-shell/hooks'
volumes:
- gitlab_config:/etc/gitlab
- gitlab_logs:/var/log/gitlab
- gitlab_data:/var/opt/gitlab
- /etc/letsencrypt:/etc/letsencrypt
- ./gitlab/server_hooks/:/home/git/gitlab-shell/hooks
networks:
- gitlab_ci_net
gitlab-runner-1:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_1:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-2:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_2:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-3:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_3:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
volumes:
gitlab_config: {}
gitlab_logs: {}
gitlab_data: {}
gitlab_runner_config_1: {}
gitlab_runner_config_2: {}
gitlab_runner_config_3: {}
networks:
gitlab_ci_net: {}
image_registry_net: {}
The ${ENVIRONMENT_HOST}:5000/environment/gitlab Dockerfile:
FROM gitlab/gitlab-ce:13.6.3-ce.0
# Install python3 for server hooks
RUN apt-get update && apt-get install -y python3
# Copy server hooks
COPY --chown=git:git ./server_hooks /home/git/gitlab-shell/hooks
# Give permission to server hooks
RUN find . -name *.py -exec chmod +x {} \
The gitlab-ctl reconfigure command has been ran successfully.
Some strange log parts:
I don't know the following error messages from log folder are related or not but I share them, perhaps they can help in debugging. The used command to find messages root#env:/var# grep -rnw 'log/' -e 'error'
In my understanding the gitaly is not able to recognize the Git repository (I can see this message for more Git repos).
log/gitlab/gitaly/#4000000061c31f1028b9701c.u:437:
{
"correlation_id":"EyMTcQIIRP9",
"error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git'",
"grpc.code":"NotFound",
"grpc.meta.auth_version":"v2",
"grpc.meta.client_name":"gitlab-web",
"grpc.meta.deadline_type":"regular",
"grpc.method":"FindDefaultBranchName",
"grpc.request.deadline":"2021-12-22T12:32:41Z",
"grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName",
"grpc.request.glProjectPath":"systemdevelopers/database",
"grpc.request.glRepository":"project-6",
"grpc.request.repoPath":"#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git",
"grpc.request.repoStorage":"default",
"grpc.request.topLevelGroup":"#hashed",
"grpc.service":"gitaly.RefService",
"grpc.start_time":"2021-12-22T12:32:31Z",
"grpc.time_ms":0.252,
"level":"info",
"msg":"finished unary call with code NotFound",
"peer.address":"#",
"pid":266,
"span.kind":"server",
"system":"grpc",
"time":"2021-12-22T12:32:31.361Z"
}
Other strange message (I see more times in my logs):
log/gitlab/gitlab-rails/application_json.log:42:
{
"severity":"ERROR",
"time":"2021-12-22T12:07:41.806Z",
"correlation_id":null,
"message":" \n!! RUNTIME IDENTIFICATION FAILED: Failed to identify runtime for process 52717 (bin/rails)\n Runtime based configuration settings may not work properly.\n If you continue to see this error, please file an issue via\n https://gitlab.com/gitlab-org/gitlab/issues/new\n"
}
Some errors from gitlab-workhorse:
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11373:
{
"command":[
"exiftool",
"-all=",
"--IPTC:all",
"--XMP-iptcExt:all",
"-tagsFromFile",
"#",
"-ResolutionUnit",
"-XResolution",
"-YResolution",
"-YCbCrSubSampling",
"-YCbCrPositioning",
"-BitsPerSample",
"-ImageHeight",
"-ImageWidth",
"-ImageSize",
"-Copyright",
"-CopyrightNotice",
"-Orientation",
"-"
],
"correlation_id":"ODWhqqnLNH2",
"error":"exit status 1",
"level":"info",
"msg":"exiftool command failed",
"stderr":" % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0Error: Writing of this type of file is not supported - -\n",
"time":"2021-12-22T12:44:50Z"
}
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11374:
{
"correlation_id":"ODWhqqnLNH2",
"error":"error while removing EXIF",
"level":"error",
"method":"POST",
"msg":"error",
"time":"2021-12-22T12:44:50Z",
"uri":"/uploads/user"
}
log/gitlab/gitlab-workhorse/#4000000061c343b71091c0ac.u:4:
{
"correlation_id":"OKWMhVphvr5",
"duration_ms":0,
"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused",
"level":"error",
"method":"GET",
"msg":"error",
"time":"2021-12-22T12:51:32Z",
"uri":"/help"
}
Results of GitLab environment info:
root#env:/# gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.1
Redis Version: 5.0.9
Git Version: 2.29.0
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.6.3
Revision: 857c6c6a6a9
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.9
URL: https://my_host.com
HTTP Clone URL: https://my_host.com/some-group/some-project.git
SSH Clone URL: ssh://git#my_host.com:60000/some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.13.0
Repository storage paths:
- default: /storage/data/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check:
root#env:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
11/1 ... yes
5/2 ... yes
11/3 ... yes
11/4 ... yes
11/5 ... yes
11/6 ... yes
11/7 ... yes
11/8 ... yes
11/9 ... yes
11/10 ... yes
11/11 ... yes
11/12 ... yes
11/13 ... yes
11/14 ... yes
11/15 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 14
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
I have already written a bug ticket for GitLab but I haven't got any feedback yet: https://gitlab.com/gitlab-org/gitlab/-/issues/349149
I would be very grateful if somebody could provide some hits how I can recover my system.