I run the GitLab from docker-compose. I had repositories with content and everything had been working as expected but now it says "The repository for this project does not exist." and I cannot get back my existing repositories and the content of them.
In /etc/gitlab/gitlab.rb file the git_data_dirs points to /var/opt/gitlab/git-data as default. The user/group set to git for /var/opt/gitlab/git-data/repositories recursively.
Content of /var/opt/gitlab/git-data/repositories:
ll /var/opt/gitlab/git-data/repositories
total 84
drwxrws--- 3 git git 4096 Dec 22 01:32 +gitaly/
drwxrws--- 4 git git 4096 Dec 22 09:28 ./
drwx------ 3 git git 4096 Aug 31 2020 ../
-rwxrwx--- 1 git git 64 Dec 22 09:28 .gitaly-metadata*
-rwxrwx--- 1 git git 80 Dec 22 03:15 .gitaly-metadata.locked*
drwxrws--- 16 git git 4096 Apr 19 2021 #hashed/
-rwxrwx--- 1 git git 57607 Dec 22 03:15 __$$RECOVERY_README$$__.html
Content of /var/opt/gitlab/git-data/repositories/#hashed: (It seems my Git repos are available)
ll /var/opt/gitlab/git-data/repositories/#hashed
total 64
drwxrws--- 16 git git 4096 Apr 19 2021 ./
drwxrws--- 4 git git 4096 Dec 22 09:28 ../
drwxrws--- 3 git git 4096 Oct 21 2020 19/
drwxrws--- 3 git git 4096 Oct 21 2020 2c/
drwxrws--- 3 git git 4096 Dec 4 2020 3f/
drwxrws--- 3 git git 4096 Oct 28 2020 4a/
drwxrws--- 3 git git 4096 Sep 20 2020 4b/
drwxrws--- 3 git git 4096 Sep 20 2020 4e/
drwxrws--- 3 git git 4096 Nov 5 2020 4f/
drwxrws--- 4 git git 4096 Nov 11 2020 6b/
drwxrws--- 3 git git 4096 Oct 21 2020 79/
drwxrws--- 3 git git 4096 Dec 11 2020 85/
drwxrws--- 3 git git 4096 Aug 28 2020 d4/
drwxrws--- 3 git git 4096 Apr 19 2021 e6/
drwxrws--- 3 git git 4096 Oct 12 2020 e7/
drwxrws--- 3 git git 4096 Sep 20 2020 ef/
The repos are even not accessible from my local PC:
>>> git fetch
Username for 'https://my_host.com': milan.balazs
Password for 'https://milan.balazs#my_host.com':
remote: A repository for this project does not exist yet.
fatal: repository 'https://my_host.com/systemdevelopers/tools.git/' not found
I have search and tried many "solution"/work-around but I couldn't solve it.
Tried:
gitlab-rake cache:clear
gitlab-rake gitlab:check
gitlab-ctl reconfigure
Restart Docker Service
Restart Docker Swarm
Restart Docker Daemon
The given error on UI:
My docker-compose.yml:
version: "3.8"
services:
gitlab:
image: ${ENVIRONMENT_HOST}:5000/environment/gitlab
ports:
- "80:80"
- "443:443"
- "60000:22"
hostname: "${ENVIRONMENT_HOST}"
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://${ENVIRONMENT_HOST}'
gitlab_rails['gitlab_shell_ssh_port'] = 60000
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/fullchain.pem'
nginx['ssl_certificate_key'] = '/etc/letsencrypt/live/${ENVIRONMENT_HOST}/privkey.pem'
gitlab_shell['custom_hooks_dir'] = '/home/git/gitlab-shell/hooks'
volumes:
- gitlab_config:/etc/gitlab
- gitlab_logs:/var/log/gitlab
- gitlab_data:/var/opt/gitlab
- /etc/letsencrypt:/etc/letsencrypt
- ./gitlab/server_hooks/:/home/git/gitlab-shell/hooks
networks:
- gitlab_ci_net
gitlab-runner-1:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_1:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-2:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_2:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
gitlab-runner-3:
image: gitlab/gitlab-runner:alpine-v13.6.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- gitlab_runner_config_3:/etc/gitlab-runner
networks:
- gitlab_ci_net
- image_registry_net
volumes:
gitlab_config: {}
gitlab_logs: {}
gitlab_data: {}
gitlab_runner_config_1: {}
gitlab_runner_config_2: {}
gitlab_runner_config_3: {}
networks:
gitlab_ci_net: {}
image_registry_net: {}
The ${ENVIRONMENT_HOST}:5000/environment/gitlab Dockerfile:
FROM gitlab/gitlab-ce:13.6.3-ce.0
# Install python3 for server hooks
RUN apt-get update && apt-get install -y python3
# Copy server hooks
COPY --chown=git:git ./server_hooks /home/git/gitlab-shell/hooks
# Give permission to server hooks
RUN find . -name *.py -exec chmod +x {} \
The gitlab-ctl reconfigure command has been ran successfully.
Some strange log parts:
I don't know the following error messages from log folder are related or not but I share them, perhaps they can help in debugging. The used command to find messages root#env:/var# grep -rnw 'log/' -e 'error'
In my understanding the gitaly is not able to recognize the Git repository (I can see this message for more Git repos).
log/gitlab/gitaly/#4000000061c31f1028b9701c.u:437:
{
"correlation_id":"EyMTcQIIRP9",
"error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository '/var/opt/gitlab/git-data/repositories/#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git'",
"grpc.code":"NotFound",
"grpc.meta.auth_version":"v2",
"grpc.meta.client_name":"gitlab-web",
"grpc.meta.deadline_type":"regular",
"grpc.method":"FindDefaultBranchName",
"grpc.request.deadline":"2021-12-22T12:32:41Z",
"grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName",
"grpc.request.glProjectPath":"systemdevelopers/database",
"grpc.request.glRepository":"project-6",
"grpc.request.repoPath":"#hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.git",
"grpc.request.repoStorage":"default",
"grpc.request.topLevelGroup":"#hashed",
"grpc.service":"gitaly.RefService",
"grpc.start_time":"2021-12-22T12:32:31Z",
"grpc.time_ms":0.252,
"level":"info",
"msg":"finished unary call with code NotFound",
"peer.address":"#",
"pid":266,
"span.kind":"server",
"system":"grpc",
"time":"2021-12-22T12:32:31.361Z"
}
Other strange message (I see more times in my logs):
log/gitlab/gitlab-rails/application_json.log:42:
{
"severity":"ERROR",
"time":"2021-12-22T12:07:41.806Z",
"correlation_id":null,
"message":" \n!! RUNTIME IDENTIFICATION FAILED: Failed to identify runtime for process 52717 (bin/rails)\n Runtime based configuration settings may not work properly.\n If you continue to see this error, please file an issue via\n https://gitlab.com/gitlab-org/gitlab/issues/new\n"
}
Some errors from gitlab-workhorse:
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11373:
{
"command":[
"exiftool",
"-all=",
"--IPTC:all",
"--XMP-iptcExt:all",
"-tagsFromFile",
"#",
"-ResolutionUnit",
"-XResolution",
"-YResolution",
"-YCbCrSubSampling",
"-YCbCrPositioning",
"-BitsPerSample",
"-ImageHeight",
"-ImageWidth",
"-ImageSize",
"-Copyright",
"-CopyrightNotice",
"-Orientation",
"-"
],
"correlation_id":"ODWhqqnLNH2",
"error":"exit status 1",
"level":"info",
"msg":"exiftool command failed",
"stderr":" % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0\r 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0Error: Writing of this type of file is not supported - -\n",
"time":"2021-12-22T12:44:50Z"
}
log/gitlab/gitlab-workhorse/#4000000061c31f28343253dc.u:11374:
{
"correlation_id":"ODWhqqnLNH2",
"error":"error while removing EXIF",
"level":"error",
"method":"POST",
"msg":"error",
"time":"2021-12-22T12:44:50Z",
"uri":"/uploads/user"
}
log/gitlab/gitlab-workhorse/#4000000061c343b71091c0ac.u:4:
{
"correlation_id":"OKWMhVphvr5",
"duration_ms":0,
"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused",
"level":"error",
"method":"GET",
"msg":"error",
"time":"2021-12-22T12:51:32Z",
"uri":"/help"
}
Results of GitLab environment info:
root#env:/# gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.1
Redis Version: 5.0.9
Git Version: 2.29.0
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.6.3
Revision: 857c6c6a6a9
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.9
URL: https://my_host.com
HTTP Clone URL: https://my_host.com/some-group/some-project.git
SSH Clone URL: ssh://git#my_host.com:60000/some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.13.0
Repository storage paths:
- default: /storage/data/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check:
root#env:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
11/1 ... yes
5/2 ... yes
11/3 ... yes
11/4 ... yes
11/5 ... yes
11/6 ... yes
11/7 ... yes
11/8 ... yes
11/9 ... yes
11/10 ... yes
11/11 ... yes
11/12 ... yes
11/13 ... yes
11/14 ... yes
11/15 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 14
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
I have already written a bug ticket for GitLab but I haven't got any feedback yet: https://gitlab.com/gitlab-org/gitlab/-/issues/349149
I would be very grateful if somebody could provide some hits how I can recover my system.
I'm trying to create a shared bare repository from an existing project so that multiple developers within the same group can have access. I have cloned the repository located in my project's directory to a bare shared repository in another centralized directory on the same machine and am able to create a clone of bare repo myself, however the clone fails with a fatal: repository does not exist error when logged in as another user within the group.
I've tried following the instructions on the git website for creating a bare repository and for cloning a repository.
https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server
https://git-scm.com/docs/git-clone
Based on the instructions, I used the following commands to create the bare repository and make it shared.
$ git clone --bare /path/to/my_dir/my_project /path/to/repos/project.git
$ cd /path/to/repos/project.git
$ git init --bare --shared
I am able to successfully make a clone using the following commands when I'm logged in as myself.
$ cd /path/to/their_dir
$ git clone /path/to/repos/project.git their_project
However, when logged in as a different user, the following error occurs when trying to use the same commands, even though I can do an ls on and cd into the project.git directory as the other user.
fatal: repository '/path/to/repos/project.git' does not exist
As far as permissions go, my account is listed as the owner of the repository and it's files, and the other user belongs to the group that is associated with the files.
$ ls -l /path/to/repos
drwxrws---. 6 me devs 4096 Jul 24 12:53 project.git
$ ls -l /path/to/repos/project.git
drwx------. 2 me devs 4096 Jul 24 12:46 branches
-rw-rw----. 1 me devs 177 Jul 24 12:53 config
-rw-------. 1 me devs 73 Jul 24 12:46 description
-rw-------. 1 me devs 23 Jul 24 12:46 HEAD
drwx------. 2 me devs 4096 Jul 24 12:52 hooks
drwx------. 2 me devs 4096 Jul 24 12:46 info
drwx------. 104 me devs 4096 Jul 24 12:46 objects
-rw-------. 1 me devs 98 Jul 24 12:46 packed-refs
drwxrws---. 4 me devs 4096 Jul 24 12:46 refs
How can I fix this so that another user can clone the repository while logged in under his or her account?
I am running Jenkins as a docker container, and have installed the NodeJS plugin and followed thoroughly the setup instructions. When I try to run a script using node, I get the following error:
/tmp/jenkins9123978873441132802.sh: line 1: node: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I checked the docker volume, the node bin is where it should be and is executable is there and it works fine when I run it from my host server:
user#server:/data/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin$ ./node --version
v9.2.0
I modified my build script to explore a bit further the problem:
echo $PATH
cd /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
ls -all
./node --version
node --version
npm --version
and look how strange this is:
Building in workspace /var/jenkins_home/workspace/release
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Done
Adding all registry entries
copy managed file [Main config] to file:/var/jenkins_home/workspace/release#tmp/config69012336710357692tmp
[release] $ /bin/sh -xe /tmp/jenkins6243047436861395796.sh
+ echo /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
/var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
+ cd /var/jenkins_home/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
+ ls -all
total 34112
drwxr-xr-x 2 jenkins jenkins 4096 Nov 20 16:16 .
drwxr-xr-x 6 jenkins jenkins 4096 Nov 20 16:16 ..
-rwxrwxrwx 1 jenkins jenkins 34921762 Nov 14 20:33 node
lrwxrwxrwx 1 jenkins jenkins 38 Nov 20 16:16 npm -> ../lib/node_modules/npm/bin/npm-cli.js
lrwxrwxrwx 1 jenkins jenkins 38 Nov 20 16:16 npx -> ../lib/node_modules/npm/bin/npx-cli.js
+ ./node --version
/tmp/jenkins6243047436861395796.sh: line 1: ./node: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
The node executable is present, and it's executable (+x). The path is correctly set, but the build still fails.
This is because the path to the node binary
/data/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/latest/bin
Does not exist on your shell path.
You should edit Jenkins' variables to adjust your PATH.
I have a main Git bare repository in my Dropbox folder and I'm using a headless Debian server which runs Jenkins to build my projects.
Until now I've been using this approach to build my projects: I created a bare git repository on my headless server and configured the Jenkins job to pull the sources from this repository with the following URL: /var/cache/git/project. So when I want Jenkins to build my changes, I have to push them to this repository. This works fine, but it's not very convenient because I have to manually push to two repositories (Dropbox and Debian) and also when I want to add a new project, I need to create a new repository on the Debian server.
So I decided to configure the Jenkins jobs to pull directly from the Dropbox repositories. I've set up Dropbox on my Debian server at /home/jardo/Dropbox.
But when I try to configure the Jenkins job to pull drom URL /home/jardo/Dropbox/git/project.git, I get the following error:
Failed to connect to repository : Command "git ls-remote -h /home/jardo/Dropbox/git/project.git HEAD" returned status code 128:
stdout:
stderr: fatal: '/home/jardo/Dropbox/git/project.git' does not appear to be a git repository
fatal: The remote end hung up unexpectedly
How come Jenkins is able to clone from /var/cache/git/project, but not from my Dropbox folder? Dropbox is completely synced and the repository content looks ok. I've set permissions on the whole repository recursively to 777 and that didn't help.
Here is the output of ls -l on my Dropbox repository:
-rwxrwxrwx 1 jardo jardo 104 Dec 15 21:10 config
-rwxrwxrwx 1 jardo jardo 73 Dec 15 21:10 description
-rwxrwxrwx 1 jardo jardo 23 Dec 15 21:10 HEAD
drwxrwxrwx 2 jardo jardo 4096 Dec 24 13:16 hooks
drwxrwxrwx 2 jardo jardo 4096 Dec 24 13:01 info
drwxrwxrwx 150 jardo jardo 4096 Dec 24 13:01 objects
drwxrwxrwx 4 jardo jardo 4096 Dec 24 12:59 refs
It turned out that user Jenkins could not access the GIT repository, which was owned by user Jardo, even when I tried to set its permissions recursively to 777. The solution was to setup Dropbox also for user Jenkins.
I setup svn on my local system /svn/repos/myproject by following this tutorial. I'm able to view the repo in browser.
But when it try to import new project I couldn't through svn client ( rapid svn ) it shows following error:
Execute: Import
Error while performing action:
Can't open file '/svn/repos/myproject/db/txn-current-lock': Permission denied
Svn directory permissions:
→ ls -l /svn
total 12
drwxrwxr-x 2 root root 4096 Feb 15 12:09 permissions
drwxrwxr-x 4 apache apache 4096 Feb 15 12:09 repos
drwxrwxr-x 2 root root 4096 Feb 15 12:09 users
Repo directory:
→ ls -l
total 8
drwxrwxr-x 3 root root 4096 Feb 15 12:09 conf
drwxrwxr-x 7 apache apache 4096 Feb 15 12:09 myproject
How to solve this issue?
I've given 777 permission to repos directory which solved this issue. But i got another issue like Couldn't perform atomic initialization.
I think this is due to incompatible sqlite version with subversion we're using, this can be solved by updating svnadmin command,
svnadmin create --pre-1.6-compatible --fs-type fsfs /svn/repos/myproject