I try to use Gitlab CI / CD with GilLab Runner for the first time. GitLab is a self-hosted version.
I have a git project in GitLab. By default users must use SSH with private key to use git commands.
But I would like to specify that access control is set to both : SSH and HTTPS.
For this project I create a simple .gitlab-ci.yml file (with a single test step) and a GitLab runner using the token specify by GitLab.
When I run the job, the following error occurs :
Running with gitlab-runner 12.1.0 (de7731dd)
on BK_runner_Test3 hcVfxLhx
Using Shell executor...
Running on ns344345...
Fetching changes...
Reinitialized existing Git repository in /home/gitlab-runner/builds/hcVfxLhx/0/XXXXXXXXX/test/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.xxxxxxxx.com/XXXXXXXX/test.git/': The requested URL returned error: 500
ERROR: Job failed: exit status 1
Finally, in the Apache logs, I found a request with error 401 and 500.
So, I think, at that time, the job try to clone the project. So I try the following command :
git clone https://gitlab-ci-token:[MASKED]#gitlab.xxxxxxxx.com/XXXXXXXX/test.git/
And the result is an error 401 !
Of course [MASKED] was replaced with the right token, used to declare tue runner.
So why this error 401 ? What did I miss in the configuration in order to authorize jobs using git commands ?
More information from log file /var/log/gitlab/gitlab-rails/production.log
Started GET "/XXXXXXXX/test.git/info/refs?service=git-upload-pack" for 111.222.333.444 at 2019-08-08 15:39:05 +0200
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"XXXXXXXX", "project_id"=>"test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 15ms (Views: 0.7ms | ActiveRecord: 1.4ms | Elasticsearch: 0.0ms)
Started GET "/XXXXXXXX/test.git/info/refs?service=git-upload-pack" for 111.222.333.444 at 2019-08-08 15:39:05 +0200
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"XXXXXXXX", "project_id"=>"test.git"}
Completed 500 Internal Server Error in 12ms (ActiveRecord: 3.0ms | Elasticsearch: 0.0ms)
JWT::DecodeError (Nil JSON web token):
It looks like the Git request is being authenticated/authorized correctly in GitLab Rails, but when it's passed over to GitLab Workhorse to actually send the data to the client, some authorization fails.
Check workhorse logs to see if there's more information. That will be in /var/log/gitlab/gitlab-workhorse/ by default in an Omnibus installation. I'm not exactly sure what would cause this but for some reason the header that authorizes workhorse isn't present.
Make sure your GitLab version is reasonably close to your runner version. I see you're running Runner 12.1 so GitLab should also be 12.1.
Related
Context
After creating a general post-receive for a GitLab server, I noticed it gets triggered directly after a new commit is detected in any repository. However, I would like the post-receive script to do something with the build status of the GitLab Runner CI on the commit that triggered the post-receive script.
Approach
Based on this question and answer, I wrote a post-receive script that gets the commit and repository, and I tried to get the build status from that commit from within the GitLab docker:
#!/bin/bash
read oldrev newrev refname
echo "Previous Commit: $oldrev"
echo "New/latest Commit: $newrev"
echo "Repository name: $refname"
# Get build status of $newrev
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/17/$refname/commits/$newrev/statuses
However, that API call does not work from within the Docker environment (which is from where the post-receive script runs).
Docker GitLab Build Status File locations
I also found the build status badges inside the Docker, they are located in: /opt/gitlab/embedded/service/gitlab-rails/public/assets/. However I do not (yet) know how to decode their filenames. For example, the build status badge accompanying Job #3, of commit: 9514d16aafc1d741ba6a9ff47718d632fa8d435b has filename: icons-6d7d4be41eac996c72b30eac2f28399ac8c6eda840a6fe8762fc1b84b30d5a2d.svg. Basically I do not know to which commit/repository that build status badge belongs.
On the other hand, I have found the location of the job logs in the hashed path of the repository:
/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2021_10_09/1/1/job.log
/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2021_10_14/3/3/job.log
Which each in turn contain their respective commit and branch as:
Checking out 9514d16a as master...
So in principle I could scan the repository path and accompanying job logs until I found the job.log that contains the commit of the post-receive script (for e.g. 5 minutes, to account for the delay between the commit and the starting of the GitLab Runner CI), and then search for the build status output in that job.log (e.g. Job succeeded) (for e.g. 60 minutes to allow for long jobs). However, that seems like a convoluted work-around.
Question
Hence, I was wondering, *Is there a better/faster/robuster method to get the GitLab Runner CI build status of the commit that triggered the general post-receive script of a GitLab server, inside that triggered instance/run of the post-receive script?
Occasionally the first upload of artifacts during a GitLab pipeline fail.
I'm getting the following error message in the logs:
2019-08-01 13:43:14,149 [http-nio-8082-exec-187] [ERROR]
(o.j.s.b.p.t.FilePersistenceHelper:87) - Failed moving
'path_to_artifactory\filestore_pre\dbRecord123.bin' to
'path_to_artifactory\filestore\5e\5ecc5f719b4442b9b04f9010646d34917aca8ca2'.
Access to file denied null 2019-08-01 13:43:14,149
[http-nio-8082-exec-187] [ERROR] (o.a.w.s.RepoFilter :251) - Upload
request of products-stage-qa:file_to_upload failed due to {}
java.nio.file.AccessDeniedException: Failed to persist file with sha1:
5ecc5f719b4442b9b04f9010646d34917aca8ca2
This seems to happen only during builds, but not during other uploads directly by a user.
It doesn't happen all the time, and only on first tries. But I haven't found any logic when the first try fails or succeeds. It doesn't seem to have anything to do with file types or the like. I can't really determine if it has anything to do with network speeds though since I only have access to part of the infrastructure.
I found an open ticket with the same error message, but only for Conan and for us it only happens with ivy repositories
We are using Artifactory 6.9.1 and GitLab 12.0.3 starter
This looks to be a permission issue. You are getting an error message that states that the move failed due to "Access to file denied".
You can try to log in to the server using the "artifactory" user and manually move the file called "path_to_artifactory\filestore_pre\dbRecord123.bin" to "path_to_artifactory\filestore\5e\5ecc5f719b4442b9b04f9010646d34917aca8ca2" and see if you have any issues with this. To log in to the server with the "artifactory" user you can use the command "sudo -s -u artifactory".
You will also need to make sure that all filestore and its subdirectories are owned by the "artifactory" user and have the correct permissions.
Hope this helps.
I have a GitLab project which is set up as follows:
myserver.com/SuperGroup/SubGroup/SubGroupProject
The tree of the following project is a top-level txt file and a txt file within a directory. I get the tree from the GitLab API with:
myserver.com/api/v4/projects/1/repository/tree?recursive=true
[{"id":"aba61143388f605d3fe9de9033ecb4575e4d9b69","name":"myDirectory","type":"tree","path":"myDirectory","mode":"040000"},{"id":"0e3a2b246ab92abac101d0eb2e96b57e2d24915d","name":"1stLevelFile.txt","type":"blob","path":"myDirectory/1stLevelFile.txt","mode":"100644"},{"id":"3501682ba833c3e50addab55e42488e98200b323","name":"top_level.txt","type":"blob","path":"top_level.txt","mode":"100644"}]
If I request the contents for top_level.txt they are returned without any issue via:
myserver.com/api/v4/projects/1/repository/files/top_level.txt?ref=master
However I am unable to access myDirectory/1stLevelFile.txt with any API call I try. E.g.:
myserver.com/api/v4/projects/1/repository/files/"myDirectory%2F1stLevelFile.txt"?ref=master
and,
myserver.com/api/v4/projects/1/repository/files/"myDirectory%2F1stLevelFile%2Etxt"?ref=master
Results in:
Not Found The requested URL /api/v4/projects/1/repository/files/myDirectory/1stLevelFile.txt was not found on this server.
Apache/2.4.25 (Debian) Server at myserver.com Port 443
myserver.com/api/v4/projects/1/repository/files/"myDirectory/1stLevelFile.txt"?ref=master and,
myserver.com/api/v4/projects/1/repository/files?ref=master&path=myDirectory%2F1stLevelFile.txt
Results in:
error "404 Not Found"
The versions of the components are:
GitLab 10.6.3-ee
GitLab Shell 6.0.4
GitLab Workhorse v4.0.0
GitLab API v4
Ruby 2.3.6p384
Rails 4.2.10
postgresql 9.6.8
According to my research there was a similar bug which was fixed with the 10.0.0 update.
I also added my ssh-key although I doubt it has any effect, following this advice with the same issue in php.
Solution:
I eventually solved it by adjusting the apache installed on the server.
Just follow these instructions: https://gitlab.com/gitlab-org/gitlab-ce/issues/35079#note_76374269
According to your code, I will go thinking you use curl.
If it is the case, why are you adding double quotes to your file path ?
The doc do not contains it.
Can you test it like that please ?
curl --request GET --header 'PRIVATE-TOKEN: XXXXXXXXX' myserver.com/api/v4/projects/1/repository/files/myDirectory%2F1stLevelFile%2Etxt?ref=master
First of all my gitlab connection was broken by error 500 and 502. The server was unattended for a while. So i decided to reinstall:
#gitlab-ctl cleanse
#apt-get purge gitlab-ce
Installation:
https://about.gitlab.com/downloads/#ubuntu1404
After the gitlab was reachable and i could create a project and clone it.
After i tried to push a new file to my repo and got following message:
git add .
git commit -m "inital"
git push --set-upstream origin master
error: Cannot access URL http://<username>#gitlab.domain.de:<port>/group/project.git/, return code 22
fatal: git-http-push failed
error: failed to push some refs to 'http://<username>#gitlab.domain.de:<port>/group/project.git/'
The gitlab-check gives this information:
#gitlab-rake gitlab:check
everything was green, besides violett ones:
Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Why i can clone but not push? I searched in several log-files, but got nothing useful.
I installed gitlab 8.2.0 from source and followed the instructions on https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.
When I run the checks with
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
I get the following error:
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: FAILED. code: 301
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Here is my config.yml file from /home/git/gitlab-shell:
user: git
gitlab_url: "http://www.example.com/gitlab"
http_settings:
#ca_path: /etc/ssl/certs
self_signed_cert: true
repos_path: "/home/git/repositories/"
auth_file: "/home/git/.ssh/authorized_keys"
redis:
bin: "/usr/bin/redis-cli"
namespace: resque:gitlab
socket: "/var/run/redis/redis.sock"
log_level: INFO
audit_usernames: false
Note that I am using the relative URL feature to run gitlab on www.example.com/gitlab. Also the server is behind a reverse proxy that terminates https to http if this should be of any relevance for this problem.
I also cant' push or pull any repositories. When I try to I get the following error:
git pull https://www.example.com/gitlab/test/testproject.git
Username for 'https://www.example.com': test
Password for 'https://test#www.example.com':
fatal: Couldn't find remote ref HEAD
Unexpected end of command stream
I also can't add any files to any project via the webUI. If I try to do so I get the following error
Commit was rejected by pre-receive hook
The log for this from production.log looks as follows:
Sarted POST "/gitlab/test/testprojekt/create/master" for 147.86.8.115 at 2015-11-26 15:24:43 +0100
Processing by Projects::BlobController#create as JSON
Parameters: {"new_branch"=>"master", "create_merge_request"=>"1", "commit_message"=>"test", "file"=># <ActionDispatch::Http::UploadedFile:0x007f776230faf8 #tempfile=# <Tempfile:/tmp/RackMultipart20151126-7584-308za1>, #original_filename="test2.txt", #content_type="text/plain", #headers="Content- Disposition: form-data; name=\"file\"; filename=\"test2.txt\"\r\nContent-Type: text/plain\r\n">, "namespace_id"=>"test", "project_id"=>"testprojekt", "id"=>"master"}
Completed 200 OK in 628ms (Views: 0.3ms | ActiveRecord: 3.3ms)
Started GET "/gitlab/test/testprojekt/new/master" for 147.86.8.115 at 2015-11-26 15:24:44 +0100
Processing by Projects::BlobController#new as HTML
Parameters: {"namespace_id"=>"test", "project_id"=>"testprojekt", "id"=>"master"}
Completed 200 OK in 85ms (Views: 49.6ms | ActiveRecord: 6.0ms)
I think you could try switching gitlab_url: "http://www.example.com/gitlab" to https -> gitlab_url: "https://www.example.com/gitlab"
Not sure if its a complete fix but its a recomendations.
EDIT:
config.yml line 4 should look like
ca_path: "/etc/ssl/certs"