Error uploading artifacts to coordinator - gitlab

Question

I’ve been having some fun setting up GitLab and after spending quite a while hacking away at it, I’ve become relatively used to setting it up, now having done that on two machines, the second time around with much more ease than originally…
However, I am faced with a rather large problem, on both machines: My CI pipeline is broken. Somehow, somewhere, my setup is providing a 403 to artifacts once builds are completed, meaning that each and every job that ever technically succeeds will only be doomed to fail…
I've been scavenging the interwebs for answers but I haven't found much that has been useful.
I upgraded GitLab CE to 10.1.4 minutes ago, as well as GitLab-runner to 10.1.0, the latest packages available to me through apt on the more important of the two machines, running a newer version of Ubuntu than the other - 17.04 zesty on the “beast” compared to 16.10 yakkety on “q2”.
Both gitlab-runner registrations use shell for execution.
The relevant output of the CI job is as follows:
Cloning repository...
Cloning into '/[clonepath]'...
Checking out 8319d586 as master...
Skipping Git submodules setup
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
$ mvn -B install
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.204 s
[INFO] Finished at: 2017-11-18T05:45:08+01:00
[INFO] Final Memory: 27M/640M
[INFO] ------------------------------------------------------------------------
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
mesg: ttyname failed: Inappropriate ioctl for device
Uploading artifacts...
target/*.jar: found 1 matching files
ERROR: Uploading artifacts to coordinator... forbidden id=35 responseStatus=403
Forbidden status=403 Forbidden token=sP9oHykF
FATAL: permission denied
ERROR: Job failed: exit status 1
I run GitLab under an Apache2 Vhost subdomain, mostly for aesthetic and omitting of the port following the host, i.e. 8080 for unicorn, since there are other sites running on Apache.
These are the configured options within my gitlab.rb:
gitlab_rails['trusted_proxies'] = [ '127.0.0.1' ]
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
nginx['enable'] = false
Setting the values in either of the following options/values as such
web_server['username'] = 'www-data'
web_server['group'] = 'www-data'
produces an error on reconfiguration:
Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- package (0.1.0)
- registry (0.1.0)
- consul (0.0.0)
- gitlab (0.0.1)
- runit (0.14.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
Converging 408 resources
* directory[/etc/gitlab] action create (up to date)
* directory[Create /var/opt/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* group[Webserver user and group] action create (up to date)
* user[Webserver user and group] action create
================================================================================
Error executing action `create` on resource 'user[Webserver user and group]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '8'
---- Begin output of ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] ----
STDOUT:
STDERR: usermod: user www-data is currently used by process 2656
---- End output of ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] ----
Ran ["usermod", "-s", "/bin/false", "-d", "/var/opt/gitlab/nginx", "www-data"] returned 8
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/account.rb
38: user params[:name] do
39: username username
40: shell params[:shell]
41: home params[:home]
42: uid params[:uid]
43: gid params[:ugid]
44: system params[:system]
45: supports params[:user_supports]
46: action params[:action]
47: end
48: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/definitions/account.rb:38 :in `block in from_file'
user("Webserver user and group") do
params {:action=>nil, :username=>"www-data", :uid=>nil, :ugid=>"www-data", :groupname=>"www-data", :gid=>nil, :shell=>"/bin/false", :home=>"/var/opt/gitlab/nginx", :system=>true, :append_to_group=>true, :group_members=>["www-data"], :user_supports=>{:manage_home=>false}, :manage=>true, :name=>"Webserver user and group"}
action [:create]
supports {:manage_home=>false}
retries 0
retry_delay 2
default_guard_interpreter :default
username "www-data"
gid 33
home "/var/opt/gitlab/nginx"
shell "/bin/false"
system true
iterations 27855
declared_type :user
cookbook_name "gitlab"
recipe_name "web-server"
end
Platform:
---------
x86_64-linux
Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 04 seconds
And as for Apache, here’s the SSL-enabled Vhost:
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName [host]
ServerAdmin [email]
DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public
ServerSignature Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode
<Location />
Order deny,allow
Allow from all
Require all granted
ProxyPassReverse http://127.0.0.1:8181/
ProxyPassReverse http://[host]/
RequestHeader set X-Forwarded-Ssl 'on'
</Location>
RewriteEngine on
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR]
RewriteCond %{REQUEST_URI} ^/uploads/.*
RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE]
SSLCertificateFile /etc/letsencrypt/live/[host]/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/[host]/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>
Any idea what’s going on? I haven’t dug the Apache log information yet since it probably won’t be Apache as the request goes straight to gitlab-worker (8181). What logs should I check for that, if necessary?
Thankyou for your time.

Answer 1

This isn't a particularly helpful answer, as the solution has little explanation for its workings.
The configurations I had have remained the same as per above, but the runner I had installed, I removed the config of, rm /etc/gitlab-runner/config.toml, and then proceeded to remove the package from the machine, apt purge gitlab-runner. (gitlab-ci-multi-runner is another package that is available but does not appear to be up to date with GitLab 10 - returns a 404 rather than connecting to the node).
I reinstalled the runner, apt install gitlab-runner, and then registered it - gitlab-runner register. The key thing to note here is that during registration, I used my FQDN, as in https://git.example.com rather than any local address such as http://localhost:8080 or http://localhost:8181 (unicorn, gitlab-workhorse, respectively). And yes, I am running my runners on my local machine. Hazardous, but I have too much trust in my team. That may be our downfall, ignorant systems administration is key to success.