Gitlab: pushes registering with repo, but pipelines not running and projects dashbaord 'last updated' is not changed - gitlab

When we push to our repository, we expect a pipeline to run. However, the pipelines have stopped starting automatically when we push.
In addition, when we try to start the pipeline manually, not all the tags and branches are showing in the dropdown list of tags and branches to choose from. When we browse the repository in Gitlab, we can see the branches and the pushed commits.
Finally, in the /dashboard/projects page, the 'last updated' date of the project is out of date, saying "yesterday" rather than "10 mins ago" (which is what shows when viewing the repository within the project.
We recently migrated server and so would expect that there is some migration issue going on here. Does anyone have any ideas where to look to solve this problem (i.e. what sub-systems could be not working/configured correctly to produce this behaviour)?
Gitlab version: 9.4.2
Run with Docker using: https://hub.docker.com/r/gitlab/gitlab-ce/
Update
I tailed the logs while pushing to the repository, what follows is a chunk of logs around that time (starting with the SSH connection for the push). Potentially the 404 around prometheus is interesting, but I'm not sure that's unexpected (we're not using it):
==> /var/log/gitlab/sshd/current <==
2017-08-01_17:05:16.86559 Accepted publickey for git from (removed) port 57680 ssh2: RSA SHA256:(removed)
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2017-08-01 17:05:17 +0000
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2017-08-01T17:05:17.088866 #2286] INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.01170
I, [2017-08-01T17:05:17.089030 #2286] INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/products/preside-ext-ems.git> for user with key key-2.
==> /var/log/gitlab/sshd/current <==
2017-08-01_17:05:17.20480 Received disconnect from x.x.x.x port 57680:11: disconnected by user
2017-08-01_17:05:17.20483 Disconnected from x.x.x.x port 57680
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2017-08-01 17:05:18 +0000
Processing by MetricsController#index as HTML
Filter chain halted as :validate_prometheus_metrics rendered or redirected
Completed 404 Not Found in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)
Started POST "/api/v4/jobs/request" for 172.17.0.1 at 2017-08-01 17:05:18 +0000
==> /var/log/gitlab/gitlab-workhorse/current <==
2017-08-01_17:05:18.16504 gitlab.mycompany.com # - - [2017-08-01 17:05:18.158505651 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)" 0.006484
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [01/Aug/2017:17:05:18 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/jobs/request" for 172.17.0.1 at 2017-08-01 17:05:23 +0000
==> /var/log/gitlab/gitlab-workhorse/current <==
2017-08-01_17:05:23.16534 gitlab.mycompany.com # - - [2017-08-01 17:05:23.159064793 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)" 0.006235
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [01/Aug/2017:17:05:23 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)"

Not exactly an answer - but I have wiped out the server and rebuilt from scratch. Manually recreating each project and importing the repositories for each project.
A royal PITA, but everything is working as expected.
I can only guess that either something was setup on the host server that was causing issues (I did a clean install on the host to start again), or that there was something about simply copying over all our configuration and data dirs from the old server to the new server that caused issues (seems unlikely).
Not much help I'm afraid :(

Related

Does Gitlab EE Have Audit Logs for "git clone", and "git pull" commands?

The audit events page of gitlab says that I can find "Project repository was downloaded" action in Project > Settings > Audit Events.
So I tried running git clone http://gitlab.example.com/testauditlog.git to download one of my projects. But then I can't find anything related to the download in the audit events. Why does this happen?
The only logs belonging to git clone I found is in /var/log/gitlab/nginx/gitlab_access.log
172.17.0.1 - - [03/Jan/2020:03:28:56 +0000] "GET /testauditlog.git/info/refs?service=git-upload-pack HTTP/1.1" 200 254 "" "git/2.21.0 (Apple Git-122.2)"
172.17.0.1 - - [03/Jan/2020:03:28:56 +0000] "POST /testauditlog.git/git-upload-pack HTTP/1.1" 200 949 "" "git/2.21.0 (Apple Git-122.2)"
But this log doesn't say which account cloned the repository. So it is not that useful for the compliance team in my company.
I used gitlab/gitlab-ee at dockerhub with a 30-day evaluation license to try the audit events feature of gitlab.

GitLab Health Check without token

I've got GitLab 10.5.6. I'd like to use Health Check information in my monitoring system. I can configure it by using Health Check endpoints with health check access token, but as this solution is depracated, I want to use IP whitelist. And I have some problems with it.
According to this article https://docs.gitlab.com/ee/administration/monitoring/ip_whitelist.html I edited /etc/gitlab/gitlab.rb and added this line (as this GitLab was installed around version 7 or even older I think):
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1', 'X.X.X.X', 'Y.Y.Y.Y']
where X.X.X.X is IP of my computer and Y.Y.Y.Y is IP of server with GitLab. After it I executed reconfiguration (gitlab-ctl reconfigure). And started tests... Below logs are from production.log file.
Execution of curl http://127.0.0.1:8888/-/readiness on server Y.Y.Y.Y returns proper JSON with expected data:
Started GET "/-/readiness" for 127.0.0.1 at 2018-03-24 20:01:31 +0100
Processing by HealthController#readiness as /
Completed 200 OK in 27ms (Views: 0.6ms | ActiveRecord: 0.5ms)
Execution of curl http://Y.Y.Y.Y:8888/-/readiness on server Y.Y.Y.Y returns error:
Started GET "/-/readiness" for Y.Y.Y.Y at 2018-03-24 21:20:04 +0100
Processing by HealthController#readiness as /
Filter chain halted as :validate_ip_whitelisted_or_valid_token! rendered or redirected
Completed 404 Not Found in 2ms (Views: 1.0ms | ActiveRecord: 0.0ms)
Accessing address http://Y.Y.Y.Y:8888/-/readiness through Firefox browser on computer X.X.X.X returns error:
Started GET "/-/readiness" for X.X.X.X at 2018-03-24 20:03:04 +0100
Processing by HealthController#readiness as HTML
Filter chain halted as :validate_ip_whitelisted_or_valid_token! rendered or redirected
Completed 404 Not Found in 2ms (Views: 0.8ms | ActiveRecord: 0.0ms)
Accessing address http://Y.Y.Y.Y:8888/-/readiness?token=ZZZZZZZZZZZZZ through Firefox browser on computer X.X.X.X returns proper JSON with expected data.
I don't have any idea what I can check more. Maybe there's lack of any more configuration in /etc/gitlab/gitlab.rb as it's quite old GitLab instance.

Docker on RHEL 7 in AWS can’t pull images

We're looking to incorporate docker on our project and I'm having some difficulty.
I installed Docker via the instructions here: https://docs.docker.com/engine/installation/linux/rhel/ and had no problems. However, when I try to pull any images from docker hub, it just hangs and nothing ever happens:
┌─[user#hostname]─(~)
└─[11:53]$ docker pull alpine:latest
Here's my docker version info:
┌─[user#hostname]─(~)
└─[11:53]$ docker version
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64
This is a redhat 7.2 instance in a VPC. I'm using another rhel 7.2 instance to handle NAT through iptables. This issue is the first connectivity problem I've had.
I read elsewhere that specifying an http proxy would solve the issue, so I set up a squid proxy on my nat server, but to no avail. I followed the instructions here to set the proxy for docker to use: https://docs.docker.com/engine/admin/systemd/#/http-proxy
The proxy is getting the requests, but docker still can't pull any images. Here's the out put of my squid logs:
1467475616.376 110 10.0.0.50 TCP_MISS/200 3250 CONNECT registry-1.docker.io:443 - HIER_DIRECT/52.71.80.248 -
1467475616.415 35 10.0.0.50 TCP_MISS/200 4444 CONNECT auth.docker.io:443 - HIER_DIRECT/52.204.167.32 -
I'm running out of things to try here. Any help would be greatly appreciated.
EDIT
Got a new error! After tweaking the squid proxy configuration, I now get this error when attempting to pull an image:
┌─[user#hostname]─(~)
└─[15:15]$ sudo docker pull alpine
Using default tag: latest
Pulling repository docker.io/library/alpine
Error while pulling image: Get https://index.docker.io/v1/repositories/library/alpine/images: Service Unavailable
Here's the output of ping:
┌─[user#hostname]─(~)
└─[15:15]$ ping index.docker.io
PING us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com (54.173.200.203) 56(84) bytes of data.
^C
--- us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
And wget can download a json file from the url in the docker error message:
┌─[user#hostname]─(~)
└─[15:17]$ wget https://index.docker.io/v1/repositories/library/alpine/images
--2016-07-02 15:17:20-- https://index.docker.io/v1/repositories/library/alpine/images
Resolving index.docker.io (index.docker.io)... 54.173.200.203, 52.22.190.106, 52.203.138.237
Connecting to index.docker.io (index.docker.io)|54.173.200.203|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘images’
[ <=> ] 6,900 --.-K/s in 0s
2016-07-02 15:17:20 (185 MB/s) - ‘images’ saved [6900]

Network printer doesn't accept job from Debian Linux, no errors in error_log

There is a shared printer at my workplace. We send jobs and then go to the printer and authenticate, so printer prints your documents only when you present at it. Periodically, we change domain passwords, so I also have to change it in /etc/cups/printers.conf (windows users just change domain password). So, that's how it works.
But, suddenly, it stop receive my jobs. When I send job I have no errors and have this:
sudo tail /var/log/cups/access_log
localhost - - [14/Apr/2015:12:15:14 +0300] "POST /printers/Generic-PCL-6-PCL-XL HTTP/1.1" 200 499 Create-Job successful-ok
localhost - - [14/Apr/2015:12:15:14 +0300] "POST /printers/Generic-PCL-6-PCL-XL HTTP/1.1" 200 1273674 Send-Document successful-ok
localhost - - [14/Apr/2015:12:17:59 +0300] "POST / HTTP/1.1" 200 183 Renew-Subscription successful-ok
On cups page in browser it shows state for job - "Pending since (date/time)".
It seems like job was sent successfully, but when I came to printer I've got nothing and no job in my queue. Our IT support fix problems only for Windows users and who on Linux - on their own. So, I don't know what to do and what logs I should inspect. Please, help.
Probably, some updates broke it down. But I have found another solution - I add printer not via samba, but via lp and it doesn't ask username/password:
cat /etc/cups/printers.conf
# Printer configuration file for CUPS v1.5.3
# Written by cupsd
# DO NOT EDIT THIS FILE WHEN CUPSD IS RUNNING
<DefaultPrinter KonicaMinolta>
UUID urn:uuid:0f60c08a-ecfb-326a-421c-86aa3519147b
Info MyCompany Office printer
Location WestCorridor
MakeModel Generic PostScript Printer Foomatic/Postscript (recommended)
DeviceURI lpd://Company_printer_server_address/lp
State Idle
StateTime 1429265417
Type 8433692
Accepting Yes
Shared Yes
JobSheets none none
QuotaPeriod 0
PageLimit 0
KLimit 0
OpPolicy default
ErrorPolicy stop-printer
</Printer>
If somebody can provide another solution or some explanation why it is so, I will be glad to see.
As far as debugging you can view more data in your CUPS logs if you edit your /etc/cups/cupsd.conf file, find the section "loglevel" change "info" to "debug"
Then you should restart CUPS with:
/etc/init.d/cups restart
Then your log will be in
/var/log/cups/error_log

Couchdb Logging

Due to application requirements, I have an externally accessible CouchDB instance. I would like to see what IP addresses are attempting to authenticate with my database. By checking the couchdb.log file, I can see failed authentication attempts. They look similar to this.
[Mon, 29 Sep 2014 13:43:32 GMT] [info] [<0.28472.7>] 127.0.0.1 - - GET
/offline_master/ 401
However, no matter where I connect from, it seems that the IP address that is logged is always 127.0.0.1. Am I mis-understanding how this works? I would really like to see the IP address that is attempting to connect.
The 127.0.0.1 is the address couchDB is bound to. It's there because you can set up couchdb to respond differently depending on what host name is being used.
The only way to get the client ip address is by turning the logging level to "debug". You can do this in the configuration page in futon.
You get records like this (client IP is on 1st line):
[Tue, 30 Sep 2014 00:14:27 GMT] [debug] [<0.451.4>] 'GET' / {1,1} from "192.168.1.52"
Headers: [{'Accept',"*/*"},
{'Host',"localhost:5984"},
{'User-Agent',"curl/7.30.0"}]
[Tue, 30 Sep 2014 00:14:27 GMT] [debug] [<0.451.4>] OAuth Params: []
[Tue, 30 Sep 2014 00:14:27 GMT] [info] [<0.451.4>] 127.0.0.1 - - GET / 200
Be careful with this. The debug logs are extremely verbose. It doesn't take long to fill up a hard drive.
It is possible to set log levels by module. The module you need to set is couch_httpd. Set the default for the rest to "error" or "fatal".
See: 3.6.2 Per module logging

Resources