I have an nginx cache server with a simple basic config, nothing special.
nginx version: openresty/1.13.6.1
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4)
built with OpenSSL 1.0.2k 26 Jan 2017
TLS SNI support enabled
log:
172.18.0.1 HIT -- - - 0.000 - - [06/Apr/2022:17:33:00 +0000] "HEAD /custom/xxxx/image/product/12.jpg HTTP/2.0" 200 0 "-" "curl/7.68.0" "-"
This file cache version is stored under custom folder /tmp/cache1.
What i want, is display the cache hash in the log file.
Eg. the cache hash for this file is:
/tmp/cache1/a/0/e6920f37af6df573815a02933c7f480a
And what i want to see:
172.18.0.1 HIT -- - - 0.000 - - [06/Apr/2022:17:33:00 +0000] "HEAD /custom/xxxx/image/product/12.jpg HTTP/2.0" | e6920f37af6df573815a02933c7f480a | 200 0 "-" "curl/7.68.0" "-"
Is it possible? Have any built-in variable to display this hash in the log files?
Related
I'm facing a problem about GET logs of schema registry. When I check the log4j properties I see it is configured as log4j.appender.file.File=${schema-registry.log.dir}/schema-registry.log which is working as intended (log files are located under /confluent-7.0.1/logs/).
My problem is there are also files under /var/log/. It seems that they are recorded in seperate files from week to week.
-rw------- 1 root root 160273230 Jan 2 12:02 messages
-rw------- 1 root root 1831024355 Dec 18 03:10 messages-20221218
-rw------- 1 root root 706439179 Dec 25 03:07 messages-20221225
-rw------- 1 root root 1158507310 Jan 1 03:06 messages-20230101
Content of these files are like that:
Dec 25 03:15:09 server_name bash: [2022-12-25 03:15:09,995] INFO 192.168.181.21 - kafkauser [25/Dec/2022:00:15:09 +0000] "GET /subjects/TOPIC_NAME-key/versions/latest HTTP/1.1" 200 178 "-" "-" GETsT (io.confluent.rest-utils.requests:62)
Dec 25 03:15:10 server_name bash: [2022-12-25 03:15:10,018] INFO 192.168.181.21 - kafkauser [25/Dec/2022:00:15:10 +0000] "GET /subjects/TOPIC_NAME-value/versions/latest HTTP/1.1" 200 2197 "-" "-" GETsT (io.confluent.rest-utils.requests:62)
Dec 25 03:15:10 server_name bash: [2022-12-25 03:15:10,078] INFO 192.168.181.20 - kafkauser [25/Dec/2022:00:15:10 +0000] "GET /subjects/TOPIC_NAME-key/versions/latest HTTP/1.1" 200 178 "-" "-" GETsT (io.confluent.rest-utils.requests:62)
Dec 25 03:15:10 server_name bash: [2022-12-25 03:15:10,098] INFO 192.168.181.20 - kafkauser [25/Dec/2022:00:15:10 +0000] "GET /subjects/TOPIC_NAME-value/versions/latest HTTP/1.1" 200 2197 "-" "-" GETsT (io.confluent.rest-utils.requests:62)
Is this logging happening because of schema registry or is it just part of the Linux system? I mean, is it result of network logging or schema registry logging? Either way, how can I make it stop or configure to be recorded at somewhere else? Thanks in advance.
I assume you have installed Confluent Platform in a way that uses systemctl? If so, then yes, journalctl will write to /var/log/messages via the process's stdout/stderr logs.
You need to disable the ConsoleAppender in the log4j file to stop this.
When we push to our repository, we expect a pipeline to run. However, the pipelines have stopped starting automatically when we push.
In addition, when we try to start the pipeline manually, not all the tags and branches are showing in the dropdown list of tags and branches to choose from. When we browse the repository in Gitlab, we can see the branches and the pushed commits.
Finally, in the /dashboard/projects page, the 'last updated' date of the project is out of date, saying "yesterday" rather than "10 mins ago" (which is what shows when viewing the repository within the project.
We recently migrated server and so would expect that there is some migration issue going on here. Does anyone have any ideas where to look to solve this problem (i.e. what sub-systems could be not working/configured correctly to produce this behaviour)?
Gitlab version: 9.4.2
Run with Docker using: https://hub.docker.com/r/gitlab/gitlab-ce/
Update
I tailed the logs while pushing to the repository, what follows is a chunk of logs around that time (starting with the SSH connection for the push). Potentially the 404 around prometheus is interesting, but I'm not sure that's unexpected (we're not using it):
==> /var/log/gitlab/sshd/current <==
2017-08-01_17:05:16.86559 Accepted publickey for git from (removed) port 57680 ssh2: RSA SHA256:(removed)
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2017-08-01 17:05:17 +0000
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2017-08-01T17:05:17.088866 #2286] INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.01170
I, [2017-08-01T17:05:17.089030 #2286] INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/products/preside-ext-ems.git> for user with key key-2.
==> /var/log/gitlab/sshd/current <==
2017-08-01_17:05:17.20480 Received disconnect from x.x.x.x port 57680:11: disconnected by user
2017-08-01_17:05:17.20483 Disconnected from x.x.x.x port 57680
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2017-08-01 17:05:18 +0000
Processing by MetricsController#index as HTML
Filter chain halted as :validate_prometheus_metrics rendered or redirected
Completed 404 Not Found in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)
Started POST "/api/v4/jobs/request" for 172.17.0.1 at 2017-08-01 17:05:18 +0000
==> /var/log/gitlab/gitlab-workhorse/current <==
2017-08-01_17:05:18.16504 gitlab.mycompany.com # - - [2017-08-01 17:05:18.158505651 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)" 0.006484
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [01/Aug/2017:17:05:18 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/jobs/request" for 172.17.0.1 at 2017-08-01 17:05:23 +0000
==> /var/log/gitlab/gitlab-workhorse/current <==
2017-08-01_17:05:23.16534 gitlab.mycompany.com # - - [2017-08-01 17:05:23.159064793 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)" 0.006235
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [01/Aug/2017:17:05:23 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-ci-multi-runner 9.4.1 (9-4-stable; go1.8.3; linux/amd64)"
Not exactly an answer - but I have wiped out the server and rebuilt from scratch. Manually recreating each project and importing the repositories for each project.
A royal PITA, but everything is working as expected.
I can only guess that either something was setup on the host server that was causing issues (I did a clean install on the host to start again), or that there was something about simply copying over all our configuration and data dirs from the old server to the new server that caused issues (seems unlikely).
Not much help I'm afraid :(
I think I am doing something wrong with HAProxy conf because my throughput drops to 25% in a real-world test done with HAProxy and one single AWS instance. Following is my relevant (extremely simple) configuration:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 30000
frontend localnodes
bind *:80
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
hash-type consistent
option httpchk /health
server w1 xx.xx.xx.xx:80 check id 1
I had enabled logging. A typical entry in log looks like this:
Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43908
[02/Dec/2016:09:29:05.940] localnodes nodes/w1 38/0/0/1/41 200 130 - -
---- 36/36/12/2/0 0/0 "GET /ep?key=123&message=XXQSYI HTTP/1.1" Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43920
[02/Dec/2016:09:29:05.941] localnodes nodes/web01 39/0/0/0/40 200 160
- - ---- 35/35/11/0/0 0/0 "GET /q1?key=123&val=123 HTTP/1.1" Dec 2 09:29:05 localhost haproxy[2782]: xx.xx.xx.xx:43933
[02/Dec/2016:09:29:05.955] localnodes nodes/web01 24/0/0/1/26 200 134
- - ---- 34/34/11/1/0 0/0 "GET /q1?key=123&val=123 HTTP/1.1"
My throughput is 25% of what a direct traffic to my instance would be. This is terrible performance. Am I doing something really wrong?
EDIT
Going down the log, some logs clearly show that time taken to reach server from HAProxy is too high
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39789 [02/Dec/2016:10:56:58.729] main app/app1 0/0/1000/1/1002 200 449 - - ---- 13/13/13/7/0 0/0 "GET / HTTP/1.1"
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39803 [02/Dec/2016:10:56:58.730] main app/app1 0/0/999/1/1000 200 377 - - ---- 12/12/12/7/0 0/0 "GET / HTTP/1.1"
Dec 2 10:56:59 localhost haproxy[25988]: xx.xx.xx.xx:39804 [02/Dec/2016:10:56:58.730] main app/app1 0/0/999/1/1000 200 277 - - ---- 11/11/11/7/0 0/0 "GET / HTTP/1.1"
From your log, most of your time is being spent connecting to the server. For example, you spend 1000, 999 and 999 milliseconds connecting. This may have to do with that you are closing the connection to the server immediately after each transaction by using option http-server-close. So, the TCP connection has to be re-established each time (if this is the same client between requests).
Overall, it looks like you're spending about 1 second per request, which doesn't sound horrible to me. What were you seeing before using HAProxy?
I have a Magento 1.9.2 system that is currently undergoing a brute force attack against its xml-rpc endpoint from an EC2 host.
I can simply firewall the source address but that is a short term solution, since it will likely face another attack from a different address. I would like to be able to detect these attacks automatically to lock them down.
Fail2ban is commonly used under such circumstances but in order for it to work, I understand that it must be able to find login failure messages in a log file somewhere, however Magento does not seem to be logging the failed attempts.
How can I prevent the xml-rpc endpoint being brute forced?
54.246.87.74 - - [20/Jul/2015:13:10:24 +0000] "POST /index.php/api/xmlrpc/ HTTP/1.1" 200 777 "-" "XML-RPC.NET"
54.246.87.74 - - [20/Jul/2015:13:10:24 +0000] "POST /index.php/api/xmlrpc/ HTTP/1.1" 200 777 "-" "XML-RPC.NET"
54.246.87.74 - - [20/Jul/2015:13:10:25 +0000] "POST /index.php/api/xmlrpc/ HTTP/1.1" 200 777 "-" "XML-RPC.NET"
54.246.87.74 - - [20/Jul/2015:13:10:26 +0000] "POST /index.php/api/xmlrpc/ HTTP/1.1" 200 777 "-" "XML-RPC.NET"
54.246.87.74 - - [20/Jul/2015:13:10:27 +0000] "POST /index.php/api/xmlrpc/ HTTP/1.1" 200 777 "-" "XML-RPC.NET"
Action taken so far
I've configured fail2ban with a new filter and jail to lock it down but I still don't know if this is the best solution.
filter.d/magento-xmlrpc.conf
[Definition]
failregex = ^<HOST> .*POST .*api\/xmlrpc\/
ignoreregex =
jail.local
[magento-xmlrpc]
enabled = true
port = http,https
filter = magento-xmlrpc
logpath = /home/user/logs/access.log
maxretry = 20
findtime = 30
bantime = 600
I have noticed numerous entries in Tomcat's local_access_log for various resources coming from IP address 127.0.0.1. These are clearly attempts to hack in. For example, here is a request to get access to the "manager" app:
127.0.0.1 - - [30/Apr/2015:13:35:13 +0000] "GET /manager/html HTTP/1.1" 401 2474
here is another one:
127.0.0.1 - - [30/Apr/2015:21:23:37 +0000] "POST /cgi-bin/php?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%22%79%65%73%22+%2D%64+%63%67%69%2E%66%69%78%5F%70%61%74%68%69%6E%66%6F%3D%31+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1" 404 1016
When decoded, the URL is this:
127.0.0.1 - - [30/Apr/2015:21:23:37 0000] "POST /cgi-bin/php?-d allow_url_include=on -d safe_mode=off -d suhosin.simulation=on -d disable_functions="" -d open_basedir=none -d auto_prepend_file=php://input -d cgi.force_redirect=0 -d cgi.redirect_status_env="yes" -d cgi.fix_pathinfo=1 -d auto_prepend_file=php://input -n HTTP/1.1" 404 1016
There are lots of such entries, all from IP address 127.0.0.1. Obviously, since this is the address of localhost, I can't block it. More over, I am not sure if there is something that I can do about it. Is there possibly an exploit that should be patched up? For instance, is there a version of Tomcat that has a related vulnerability? I am running Tomcat 8.
Much thanks for any advice!
UPDATE: thanks for the suggestion about a proxy. Turned out that httpd was indeed installed and not surprisingly, there are suspicious request. For example:
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir HTTP/1.0
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir%20c:\\ HTTP/1.0
[Sat Mar 30 17:26:49 2013] [error] [client 5.34.247.59] Invalid URI in request GET /_mem_bin/../../../../winnt/system32/cmd.exe?/c+dir%20c:\\ HTTP/1.0
This is not a windows system so cmd.exe has not place for it...
If you have a proxy server running on your computer, that will often receive requests and then call the primary server using the localhost (127.0.0.1) interface.
This could explain why you're logging these requests.