I'm trying to send Apache/2.2.22 (Ubuntu) logs to remote rsyslogd 8.2001.0 (aka 2020.01) server and then use awstats 7.6 (build 20161204). I have problem with format and awstats shows that lines a corrupted I'm guessing that lines corrupted because of one extra white space in the beginning. Can someone tell me why rsyslog adds this extra space or how to remove it ? Log looks like:
x.x.x.x - - [06/Jan/2022:08:39:07 +0200] "GET /1.php HTTP/1.1" 200 2906 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
But should look like:
x.x.x.x - - [06/Jan/2022:08:39:07 +0200] "GET /1.php HTTP/1.1" 200 2906 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
My apache config to send logs to local7:
CustomLog "| /bin/bash -c /usr/bin/tee -a ${APACHE_LOG_DIR}/access-my.domain.log | /usr/bin/logger -t my.domain.com -p local7.info" combined
Sender rsyslog config:
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /var/log/apache2/access-*.log
$InputFileTag apache2-access
$InputFileStateFile stat-apache-access
$InputFileSeverity info
$InputRunFileMonitor
$InputFileFacility local7
local7.* #x.x.x.x
Receiver rsyslog:
module(load="imudp")
input(type="imudp" port="514")
$template mydomain, "/var/log/remote-logs/access-my.domain.com.log"
$template mydomain2, "%msg%\n"
if $syslogtag == "my.domain.com:" then ?mydomain;mydomain2
& stop
Please help, and if you know this is extra: Apache should log to local7 and keep files locally, but this CustomLog only sends to local7 without saving files locally, i know this is OLD (DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"), that's why I sending logs to remote server , because in this distro no more packages for awstats and Perl modules.
for white space i need to strip log with
"%msg:2:$%\n" not only "%msg%\n"
But i have problem why logs not saving locally ?
I upgraded gitlal omnibus from gitlab-ce-12.0.2-ce.0.el7.x86_64 to gitlab-ce-12.0.3-ce.0.el7.x86_64.
Post which when I launch the URL http://10.28.19.103:8080 it redirects to http://10.28.19.103:8080/users/sign_in.
In that I only see a sign in btn. Upon clicking nothing happens. I have no space to enter to enter username and password.
The logs are as below:
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/" for 10.28.208.19 at 2019-07-05 01:02:15 +0800
Processing by RootController#index as HTML
Redirected to http://10.28.19.103:8080/users/sign_in
**Filter chain halted as :redirect_unlogged_user rendered or redirected**
Completed 302 Found in 16ms (ActiveRecord: 0.5ms)
Started GET "/users/sign_in" for 10.28.208.19 at 2019-07-05 01:02:16 +0800
Processing by SessionsController#new as HTML
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":302,"duration":17.38,"view":0.0,"db":0.54,"location":"http://10.28.19.103:8080/users/sign_in","time":"2019-07-04T17:02:15.975Z","params":[],"remote_ip":"10.28.208.19","user_id":null,"username":null,"ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36","queue_duration":3.56,"correlation_id":"fqMPRtqjdO3"}
==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 200 OK in 52ms (Views: 32.5ms | ActiveRecord: 2.2ms)
Started GET "/uploads/-/system/appearance/header_logo/1/ytlc.png" for 10.28.208.19 at 2019-07-05 01:02:16 +0800
Processing by UploadsController#show as HTML
Parameters: {"model"=>"appearance", "mounted_as"=>"header_logo", "id"=>"1", "filename"=>"ytlc.png"}
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/appearance/header_logo/1/ytlc.png (0.3ms)
Completed 200 OK in 16ms (ActiveRecord: 1.5ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"new","status":200,"duration":54.1,"view":32.47,"db":2.17,"time":"2019-07-04T17:02:16.020Z","params":[],"remote_ip":"10.28.208.19","user_id":null,"username":null,"ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36","queue_duration":5.03,"correlation_id":"T9vwNeRZZZ6"}
{"method":"GET","path":"/uploads/-/system/appearance/header_logo/1/ytlc.png","format":"html","controller":"UploadsController","action":"show","status":200,"duration":17.42,"view":0.0,"db":1.47,"time":"2019-07-04T17:02:16.768Z","params":[{"key":"model","value":"appearance"},{"key":"mounted_as","value":"header_logo"},{"key":"id","value":"1"},{"key":"filename","value":"ytlc.png"}],"remote_ip":"10.28.208.19","user_id":null,"username":null,"ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36","queue_duration":4.3,"correlation_id":"Wsuv3JkKIj2"}
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2019-07-05 01:02:18 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.7ms | ActiveRecord: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":6.82,"view":0.67,"db":0.0,"time":"2019-07-04T17:02:18.715Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"2e2fdaf8-4f81-4075-b9b5-1c34055bafba"}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <==
[2019-07-05 01:02:18] 127.0.0.1 - - [05/Jul/2019:01:02:18 +08] "GET /metrics HTTP/1.1" 200 3501 "-" "Prometheus/2.8.1"
I took a backup of the current repositories, installed a new GIT on temp VM. Imported the above repository.
But again I faced the same problem.
Please help.
You should not be accessing GitLab via port 8080. That's Unicorn, and it shouldn't be listening externally by default. You should access GitLab via port 80 or 443 through Nginx.
If you've set Unicorn to listen on port 8080 on something other than localhost, I suggest setting that back to default and accessing via the configured external URL (which should be port 80 or 443).
I want to be able to convert hostnames to IP addresses in an Apache access log file (i.e. the opposite of what logresolve does).
I have an accesslog file that has been converted with logresolve but I want to revert it.
Each line starts, as an example:
hostname.com - - [01/Jan/2016:00:00:00 +0000] "GET /stuff HTTP/1.1" 200 1046 "http://mywebsite.com" "Mozilla/5.0 (Windows NT 6.1)"
How do I convert hostname.com to an IP address for every line?
I've managed to migrate gitlab-CE 8.1 to 8.2, but I get an annoying issue.
Everytime I try to create an issue, I get this error:
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/api/issues" for 93.93.xx.xxx at 2015-12-15 15:05:13 +0100
==> /var/log/gitlab/nginx/gitlab_access.log <==
93.93.xx.xxx - - [15/Dec/2015:15:05:13 +0100] "POST /api/api/issues HTTP/1.1" 405 2 "https://git.myhost.name/api/api/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
It only occurs on this project. Do you have any idea where it comes from?
For information, I use a custom nginx.
Thanks,
Edit: To give more details, I have a white page like this when creating the issue.
We are using logstash and it's grok filtering to pre-process our Apache Logfiles.
All our machines are behind load balancers, so client IPs are logged into the "X-Forwarded-For" header.
Our access logs look like this:
"18.32.120.32, 192.168.12.118" [07/Sep/2014:15:53:48 +0200] "GET /login HTTP/1.1" 200 137 "http://www.google.com" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/32.0"
"18.32.120.32, 88.32.240.21, 192.168.12.118" [07/Sep/2014:15:53:48 +0200] "GET /login HTTP/1.1" 200 137 "http://www.google.com" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/32.0"
the corresponding apache logging directive looks like this:
LogFormat "\"%{X-Forwarded-For}i\" %t %{Host}i \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
As you can see, the x-forwarded-for header can consist from 1 to 3 IP Addresses, depending on the way the request is received.
We interpret the x-Forwarded-for header as "QuotedString" in the logstash/grok pattern:
CUSTOMLOG %{QUOTEDSTRING:xforwardedfor_header} \[%{HTTPDATE:time}\] %{HOSTNAME:host_header} \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QUOTEDSTRING:http_referrer} %{QUOTEDSTRING:http_useragent}
If we try to use the GeoIP Module from grok on the xforwardedfor_header field, the geo resolution fails. Shouldn't the module search and use the first IP Address it encounters?
Do we need the interpret the x-forwarded-for entry another way? If so, how?
Thanks very much.
Looking at the geoip source code, it does hanlde an array:
ip = event[#source]
ip = ip.first if ip.is_a? Array
So that tells me that you don't have an array in your xforwardedfor_header -- you have a comma seperated string... so you just need to split it.
filter {
split {
field => xforwardedfor_header
terminator => ','
}
}
Doing that before your geoip filter should fix your issue.