When I don't specify a logfile in the virtual host sections of my conf-file the logs are written in the file specified in httpd.conf (=access_log).
A log-entry would look like this:
SOMEIP - - [22/Jan/2013:18:34:08 +0100] "GET / HTTP/1.1" 200 1752 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/SOMEIP Safari/537.17"
SOMEIP - - [22/Jan/2013:18:34:08 +0100] "GET /img/homepage_bg.png HTTP/1.1" 304 - "http://DOMAIN/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) C$
But when I define a log file in the virtual host section the new log file contains different information:
SOMEIP - - [22/Jan/2013:18:33:34 +0100] "GET / HTTP/1.1" 200 1752
SOMEIP - - [22/Jan/2013:18:33:34 +0100] "GET /img/homepage_bg.png HTTP/1.1" 304 -
i define the log file like this:
CustomLog logs/DOMAIN-access_log common
Why does a custom log contain less information than the general log where all virtual hosts log in by default?
You need to define the alias "common" with a log format that includes the user-agent.
LogFormat "%h %l %u %t \"%r\" %>s %b "%{User-agent}i" common
You didn't say what flavour of Linux you're using. Any decently configured Apache (for example the Debian-based ones like Ubuntu, Mint, etc.) will already have a fitting LogFormat containing the user-agent in their configuration. Look for all the lines matching LogFormat. You should find something like this:
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Just use the combined or even the vhost_combined parameter for your logfile:
CustomLog logs/DOMAIN-access_log combined
You should also look at the documentation for the Custom Log Formats.
Related
I try to modify the error log to show runtime duration.
I added %T/%D at the end of the LogFormat at httpd.conf
like this:
<IfModule log_config_module>
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
# NOTE: "combined" and "common" are required by WHM
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%T%D %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined
# access_log format can be set in WHM under 'Basic cPanel & WHM Setup'
CustomLog logs/access_log combined
</IfModule>
I restarted the httpd service but the change didn't take place.
At Cpanel it is specified to use the combined format.
Can someone help me please?
My log4j properties as follows,
#Application Logger
log4j.rootLogger=DEBUG,file
#log4j.rootLogger=DEBUG,file,console
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=../common/logs/api/log4j-app.log
log4j.appender.file.maxFileSize=6MB
log4j.appender.file.maxBackupIndex=5
log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{ABSOLUTE} %5p %c<strong><strong>{1}</strong></strong>:%L - %m%n
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.file.Append=false
#Manual WebLogs
log4j.logger.webLogger=INFO, web
log4j.appender.web=org.apache.log4j.RollingFileAppender
log4j.appender.web.maxFileSize=900KB
log4j.appender.web.maxBackupIndex=6
log4j.appender.web.layout=org.apache.log4j.PatternLayout
#log4j.appender.web.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.web.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.web.File=../common/logs/web/manual.log
log4j.appender.web.Append=false
#Manual ApiLogs
log4j.logger.apiLogger=INFO, api ,console
log4j.appender.api=org.apache.log4j.RollingFileAppender
log4j.appender.api.maxFileSize=900KB
log4j.appender.api.maxBackupIndex=6
log4j.appender.api.layout=org.apache.log4j.PatternLayout
#log4j.appender.api.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.api.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.api.File=../common/logs/api/manual.log
log4j.appender.api.Append=false
#Manual ApiSecurityLogs
log4j.logger.apiSecurityLogger=INFO, apiSecurity
log4j.appender.apiSecurity=org.apache.log4j.RollingFileAppender
log4j.appender.apiSecurity.maxFileSize=900KB
log4j.appender.apiSecurity.maxBackupIndex=6
log4j.appender.apiSecurity.layout=org.apache.log4j.PatternLayout
#log4j.appender.apiSecurity.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.apiSecurity.layout.ConversionPattern=%d{ISO8601} : %L -- %m%n
log4j.appender.apiSecurity.File=../common/logs/apiSecurity/manualSecurity.log
log4j.appender.apiSecurity.Append=false
# Appender which writes to console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss} %c %m%n
We are using the above log4j.properties to create web logs and api logs.
web related logs goes to web/manual.log.
api related logs goes to api/manual.log.
api security related logs goes to apiSecurity/manualSecurity.log
for api manual.log and log4j-app.log files are creating as
manual.log, manual.log.1, manual.log.2 ...etc.,
log4j-app.log, log4j-app.log.1, log4j-app.log.2 ... so on.
I want all the log details in a single file as manual.log and log4j-app.log, please let me know how to get it?
Appenders type needs to be changed from RollingFileAppender to FileAppender.
So after changing it, your log4j.properties would look like:
#Application Logger
log4j.rootLogger=DEBUG,file
#log4j.rootLogger=DEBUG,file,console
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=log4j-app.log
#log4j.appender.file.maxFileSize=6MB
#log4j.appender.file.maxBackupIndex=5
log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{ABSOLUTE} %5p %c<strong><strong>{1}</strong></strong>:%L - %m%n
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.file.Append=false
#Manual WebLogs
log4j.logger.webLogger=INFO, web
log4j.appender.web=org.apache.log4j.FileAppender
#log4j.appender.web.maxFileSize=900KB
#log4j.appender.web.maxBackupIndex=6
log4j.appender.web.layout=org.apache.log4j.PatternLayout
#log4j.appender.web.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.web.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.web.File=../common/logs/web/manual.log
log4j.appender.web.Append=false
#Manual ApiLogs
log4j.logger.apiLogger=INFO, api ,console
log4j.appender.api=org.apache.log4j.FileAppender
#log4j.appender.api.maxFileSize=900KB
#log4j.appender.api.maxBackupIndex=6
log4j.appender.api.layout=org.apache.log4j.PatternLayout
#log4j.appender.api.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.api.layout.ConversionPattern=%d{ISO8601} %5p : %L -- %m%n
log4j.appender.api.File=manual.log
log4j.appender.api.Append=false
#Manual ApiSecurityLogs
log4j.logger.apiSecurityLogger=INFO, apiSecurity
log4j.appender.apiSecurity=org.apache.log4j.RollingFileAppender
log4j.appender.apiSecurity.maxFileSize=900KB
log4j.appender.apiSecurity.maxBackupIndex=6
log4j.appender.apiSecurity.layout=org.apache.log4j.PatternLayout
#log4j.appender.apiSecurity.layout.ConversionPattern=%d{ISO8601} %5p %F <Method>%M : %L -- %m%n
log4j.appender.apiSecurity.layout.ConversionPattern=%d{ISO8601} : %L -- %m%n
log4j.appender.apiSecurity.File=../common/logs/apiSecurity/manualSecurity.log
log4j.appender.apiSecurity.Append=false
# Appender which writes to console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{dd/MM/yyyy HH:mm:ss} %c %m%n
I'm looking to pull out various metrics from some server logs. The first is the total number of requests to just pages, not images, CSS files etc.
So I want to include requests like:
140.77.167.177 - - [01/Apr/2016:22:40:09 +1100] "GET /bad-credit-loans/abc/ HTTP/1.1" 200 7532 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
but ignore requests like:
158.165.213.180 - - [01/Apr/2016:23:00:55 +1100] "GET /assets/img/lenders/png/insurance.png HTTP/1.1" 200 17866 "https://www.example.au/lp/tradie-loans/?utm_source=facebook&utm_medium=cpc&utm_content=mobilead&utm_campaign=abcs/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G920I Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/48.0.2564.106 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/70.0.0.22.83;]"
grep "GET " | wc -l will get me all requests; how to I disregard those that are in a range (*.png, .css, .jpg and .js), and how do I extend this to ignore any file?
You can do:
grep -Ev '\.(png|jpg|css|js)' file.log
I have a apache server running on CentOS 6.5 server. In log files 25% of records are like this
000.000.000.000 - - [24/Jul/2014:04:15:10 +0000] "GET /address/of/the/page.html HTTP/1.1" 200 20341 "-" "Mozilla/4.0
But rest of 75% of logs are like this, without IP
- - [24/Jul/2014:04:15:10 +0000] "GET /address/of/the/page.html HTTP/1.1" 200 20341 "-" "Mozilla/4.0
Can somebody tell me what is the problem?
Thanks!
By default Apache prints the REMOTE_ADDR header value in the access log, it may be that in your environment that value is blank in some cases. You can try printing the X-Forwarded-For header in the access log and see if that has the values that you need.
Typically if the requests are passing through proxy servers/load balancers its a good idea to check the value of the XFF header for the correct client IP.
Hope this help.
-Avijit
Complementing Avi's response, you will need to open the file /etc/apache2/apache2.conf and change the following lines:
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
to:
LogFormat "%{X-Forwarded-For}i %v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %h %l %u %t \"%r\" %>s %O" common
LogFormat "%{X-Forwarded-For}i %{Referer}i -> %U" referer
LogFormat "%{X-Forwarded-For}i %{User-agent}i" agent
Then, restart apache service with the command: sudo service apache2 restart
and the client ip will appear on apache log.
Check the result with:
cat /var/log/apache2/access.log
With RHEL and Apache (httpd), is there a way to:
Detect which v.host (domain) is gaining how many incoming connections?
Because in this few days, i got a lot of incoming connections on the Server being acknowledged by using:
ps aux | grep httpd -c
734
That means i have 734 incoming connections currently being handled by Apache but the problem is i have (lets say) 10 Vhosts (domains).
So which Domain is getting the high loads?
Any sharp solutions please.
Say you have a file like /etc/apache2/sites-enabled/000-example.com
You may see lines like:
<VirtualHost *:80>
ServerName www.example.com
ServerAlias example.com
DocumentRoot /www/example.com
Further down, If you want, you can create logfiles for each VirtualHost by changing lines
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
Too:
ErrorLog ${APACHE_LOG_DIR}/example_com.error.log
CustomLog ${APACHE_LOG_DIR}/example_com.access.log combined
Otherwise, if you don't want to create logfiles for each VirtualHost, then you can create your own custom log format http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
For example:
LogFormat "%v %h %l %u %t \"%r\" %>s %b" common
CustomLog ${APACHE_LOG_DIR}/access.log common
When you have decided how you want to log, then you can write all kinds of scripts to examine or compare your logs. It would be far easier if you are using separate logfiles for each virtualhost, because you may try this.
wc -l /var/log/apache2/*access.log