I am pretty new to grok. Am trying to understand it by testing some grok patterns using online Online Grok Tool.
I followed the instruction provided in the Link for the same.
When am testing as mentioned in the instructions link I don't see any output. please see below for error,
127.0.0.1 – – [11/Dec/2013:00:01:45 -0800] “GET /xampp/status.php HTTP/1.1” 200 3891 “http://cadenza/xampp/navi.php” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0”
This is because you copy/pasted the log directly from website, which copied html entity characters.
observe how – – are not - -, and “ ” are not " " in the following log,
127.0.0.1 – – [11/Dec/2013:00:01:45 -0800] “GET /xampp/status.php HTTP/1.1” 200 3891 “http://cadenza/xampp/navi.php” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0”
Following should work fine,
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
Related
I'm trying to send Apache/2.2.22 (Ubuntu) logs to remote rsyslogd 8.2001.0 (aka 2020.01) server and then use awstats 7.6 (build 20161204). I have problem with format and awstats shows that lines a corrupted I'm guessing that lines corrupted because of one extra white space in the beginning. Can someone tell me why rsyslog adds this extra space or how to remove it ? Log looks like:
x.x.x.x - - [06/Jan/2022:08:39:07 +0200] "GET /1.php HTTP/1.1" 200 2906 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
But should look like:
x.x.x.x - - [06/Jan/2022:08:39:07 +0200] "GET /1.php HTTP/1.1" 200 2906 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
My apache config to send logs to local7:
CustomLog "| /bin/bash -c /usr/bin/tee -a ${APACHE_LOG_DIR}/access-my.domain.log | /usr/bin/logger -t my.domain.com -p local7.info" combined
Sender rsyslog config:
$ModLoad imfile
$InputFilePollInterval 10
$InputFileName /var/log/apache2/access-*.log
$InputFileTag apache2-access
$InputFileStateFile stat-apache-access
$InputFileSeverity info
$InputRunFileMonitor
$InputFileFacility local7
local7.* #x.x.x.x
Receiver rsyslog:
module(load="imudp")
input(type="imudp" port="514")
$template mydomain, "/var/log/remote-logs/access-my.domain.com.log"
$template mydomain2, "%msg%\n"
if $syslogtag == "my.domain.com:" then ?mydomain;mydomain2
& stop
Please help, and if you know this is extra: Apache should log to local7 and keep files locally, but this CustomLog only sends to local7 without saving files locally, i know this is OLD (DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"), that's why I sending logs to remote server , because in this distro no more packages for awstats and Perl modules.
for white space i need to strip log with
"%msg:2:$%\n" not only "%msg%\n"
But i have problem why logs not saving locally ?
Reviewing access logs we've noticed Google PageSpeed Insights crops long URLs at around 70 chars and an ellipsis is appended. This results in a 404. Example:
8.8.8.8 - - [17/Sep/2020:10:32:22 +0200] "GET /wp-content/uploads/2016/06/petey-peeking-through-d%E2%80%A6 HTTP/1.1" 404 4650 "https://example.com/" "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Mobile Safari/537.36 Chrome-Lighthouse"
On sites with many long URLs this causes a lot of 404s, which negatively impacts WordPress sites for example as they handle 404s via PHP. I presume it will also result in incomplete/incorrect test analysis and results. I can't seem to find any information about this online. Is it intended behavior?
Additional examples:
66.249.93.34 - - [17/Sep/2020:14:15:20 +0200] "GET /wp-content/uploads/2020/09/test-picture-with-a-very-very-very-long-name-1024x402.jpg HTTP/1.1" 200 17896 "https://wpland.se/" "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Mobile Safari/537.36 Chrome-Lighthouse"
66.249.93.34 - - [17/Sep/2020:14:17:33 +0200] "GET /wp-content/uploads/2020/09/test-picture-with-a-very-very%E2%80%A6 HTTP/1.1" 404 4925 "http://wpland.se/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Safari/537.36 Chrome-Lighthouse"
We have the same issue, this function seems to truncate the urls:
function getOuterHTMLSnippet(element,ignoreAttrs=[],snippetCharacterLimit=500){const ATTRIBUTE_CHAR_LIMIT=75;try{if(element instanceof ShadowRoot){element=element.host;}
const clone=element.cloneNode();ignoreAttrs.forEach(attribute=>{clone.removeAttribute(attribute);});let charCount=0;for(const attributeName of clone.getAttributeNames()){if(charCount>snippetCharacterLimit){clone.removeAttribute(attributeName);}else{let attributeValue=clone.getAttribute(attributeName);if(attributeValue.length>ATTRIBUTE_CHAR_LIMIT){attributeValue=attributeValue.slice(0,ATTRIBUTE_CHAR_LIMIT-1)+'…';clone.setAttribute(attributeName,attributeValue);}
charCount+=attributeName.length+attributeValue.length;}}
const reOpeningTag=/^[\s\S]*?>/;const[match]=clone.outerHTML.match(reOpeningTag)||[];if(match&&charCount>snippetCharacterLimit){return match.slice(0,match.length-1)+' …>';}
return match||'';}catch(_){return`<${element.localName}>`;}};
https://github.com/GoogleChrome/lighthouse/issues/11465
I've managed to migrate gitlab-CE 8.1 to 8.2, but I get an annoying issue.
Everytime I try to create an issue, I get this error:
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/api/issues" for 93.93.xx.xxx at 2015-12-15 15:05:13 +0100
==> /var/log/gitlab/nginx/gitlab_access.log <==
93.93.xx.xxx - - [15/Dec/2015:15:05:13 +0100] "POST /api/api/issues HTTP/1.1" 405 2 "https://git.myhost.name/api/api/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
It only occurs on this project. Do you have any idea where it comes from?
For information, I use a custom nginx.
Thanks,
Edit: To give more details, I have a white page like this when creating the issue.
I have built and run Rendr's example apps on Ubuntu 13.10 using Node v0.8.6. When I click on the Repos or Users links, I get an HTTP 502 - Bad Gateway error, but when I refresh page (load from server) it works (200 - OK) and the repos or users are displayed
Here is server output for the working case - (page refresh):
127.0.0.1 - - [Fri, 31 Jan 2014 22:47:56 GMT] "GET /repos HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
like Gecko) Ubuntu Chromium/32.0.1700.102 Chrome/32.0.1700.102
Safari/537.36"
And here is the failure case - (link navigation):
127.0.0.1 - - [Fri, 31 Jan 2014 22:48:07 GMT] "GET /api/-/users HTTP/1.1" 502 - "http://localhost:3030/users" "Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu
Chromium/32.0.1700.102 Chrome/32.0.1700.102 Safari/537.36"
Any ideas or pointers to what the problem might be?
Thanks.
The solution for this problem is here: https://github.com/airbnb/rendr/issues/266
I'm looking for all the activities of a particular person with a specific ip in a huge access log and save the results into a text file.
What I'm currently doing in Ubuntu Server is:
grep "255.255.255.255" access.log >> search.txt
which gives me no results! but I'm sure there are hundreds of activity by this ip in my log.
What is wrong with my command?
Update: here is some exmples of my access log (sorry I can't show the actual ip addresses and my domain name):
37.98.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /stats/piwik.php?action_name=%D8%AA%D8%A7%D9%85%20%D9%88%20%D8%AC%D8%B1%DB%8C&idsite=1&rec=1&r=414317&h=18&m=46&s=45&url=http%3A%2F%2Fpooyatv.ir%2F%25D8%25A8%25D8%25B1%25D9%2586%25D8%25A7%25D9%2585%25D9%2587%25E2%2580%258C%25D9%2587%25D8%25A7%2Fitem%2F26-%25D8%25AA%25D8%25A7%25D9%2585-%25D9%2588-%25D8%25AC%25D8%25B1%25DB%258C&urlref=http%3A%2F%2Fpooyatv.ir%2F&_id=3cbd28047168de3d&_idts=1377354671&_idvc=5&_idn=0&_refts=0&_viewts=1377530242&pdf=1&qt=0&realp=0&wma=1&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1024x768>_ms=649 HTTP/1.1" 200 269 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/26-%D8%AA%D8%A7%D9%85-%D9%88-%D8%AC%D8%B1%DB%8C" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
2.144.x.x - - [27/Aug/2013:18:46:35 +0430] "GET /media/galleries/409/14.jpg HTTP/1.1" 304 190 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/409-%D8%A8%D8%A7%D8%A8-%D8%A7%D8%B3%D9%81%D9%86%D8%AC%DB%8C" "Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0"
37.98.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /media/items/cache/feb4274796d93ff716e9650163a77fb8_XL.jpg HTTP/1.1" 200 53717 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/26-%D8%AA%D8%A7%D9%85-%D9%88-%D8%AC%D8%B1%DB%8C" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
95.38.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /cache/jw_sigpro/jwsigpro_cache_d9f361e470_42.jpg HTTP/1.1" 200 4679 "http://www.poyatv.ir/\xc8\xd1\xe4\xc7\xe3\xe5\x9d\xe5\xc7/item/53461-\xdd\xe6\xca\xc8\xc7\xe1\xed\xd3\xca-\xe5\xc7" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)"
91.99.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /media/k2/items/cache/245effadf41c6129f4fe7accc564ef86_S.jpg HTTP/1.1" 200 7589 "http://www.mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/651-%D8%B3%DB%8C%D9%86%D9%85%D8%A7%DB%8C%DB%8C-%D8%A8%D8%A7%D8%A8-%D8%A7%D8%B3%D9%81%D9%86%D8%AC%DB%8C" "Mozilla/5.0 (Windows NT 6.1; rv:23.0) Gecko/20100101 Firefox/23.0"
5.113.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /cache/jw_sigpro/jwsigpro_cache_0f4194d152_6.jpg HTTP/1.1" 200 10693 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/52604-%D8%A8%D8%A7%D8%A8%D8%A7-%D9%84%D9%86%DA%AF-%D8%AF%D8%B1%D8%A7%D8%B2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0"
46.225.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /images/backgrounds/painting_bg.jpg HTTP/1.1" 200 41400 "http://www.mysite.ir/upload-center" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; AskTbPTV2/5.15.15.35882)"
151.241.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /index.php?option=com_k2&view=item&task=vote&format=raw&user_rating=5&itemID=240 HTTP/1.1" 200 343 "http://www.mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/240-%D8%AE%D8%A7%D9%86%D8%AF%D8%A7%D9%86-%D9%BE%D9%87%D9%84%D9%88%DB%8C" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36"
Update 2: When I search for example for 192.168 it finds every row containing it like 192.168.150.160, but if I search for 192.168.150.160 it returns nothing!