URLs getting cropped - pagespeed-insights

Reviewing access logs we've noticed Google PageSpeed Insights crops long URLs at around 70 chars and an ellipsis is appended. This results in a 404. Example:
8.8.8.8 - - [17/Sep/2020:10:32:22 +0200] "GET /wp-content/uploads/2016/06/petey-peeking-through-d%E2%80%A6 HTTP/1.1" 404 4650 "https://example.com/" "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Mobile Safari/537.36 Chrome-Lighthouse"
On sites with many long URLs this causes a lot of 404s, which negatively impacts WordPress sites for example as they handle 404s via PHP. I presume it will also result in incomplete/incorrect test analysis and results. I can't seem to find any information about this online. Is it intended behavior?
Additional examples:
66.249.93.34 - - [17/Sep/2020:14:15:20 +0200] "GET /wp-content/uploads/2020/09/test-picture-with-a-very-very-very-long-name-1024x402.jpg HTTP/1.1" 200 17896 "https://wpland.se/" "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Mobile Safari/537.36 Chrome-Lighthouse"
66.249.93.34 - - [17/Sep/2020:14:17:33 +0200] "GET /wp-content/uploads/2020/09/test-picture-with-a-very-very%E2%80%A6 HTTP/1.1" 404 4925 "http://wpland.se/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Safari/537.36 Chrome-Lighthouse"

We have the same issue, this function seems to truncate the urls:
function getOuterHTMLSnippet(element,ignoreAttrs=[],snippetCharacterLimit=500){const ATTRIBUTE_CHAR_LIMIT=75;try{if(element instanceof ShadowRoot){element=element.host;}
const clone=element.cloneNode();ignoreAttrs.forEach(attribute=>{clone.removeAttribute(attribute);});let charCount=0;for(const attributeName of clone.getAttributeNames()){if(charCount>snippetCharacterLimit){clone.removeAttribute(attributeName);}else{let attributeValue=clone.getAttribute(attributeName);if(attributeValue.length>ATTRIBUTE_CHAR_LIMIT){attributeValue=attributeValue.slice(0,ATTRIBUTE_CHAR_LIMIT-1)+'…';clone.setAttribute(attributeName,attributeValue);}
charCount+=attributeName.length+attributeValue.length;}}
const reOpeningTag=/^[\s\S]*?>/;const[match]=clone.outerHTML.match(reOpeningTag)||[];if(match&&charCount>snippetCharacterLimit){return match.slice(0,match.length-1)+' …>';}
return match||'';}catch(_){return`<${element.localName}>`;}};
https://github.com/GoogleChrome/lighthouse/issues/11465

Related

Express.js server strange calls

I have an Express.js server running on a Windows server, in my tests to put it into production I received a strange call that I did not make, from what I understand it is an attempt to access my server, what I do not understand is if these calls are normal for all server / webpages online.
My server is running with Https with certificates created in Certbot, I have helmet enabled and x-power-by disabled. I have the server listening on port 443, but I plan to change this to another port.
Previously I received many calls like the following:
138.197.190.182 - - [01/Jun/2022:21:00:40 +0000] "HEAD / HTTP/1.0" 404 140 "-" "-"
138.197.190.182 - - [01/Jun/2022:21:00:46 +0000] "GET /system_api.php HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
138.197.190.182 - - [01/Jun/2022:21:00:48 +0000] "GET /c/version.js HTTP/1.1" 404 151 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
138.197.190.182 - - [01/Jun/2022:21:00:50 +0000] "GET /streaming/clients_live.php HTTP/1.1" 404 165 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
138.197.190.182 - - [01/Jun/2022:21:00:52 +0000] "GET /stalker_portal/c/version.js HTTP/1.1" 404 166 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
138.197.190.182 - - [01/Jun/2022:21:00:54 +0000] "GET /stream/live.php HTTP/1.1" 404 154 "-" "VLC/3.0.8 LibVLC/3.0.8"
138.197.190.182 - - [01/Jun/2022:21:00:57 +0000] "GET /flu/403.html HTTP/1.1" 404 151 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
138.197.190.182 - - [01/Jun/2022:21:00:59 +0000] "GET / HTTP/1.1" 404 139 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
92.226.2.139 - - [11/May/2022:16:14:45 +0000] "GET /anaesthetist/goddaughters/betterment/Colombias.jsp HTTP/1.1" 404 189 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0)"
92.226.2.139 - - [11/May/2022:16:14:45 +0000] "GET /Yorkshires/TKO/chromes/limestone.jsp HTTP/1.1" 404 175 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; BOIE8;ENUS)"
82.102.17.180 - - [11/May/2022:16:25:19 +0000] "GET http://dyn.epicgifs.net/test6956.php HTTP/1.1" 404 151 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
103.178.236.40 - - [22/Apr/2022:22:17:20 +0000] "GET http://example.com/ HTTP/1.1" 404 139 "-" "Go-http-client/1.`1"
92.118.160.1 - - [23/Apr/2022:14:23:00 +0000] "GET / HTTP/1.0" 404 139 "-" "NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com"
I understand that as long as I don't have anything in the addresses they are trying to access there is no problem, or am I wrong?
My concern is that I received several identical calls in a short period of time from the same IP, like this:
193.19.109.230 - - [26/Jul/2022:22:59:03 +0000] "GET / HTTP/1.1" 404 139 "-" "python-requests/2.22.0"
My question is,
With the security that I currently have, should I be very concerned about these calls?
Public Servers often get spammed with Requests like these. Attackers try to get Informations about your server by scanning for specific Sites. So they can find attack vectors (for example old PHP/Wordpress Versions with known issues).
Other Requests can come from Scanners searching indexing security leaks or sites in general.
This is completely normal for Servers exposed to the Internet.
Another Question like this can be found here

AWS Gateway Timeout under threshold seconds

I have a node (express) app running on AWS that is randomly returning 504 (GATEWAY_TIMEOUT) without reaching the actual timeout (60 sec) threshold:
You can see that the requests following the failed ones take more time than the "timed out" ones...
On my express app I have:
server.keepAliveTimeout = 65000;
Any ideas?
EDIT: Adding ELB Logs:
2019-01-18T09:06:56.554353Z a38e67823174c11e9a984022fe7c311b 189.58.239.206:51399 - -1 -1 -1 504 0 0 0 "GET <app_endpoint> HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
2019-01-18T09:06:56.564478Z a38e67823174c11e9a984022fe7c311b 189.58.239.206:51400 - -1 -1 -1 504 0 0 0 "GET <app_endpoint> HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
2019-01-18T09:06:56.580591Z a38e67823174c11e9a984022fe7c311b 189.58.239.206:51401 - -1 -1 -1 504 0 0 0 "GET <app_endpoint> HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
2019-01-18T09:06:56.602049Z a38e67823174c11e9a984022fe7c311b 189.58.239.206:51398 - -1 -1 -1 504 0 0 0 "GET <app_endpoint> HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
Don't know if this a usable comment:
I just registered a domain at Route53
I linked my ELB configuration (running HTTP Express server to the domain)
-> Worked like a charm
Then I added security certificates and created the 443 part in ELB
Certificates where shown -> link went perfect...
typed domain in browser -> 504 (spinning busy and after a certain time 504)
I looked i lots of places - Even wanted to add hhtps to my webserver.
Then I realised it was not needed because of the ELB setup after a reverse proxy
What solved it for me was a total cache refresh in Chrom (command+option+R)!
Litteraly nothing changed at all and my webapp is now HTTPS.

Wordpress un controlled feed request from different IP

I am getting a lot of continuous feed request from different IPs, causing the server to occupy all the RAM. I get the following feed request. Can anyone help me to stop the feed...
37.210.162.69 - - [12/Sep/2016:04:34:43 -0400] "GET /category/from-newspapers/feed/ HTTP/1.1" 200 56908 "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-J700F Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2704.81 Mobile Safari/537.36"
188.52.101.103 - - [12/Sep/2016:04:35:35 -0400] "GET /category/society/feed/ HTTP/1.1" 500 554 "-" "Mozilla/5.0 (Linux; Android 4.4.2; Lenovo TAB 2 A7-30HC Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Safari/537.36;"
86.96.97.72 - - [12/Sep/2016:04:35:35 -0400] "GET /feed/ HTTP/1.1" 500 554 "-" "Mozilla/5.0 (Linux; Android 5.0; Lenovo A7000-a Build/LRX21M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36"
113.199.255.115 - - [12/Sep/2016:04:34:24 -0400] "GET /category/society/feed/ HTTP/1.1" 200 56908 "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-G7202 Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2704.81 Mobile Safari/537.36"
94.129.248.98 - - [12/Sep/2016:04:35:34 -0400] "GET /category/society/feed/ HTTP/1.1" 500 554 "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-J120F Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/52.0.2743.98 Mobile Safari/537.36"
49.244.190.144 - - [12/Sep/2016:04:35:34 -0400] "GET /category/from-newspapers/feed/ HTTP/1.1" 500 554 "-" "Mozilla/5.0 (Linux; U; Android 4.1.2; en-; SC-06D Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"
Hi :) The easiest way to secure your WordPress site that I know of (and used myself on 8 WP sites) is to use the WordFence plugin.
WordFence offers different types of protection and features. Out of the box it might be good, but read the documentation to learn which parameters to set for optimal security.
WordFence website

Running Rendr Examples Results in HTTP 502 Error When Links Clicked

I have built and run Rendr's example apps on Ubuntu 13.10 using Node v0.8.6. When I click on the Repos or Users links, I get an HTTP 502 - Bad Gateway error, but when I refresh page (load from server) it works (200 - OK) and the repos or users are displayed
Here is server output for the working case - (page refresh):
127.0.0.1 - - [Fri, 31 Jan 2014 22:47:56 GMT] "GET /repos HTTP/1.1" 200 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,
like Gecko) Ubuntu Chromium/32.0.1700.102 Chrome/32.0.1700.102
Safari/537.36"
And here is the failure case - (link navigation):
127.0.0.1 - - [Fri, 31 Jan 2014 22:48:07 GMT] "GET /api/-/users HTTP/1.1" 502 - "http://localhost:3030/users" "Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu
Chromium/32.0.1700.102 Chrome/32.0.1700.102 Safari/537.36"
Any ideas or pointers to what the problem might be?
Thanks.
The solution for this problem is here: https://github.com/airbnb/rendr/issues/266

Search in webserver logs with grep and save results to file

I'm looking for all the activities of a particular person with a specific ip in a huge access log and save the results into a text file.
What I'm currently doing in Ubuntu Server is:
grep "255.255.255.255" access.log >> search.txt
which gives me no results! but I'm sure there are hundreds of activity by this ip in my log.
What is wrong with my command?
Update: here is some exmples of my access log (sorry I can't show the actual ip addresses and my domain name):
37.98.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /stats/piwik.php?action_name=%D8%AA%D8%A7%D9%85%20%D9%88%20%D8%AC%D8%B1%DB%8C&idsite=1&rec=1&r=414317&h=18&m=46&s=45&url=http%3A%2F%2Fpooyatv.ir%2F%25D8%25A8%25D8%25B1%25D9%2586%25D8%25A7%25D9%2585%25D9%2587%25E2%2580%258C%25D9%2587%25D8%25A7%2Fitem%2F26-%25D8%25AA%25D8%25A7%25D9%2585-%25D9%2588-%25D8%25AC%25D8%25B1%25DB%258C&urlref=http%3A%2F%2Fpooyatv.ir%2F&_id=3cbd28047168de3d&_idts=1377354671&_idvc=5&_idn=0&_refts=0&_viewts=1377530242&pdf=1&qt=0&realp=0&wma=1&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1024x768&gt_ms=649 HTTP/1.1" 200 269 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/26-%D8%AA%D8%A7%D9%85-%D9%88-%D8%AC%D8%B1%DB%8C" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
2.144.x.x - - [27/Aug/2013:18:46:35 +0430] "GET /media/galleries/409/14.jpg HTTP/1.1" 304 190 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/409-%D8%A8%D8%A7%D8%A8-%D8%A7%D8%B3%D9%81%D9%86%D8%AC%DB%8C" "Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0"
37.98.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /media/items/cache/feb4274796d93ff716e9650163a77fb8_XL.jpg HTTP/1.1" 200 53717 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/26-%D8%AA%D8%A7%D9%85-%D9%88-%D8%AC%D8%B1%DB%8C" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36"
95.38.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /cache/jw_sigpro/jwsigpro_cache_d9f361e470_42.jpg HTTP/1.1" 200 4679 "http://www.poyatv.ir/\xc8\xd1\xe4\xc7\xe3\xe5\x9d\xe5\xc7/item/53461-\xdd\xe6\xca\xc8\xc7\xe1\xed\xd3\xca-\xe5\xc7" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)"
91.99.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /media/k2/items/cache/245effadf41c6129f4fe7accc564ef86_S.jpg HTTP/1.1" 200 7589 "http://www.mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/651-%D8%B3%DB%8C%D9%86%D9%85%D8%A7%DB%8C%DB%8C-%D8%A8%D8%A7%D8%A8-%D8%A7%D8%B3%D9%81%D9%86%D8%AC%DB%8C" "Mozilla/5.0 (Windows NT 6.1; rv:23.0) Gecko/20100101 Firefox/23.0"
5.113.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /cache/jw_sigpro/jwsigpro_cache_0f4194d152_6.jpg HTTP/1.1" 200 10693 "http://mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/52604-%D8%A8%D8%A7%D8%A8%D8%A7-%D9%84%D9%86%DA%AF-%D8%AF%D8%B1%D8%A7%D8%B2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0"
46.225.x.x - - [27/Aug/2013:18:46:34 +0430] "GET /images/backgrounds/painting_bg.jpg HTTP/1.1" 200 41400 "http://www.mysite.ir/upload-center" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; AskTbPTV2/5.15.15.35882)"
151.241.x.x - - [27/Aug/2013:18:46:36 +0430] "GET /index.php?option=com_k2&view=item&task=vote&format=raw&user_rating=5&itemID=240 HTTP/1.1" 200 343 "http://www.mysite.ir/%D8%A8%D8%B1%D9%86%D8%A7%D9%85%D9%87%E2%80%8C%D9%87%D8%A7/item/240-%D8%AE%D8%A7%D9%86%D8%AF%D8%A7%D9%86-%D9%BE%D9%87%D9%84%D9%88%DB%8C" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.57 Safari/537.36"
Update 2: When I search for example for 192.168 it finds every row containing it like 192.168.150.160, but if I search for 192.168.150.160 it returns nothing!

Resources