I am hooking firefox's send and WSARecv. Send should remove the gzip header (=> Accept-Encoding: none) and WSARecv hook shall edit the data.
At the moment the WSARecv function doesn't change any data, but only prints it out for the debug log. Many sites work, but sometimes images do not get loaded or the page doesn't.
I could find the following server response (Error 400 Bad Request) in bing.com's answer in the WSARecv function:
Hooked Send
GET /fd/ls/l?IG=c5c4c27da26c4c5db3e1ed6adbfafc0d&TYPE=Event.ClientInst&DATA=%5B%7B%22T%22:%22CI.Init%22,%22FID%22:%22CI%22,%22Name%22:%22Base%22,%22TS%22:1253465714457%7D,%7B%22W%22%3A1670%2C%22H%22%3A701%2C%22T%22:%22CI.Dims%22,%22FID%22:%22HP%22,%22Name%22:%22br%22,%22TS%22:1253465714457%7D,%7B%22Data%22%3A-1%2C%22T%22:%22CI.Test%22,%22FID%22:%22HP%22,%22Name%22:%22IsLive%22,%22TS%22:1253465714457%7D,%7B%22Time%22%3A489%2C%22T%22:%22CI.Load%22,%22FID%22:%22HP%22,%22Name%22:%22bg%22,%22TS%22:1253465714946%7D%5D HTTP/1.1
00001138 3.75038576 [4024] Host: a4.bing.com
00001139 3.75038576 [4024] User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.3) Gecko/20090824 YFF35 Firefox/3.5.3 (.NET CLR 3.5.30729)
00001140 3.75038576 [4024] Accept: image/png,image/;q=0.8,/;q=0.5
00001141 3.75038576 [4024] Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
00001142 3.75038576 [4024] Accept-Encoding: none
00001143 3.75038576 [4024] Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7
00001144 3.75038576 [4024] Keep-Alive: 300
00001145 3.75038576 [4024] Connection: keep-alive
00001146 3.75038576 [4024] Referer: http://www.bing.com/
00001147 3.75038576 [4024] Cookie: MUID=D22A51BCBBD54DD3BE74682327101C0A; SRCHD=MS=904338&D=792606&AF=IE8SRC; SRCHUSR=AUTOREDIR=0&GEOVAR=&DOB=20090704; SRCHUID=V=2&GUID=2069ECD3092B453ABFCB9C4762409ACD; _HOP=; _FS=mkt=de-DE&ui=de-DE
(end of header)
Hooked WSARecv
00001152 4.03753757 [4024] HTTP/1.1 204 No Content
00001153 4.03753757 [4024] Cache-Control: private
00001154 4.03753757 [4024] Content-Length: 0
00001155 4.03753757 [4024] P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND", policyref="http://privacy.msn.com/w3c/p3p.xml"
00001156 4.03753757 [4024] Date: Sun, 20 Sep 2009 16:55:17 GMT
00001157 4.03753757 [4024] Connection: keep-alive
00001158 4.03753757 [4024]
00001159 4.03753757 [4024] HTTP/1.0 400 Bad Request
00001160 4.03753757 [4024] Server: AkamaiGHost
00001161 4.03753757 [4024] Mime-Version: 1.0
00001162 4.03753757 [4024] Content-Type: text/html
00001163 4.03753757 [4024] Content-Length: 216
00001164 4.03753757 [4024] Expires: Sun, 20 Sep 2009 16:55:17 GMT
00001165 4.03753757 [4024] Date: Sun, 20 Sep 2009 16:55:17 GMT
00001166 4.03753757 [4024] Connection: close
00001167 4.03753757 [4024]
00001168 4.03753757 [4024]
00001169 4.03753757 [4024] Bad Request
00001170 4.03753757 [4024]
00001171 4.03753757 [4024] Bad Request
00001172 4.03753757 [4024] Your browser sent a request that this server could not understand.
00001173 4.03753757 [4024] Reference #7.cea432c3.1253465717.0
00001174 4.03753757 [4024]
00001175 4.03753757 [4024]
My question is: What is wrong with my HTTP headers? Why do some pages come up with a 400 error. Is it the Accept-Encoding:none?
Thanks for reading :)
"none" is not an acceptable value for the "Accept-Encoding" header. Not all servers look at that header, but those that do will usually validate it before using it. To make the server send uncompressed data, omit the header altogether, or set it to "identity" instead.
Related
From lighthouse chrome test page:
URL Transfer Size Potential Savings
/three.module.min.js 630.3 KiB 477.3 KiB
I minify but performance still need compression.
...
function shouldCompress (req, res) {
if (req.headers['x-no-compression']) {
// don't compress responses with this request header
return false
}
// fallback to standard filter function
return compression.filter(req, res)
}
hostingHTTP.use(compression({ filter: shouldCompress }))
...
My request :
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,ru;q=0.8
Cache-Control: no-cache
Connection: keep-alive
Host: maximumroulette.com
Pragma: no-cache
Referer: https://maximumroulette.com/apps/magic/public/module.html
sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: script
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36
My response:
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Connection: keep-alive
Content-Length: 394590
Content-Type: application/javascript; charset=UTF-8
Date: Mon, 13 Feb 2023 20:39:00 GMT
ETag: W/"6055e-1864c7208a7"
Keep-Alive: timeout=5
Last-Modified: Mon, 13 Feb 2023 20:23:07 GMT
X-Powered-By: Express
Did i need some action onclietn part for decopression ?
If i setup on server res.set('Content-Encoding', 'deflate'); // gzip, deflate, br a get error on chrome:
net::ERR_CONTENT_DECODING_FAILED 200 (OK)
Any suggestion ?
Im trying to POST request using Python to retreive a specific File. Since the URL is behind a server with authorized access theres no use posting it here
However the form data contains a field called base64 and lengthy which I cant figure out if its a form data value or base64 encoding of post request
Here are browser parameters
General:
Request URL: http://exampleapi.com/api/Document/Export
Request Method: POST
Status Code: 200 OK
Remote Address: XX.XXX.XXX.XX:XX
Referrer Policy: no-referrer-when-downgrade
Response Headers:
Access-Control-Allow-Origin: http://example.com
Cache-Control: no-cache
Content-Disposition: attachment; filename=location-downloads.xlsx
Content-Length: 7148
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Date: Tue, 23 Jul 2019 21:00:18 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 10162
Content-Type: application/x-www-form-urlencoded
Cookie: abcConnection=!UA7tkC3iZCmVNGRUyRpDWARVBWk/lY6SZvgxLlaygsQKk+vuwA1NxvhwE9ph4i+3NZlKeepIfuHhUvyQjl68fhhrT9ueqMx/3mBKUDcT
DNT: 1
Host: exampleapi.com
Origin: http://example.com
Referer: http://example.com/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Form Data:
fileName: location-downloads.xlsx
contentType: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
base64: UEsDBAoAAAAAAAh4904AAAAAAAAAAAAAAAAJAAAAZG9jUHJvcHMvUEsDBAoAAAAIAAh490(shortened for simplicity)
Here is what I tried
url='http://example.com'
urllib3.disable_warnings()
headers = {
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0",
}
with requests.session() as s:
r=s.get(url,headers={"User-Agent":"Mozilla/5.0"},verify=False)
data=r.content
soup=BeautifulSoup(data,'html.parser')
form_data = {
"fileName":"location-downloads.xlsx",
"contentType":"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
}
r2=s.post('http://exampleapi.com/api/Document/Export',data=json.dumps(form_data,ensure_ascii=True).encode('utf-8'),headers=headers,verify=False)
print(r2.status_code)
Any idea how i should proceed. My status code also shows 500 for the post here
What could possibly be the reason for expressjs route to contain the following data? I am expecting it to return JSON data. I am making an ajax call to the server(expressjs) which gives me the below data with weird characters. Is this data gzipped? I have set the headers and contentType as follows:
headers: {"Access-Control-Allow-Origin":"*"}
contentType: 'application/json; charset=utf-8'
�=O�0�b��K�)�%7�܈9���G��%NOU���O'6��k�~6��S.���,��/�wأ%6�K�)��e�
The HTTP response is as follows:
General:
Request URL: http://localhost/expressRoute.js
Request Method: GET
Status Code: 200 OK
Remote Address: [::1]:80
Referrer Policy: no-referrer-when-downgrade
Response Headers:
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Length: 29396
Content-Type: application/javascript
Date: Thu, 22 Nov 2018 00:50:36 GMT
ETag: "72d4-57b124e0c372e"
Keep-Alive: timeout=5, max=100
Last-Modified: Tue, 20 Nov 2018 05:57:12 GMT
Server: Apache/2.4.34 (Win32) OpenSSL/1.1.0i PHP/7.2.10
Request Headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: localhost
Pragma: no-cache
Referer: http://localhost/index.html
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
I think this a HTTP-related problem.
I want to use my (JAX-RS) RESTeasy Service on a (JEE6) JBoss AS 7 Server from an Android Device. The RESTeasy Service is working fine. I am using on the Client-Side the Restlet-Client. This works too - without Security.
I want to use my JAAS-Formbased Security for the Pattern /rest/* in web.xml. So I have to send a HTTP-POST-Request with the Form-Data (j_username and j_password) to /foo/j_security_check.
I get the JSESSIONID from the first Response by the Server:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined; Path=/foo
Pragma: No-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 01:00:00 CET
X-Powered-By: JSF/2.0
Content-Type: text/html;charset=utf-8
Content-Length: 1028
Date: Wed, 15 Aug 2012 11:42:59 GMT
For this anonymous session I am authenticating ...
Header:
POST /foo/j_security_check HTTP/1.1
Date: Wed, 15 Aug 2012 11:42:58 GMT
Accept: text/html
Host: 172.24.47.5:8080
User-Agent: Restlet-Framework/2.0.14
Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined
Content-Length: 62
Content-Language: *
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content:
j_username=Bob&j_password=a
... and it works: JBoss-Security-TRACE:
2012-08-15 13:22:26,829 TRACE
[org.jboss.security.auth.spi.DatabaseServerLoginModule]
(http-0.0.0.0-0.0.0.0-8080-4) User 'Bob' authenticated, loginOk=true
Now the Problem: In the following request I want to GET the REST-URL (using the Cookie JSESSIONID):
GET /foo/rest/sync/products HTTP/1.1
Date: Wed, 15 Aug 2012 11:42:59 GMT
Accept: application/json
Host: 172.24.47.5:8080
User-Agent: Restlet-Framework/2.0.14
Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined
Content-Length: 0
But instead of returning the Response with JSON Content, the server is returning the JSF-Login-Page, because it want's me to authenticate again(?):
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 01:00:00 CET
X-Powered-By: JSF/2.0
Content-Type: text/html;charset=utf-8
Content-Length: 936
Date: Wed, 15 Aug 2012 11:42:59 GMT
<?xml version="1.0" encoding="utf-8"?> ... ... ... </html>
If I login with the Browser and then open the REST-URL it works fine. This is the GET-Request by the Browser:
GET http://localhost:8080/foo/rest/sync/products HTTP/1.1
Host: localhost:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko)
Chrome/19.0.1084.56 Safari/536.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=royq26yLd7REOz2otiZdTl6j.undefined
Anyone has an idea? I think the problem lays in the last request (GET /foo/rest/sync/products), because in the Browser it works fine.
Thanks
I have just checked the server headers for my website and this is what i got:
1 Server Response: http://www.pjnsports.co.uk
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?6690d3e0
I havent set up any 302 redirects - i assume this will be my host that is doing this. Is this normal practice?? will it have a negative affect on search results, site load speed etc? basically should i be going to them and telling them to do something about it?
Cheers
Paul
I'm not getting any 302 / 301 going to that site:
GET / HTTP/1.1
Host: www.pjnsports.co.uk
Connection: keep-alive
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: PHPSESSID=emr10g0gs9srtjccadb4k7t846; language=en; currency=GBP; __utmz=239376578.1300041169.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=239376578.1365490247.1300041169.1300041169.1300041169.1; __utmc=239376578; __utmb=239376578.1.10.1300041169
HTTP/1.1 200 OK
Date: Sun, 13 Mar 2011 18:33:41 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 6531
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8