I'm trying to make an HTTPS proxy server, but I cant make a connection to any server.
Edit: this is the part of the code that, after a client connect to my server, I get the message and try to send to the web. This is the message of a FireFox client trying to connect to Google:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = socket.create_connection((google.com, 443))
ssl_sock = ssl.wrap_socket(sock)
fullData=b''
ssl_sock.send(b'CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n')
while 1:
# receive data from web server
data = ssl_sock.recv(4026)
print(data)
if (len(data) > 0):
fullData+=data
else:
break
clientSock.send(fullData)
Google should got me a ok message but its getting me an error
HTTP/1.1 405 Method Not Allowed
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1592
Date: Fri, 24 May 2019 05:28:17 GMT
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
Connection: close
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 405 (Method Not Allowed)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>405.</b> <ins>That\xe2\x80\x99s an error.</ins>
<p>The request method <code>CONNECT</code> is inappropriate for the URL <code>/</code>. <ins>That\xe2\x80\x99s all we know.</ins>
Related
I try to figure out why webhook.site is responding with the error 404 when I´m sending an HTTP POST request with a TCP client into the endpoint:
import socket
HOST = "46.4.105.116"
PORT = 80
Payload = "POST /62b69843-f5b2-4d49-81b7-c3a61f6bdeda HTTP/1.1\r\n"
Payload += "Host: Python test\r\n"
Payload += "Content-Length: {}\r\n".format(0)
Payload += "\r\n"
print(Payload)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(str.encode(Payload))
data = s.recv(1024)
print(f"Received {data!r}")
Response:
Received b'HTTP/1.1 404 Not Found\r\nServer: nginx\r\nContent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\nVary: Accept-Encoding\r\nCache-Control: no-cache, private\r\ndate: Sun, 28 Aug 2022 17:32:45 GMT\r\n\r\n610\r\n<!DOCTYPE html>\n<html lang="en">\n <head>\n <meta charset="utf-8">\n <meta name="viewport" content="width=device-width, initial-scale=1">\n\n <title>Not Found</title>\n\n <!-- Fonts -->\n <link rel="dns-prefetch" href="//fonts.gstatic.com">\n <link href="https://fonts.googleapis.com/css?family=Nunito" rel="stylesheet">\n\n <!-- Styles -->\n <style>\n html, body {\n background-color: #fff;\n color: #636b6f;\n font-family: \'Nunito\', sans-serif;\n font-weight: 100;\n height: 100vh;\n margin: 0;\n }\n\n .full-height {\n height: 100vh;\n }\n\n .flex-center {\n align-items: center;\n display: f'
What is the issue here?
Your payload has invalid Host, use real host, from your variable
I'm using the Docusign Ruby SDK to make calls on the API. I have my code working when pointed to the developer docusign, when pointed to prod I'm getting the DocuSign_eSign::ApiError: Bad Request error.
I have already gotten my integration key/client id approved in production and I've also already done the authorization grant part, where you allow the integration key to send envelopes on a user's behalf. This has been done in dev and production.
desc 'Perform API call to list envelopes'
task list: :initialize do
options = DocuSign_eSign::ListStatusChangesOptions.new
options.from_date = (Date.today - 10).strftime('%Y/%m/%d')
options.status = 'completed'
#list_results = #envelopes_api.list_status_changes #account_id, options
end
desc 'Build list results hash'
task build_list_hash: :list do
#list_results_hash = []
#list_results.envelopes.each do |list_hash|
#list_results_hash << { envelope_id: list_hash.envelope_id, status: 'pending', archive_at: Time.now + (86400 *30) }
end
puts "List results hash:\n #{#list_results_hash}"
File.open(ENV['MASTER_LIST_FILE'], 'w') { |file| file.write(#list_results_hash.to_yaml) }
end
I expect the output to be a list of envelopes which is what happens when running on demo.docusign
Update: Thanks for showing me how to get logs. It seems the error is with GetUserProfileImage
GET https://na3-app.docusign.net:8832/restapi/v2.1/accounts/91608b5d-d418-4c45-89f0-cd088504f99d/users/63757e15-8365-4fb9-9e7d-d9a8309f8e94/profile/image
TraceToken: 996d3b3f-2857-47d6-9d75-26a75b6b0a37
Timestamp: 2019-10-17T20:58:32.0932823Z
Content-Length: 0
Connection: keep-alive
Accept: application/json;text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzipdeflatebr
Accept-Language: en-USen; q=0.5
Authorization: Bearer [omitted]
Host: na3-app.docusign.net
Referer: https://app.docusign.com/preferences/security
User-Agent: Mozilla/5.0(Windows NT 10.0; Win64; x64; rv:69.0)Gecko/20100101Firefox/69.0
x-docusign-clienttransactionid: fa4e1202-09a4-431b-9abf-fe4d00acf565
x-csrf-token: c34f42c5aad288063afc8a6615be03c9
x-forwarded-for: 170.140.186.226, 162.248.185.11
x-docusign-authentication: {"IntegratorKey":"[omitted]"}
x-docusign-prettyprint: false
content-transfer-encoding: base64
x-forwarded-for-martini: 170.140.186.226
x-docusign-diagnostics: {"storedProcedureEventLogThreshold":"300"}
x-docusign-timetrack: CONN_START,2019-10-17T20:58:31.946Z;;REQ_SENT,2019-10-17T20:58:31.950Z;REST0_Start,2019-10-17T20:58:32.0620315Z
x-docusign-correlationtoken: fa4e1202-09a4-431b-9abf-fe4d00acf565
X-SecurityProtocol-Version: TLSv1.2
X-SecurityProtocol-CipherSuite: ECDHE-RSA-AES256-GCM-SHA384
404 NotFound
Content-Type: application/json; charset=utf-8
Content-Length: 95
X-DocuSign-ClientTransactionId: fa4e1202-09a4-431b-9abf-fe4d00acf565
X-DocuSign-TimeTrack: CONN_START,2019-10-17T20:58:31.946Z;;REQ_SENT,2019-10-17T20:58:31.950Z;;REST0_Start,2019-10-17T20:58:32.0620315Z;REST0_End,2019-10-17T20:58:32.0932823Z
X-DocuSign-TraceToken: 996d3b3f-2857-47d6-9d75-26a75b6b0a37
{"errorCode":"RESOURCE_NOT_FOUND","message":"The URL provided does not resolve to a resource."}```
When my python code run in localhost,it work. But, it doesn't in AWS ec2?
My code is simple: send a https request with post method. It works in localhost, but it occurs problem in AWS ec2.
I change another https request, it works.So the network is ok.
The code and the received content are below:
#!/usr/bin/python
# -*- coding:utf-8 -*-
import random
import requests
url = 'https://www.haidilao.com/eportal/ui?moduleId=5&pageId=9c8cf76c4ca84fc686ca11aaa936f5c7&struts.portlet.action=/portlet/map-portlet!getMapDotDataByRegion.action&random='+str(random.random())
header = {
'host':'www.haidilao.com',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'accept': 'text/plain, */*; q=0.01',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'zh-CN,zh;q=0.9,en;q=0.8'
}
data = {
'queryContent': 'CA',
'country':'6bf4038f63234217aecf93668a63f04b',
'myLat':'',
'myLng':''
}
def test():
data = parse.urlencode(data).encode('utf-8')
req = requests.post(url,headers= header,data= data, verify= False)
print(req.status_code)
print(req.headers)
print(req.text)
test()
```python
```html
Content-Type: text/html
Content-Length: 60416
Connection: close
Date: Sat, 10 Aug 2019 13:02:31 GMT
Server: nginx
Last-Modified: Fri, 09 Aug 2019 14:05:06 GMT
ETag: "5d4d7d92-ec00"
Accept-Ranges: bytes
Vary: Accept-Encoding
X-Cache: Miss from cloudfront
Via: 1.1 46dd9ae2d97161deaefbdceeae5f57ac.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: SIN2-C1
X-Amz-Cf-Id: XNkaD2emKes3BpaY3ZVSGb1bxlnsHD1KZeHCZPXnOcspTaYXXjVzKA==
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
<title>sorry，the channel has expired!</title>
<link rel="stylesheet" type="text/css" href="/eportal/uiFramework/css/tip.css">
</head>
<body>
<div class="easysite-error">
<img src="/eportal/uiFramework/images/picNews/tip.png" />
<font class="easysite-404"></font>
<font>sorry，the channel has expired!</font>
</div>
</body>
</html>
I need to occasionally access an HTML page to update a database. This page is easily accessible via a web browser, but when I try to access it via a node.js application it doesn't work (the website detect that the request is made by a robot). However,
The robot request contains the same headers (including the
user-agent) that the web browser request.
The robot request doesn't contains referer header or cookie header, but the browser request either.
The IP of the robot is the same that the IP that I use
to browse the website.
In my eyes the robot request and the browser request are strictly identical. Nevertheless they are processed differently.
I'm running out of ideas... Maybe the request contains metadata like "this request was sent by node.js" but it would be really weird.
EDIT, here is a code sample :
// callback (error, responseContent)
function getPage (callback){
let options = {
protocol : 'https:',
hostname : 'xxx.yyy.fr',
port : 443,
path : '/abc/def',
agent : false,
headers : {
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding' : 'gzip, deflate, br',
'Accept-Language' : 'fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3',
'Cache-Control' : 'no-cache',
'Connection' : 'keep-alive',
'DNT' : '1',
'Host' : 'ooshop.carrefour.fr',
'Pragma' : 'no-cache',
'Upgrade-Insecure-Requests' : '1',
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0'
}
};
https.get (options, function (res){
if (res.statusCode !== 200){
res.resume ();
callback ('Error : res code != 200, res code = ' + res.statusCode);
return;
}
res.setEncoding ('utf-8');
let content = '';
res.on ('data', chunk => content += chunk);
res.on ('end', () => callback (null, content));
}).on ('error', e => callback (e));
}
EDIT : here is a comparison of the requests/responses :
Mozilla Firefox
request headers :
GET /3274080001005/eau-de-source-cristaline HTTP/1.1
Host: ooshop.carrefour.fr
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
response headers :
HTTP/2.0 200 OK
date: Wed, 11 Jul 2018 21:25:25 GMT
server: Unknown
content-type: text/html; charset=UTF-8
age: 0
x-varnish-cache: MISS
accept-ranges: bytes
set-cookie: visid_incap_1213048=G8a0mWzmQYi0GKuT2Ht7YeQ9QVsAAAAAQkIPAAAAAADvVZnsZHK18dQQxHakBprg; expires=Thu, 11 Jul 2019 11:17:56 GMT; path=/; Domain=.carrefour.fr
incap_ses_466_1213048=/2NKHS4HXU0T7FpkwpJ3BsV1RlsAAAAAAY3wbUkXacAceu2NkgUrhw==; path=/; Domain=.carrefour.fr
x-iinfo: 7-11020186-11020187 NNNN CT(1 2 0) RT(1531344324722 0) q(0 0 0 0) r(4 4) U12
x-cdn: Incapsula
content-encoding: gzip
X-Firefox-Spdy: h2
response content : expected HTML page
Node.js robot
request headers :
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding : gzip, deflate, br
Accept-Language : fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3
Cache-Control : no-cache
Connection : keep-alive
DNT : 1
Host : ooshop.carrefour.fr
Pragma : no-cache
Upgrade-Insecure-Requests : 1
User-Agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
response headers :
content-type : text/html
connection : close, close
cache-control : no-cache
content-length : 210
x-iinfo : 1-17862634-0 0NNN RT(1531344295049 65) q(0 -1 -1 0) r(0 -1) B10(4,314,0) U19
set-cookie : incap_ses_466_1213048=j34jMBWkPFYT7FpkwpJ3Bqd1RlsAAAAAVBfoZBShAvoun/M8UFxPPA==; path=/; Domain=.carrefour.fr
response content :
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>
salt-api returns HTTP/1.1 500 Internal Server Error when trying to log in:
curl -si localhost:8000/login -H "Accept: application/json" -d username='<notshown>' -d password='<notshown>' -d eauth='pam'
HTTP/1.1 500 Internal Server Error
Content-Length: 1607
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/3.2.2
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Thu, 14 Apr 2016 13:23:49 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 6a529d945c6654848c531337c1a1193f8635b482
Content-Type: text/html;charset=utf-8
Set-Cookie: session_id=6a529d945c6654848c531337c1a1193f8635b482; expires=Thu, 14 Apr 2016 23:23:49 GMT; Path=/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>500 Internal Server Error</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}
#traceback {
color: red;
}
</style>
</head>
<body>
<h2>500 Internal Server Error</h2>
<p>Configuration for external_auth could not be read.</p>
<pre id="traceback">Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.6/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 506, in hypermedia_handler
ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 1525, in POST
'Configuration for external_auth could not be read.')
HTTPError: (500, 'Configuration for external_auth could not be read.')
</pre>
<div id="powered_by">
<span>Powered by CherryPy 3.2.2</span>
</div>
</body>
</html>
When you enter the wrong credentials, the error is different so it definitely knows that the credentials entered before are correct:
curl -si localhost:8000/login -H "Accept: application/json" -d username='<notshown>' -d password='<notshown>' -d eauth='pam'
HTTP/1.1 401 Unauthorized
Content-Length: 1586
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/3.2.2
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Thu, 14 Apr 2016 13:23:34 GMT
Access-Control-Allow-Origin: *
Content-Type: text/html;charset=utf-8
Set-Cookie: session_id=b71f4afcc8f1caf3e0f5f33d0542c77bc1b9875a; expires=Thu, 14 Apr 2016 23:23:34 GMT; Path=/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>401 Unauthorized</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}
#traceback {
color: red;
}
</style>
</head>
<body>
<h2>401 Unauthorized</h2>
<p>Could not authenticate using provided credentials</p>
<pre id="traceback">Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.6/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 506, in hypermedia_handler
ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 1497, in POST
'Could not authenticate using provided credentials')
HTTPError: (401, 'Could not authenticate using provided credentials')
</pre>
<div id="powered_by">
<span>Powered by CherryPy 3.2.2</span>
</div>
</body>
</html>
/etc/salt/master:
rest_cherrypy:
port: 8000
host: 0.0.0.0
disable_ssl: true
debug: true
.....
external_auth:
pam:
<notshownhere>:
-.*
- '#runner'
- '#wheel'
versions used:
salt-api-2015.8.8-2.el6.noarch
salt-master-2015.8.8-2.el6.noarch
salt-minion-2015.8.8-2.el6.noarch
salt-2015.8.8-2.el6.noarch
Any idea why this is happening?
I had a typo in my master configuration:
I had:
external_auth:
pam:
<notshownhere>:
-.*
- '#runner'
- '#wheel'
Shoud be:
external_auth:
pam:
<notshownhere>:
- .*
- '#runner'
- '#wheel'