I try to figure out why webhook.site is responding with the error 404 when I´m sending an HTTP POST request with a TCP client into the endpoint:
import socket
HOST = "46.4.105.116"
PORT = 80
Payload = "POST /62b69843-f5b2-4d49-81b7-c3a61f6bdeda HTTP/1.1\r\n"
Payload += "Host: Python test\r\n"
Payload += "Content-Length: {}\r\n".format(0)
Payload += "\r\n"
print(Payload)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(str.encode(Payload))
data = s.recv(1024)
print(f"Received {data!r}")
Response:
Received b'HTTP/1.1 404 Not Found\r\nServer: nginx\r\nContent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\nVary: Accept-Encoding\r\nCache-Control: no-cache, private\r\ndate: Sun, 28 Aug 2022 17:32:45 GMT\r\n\r\n610\r\n<!DOCTYPE html>\n<html lang="en">\n <head>\n <meta charset="utf-8">\n <meta name="viewport" content="width=device-width, initial-scale=1">\n\n <title>Not Found</title>\n\n <!-- Fonts -->\n <link rel="dns-prefetch" href="//fonts.gstatic.com">\n <link href="https://fonts.googleapis.com/css?family=Nunito" rel="stylesheet">\n\n <!-- Styles -->\n <style>\n html, body {\n background-color: #fff;\n color: #636b6f;\n font-family: \'Nunito\', sans-serif;\n font-weight: 100;\n height: 100vh;\n margin: 0;\n }\n\n .full-height {\n height: 100vh;\n }\n\n .flex-center {\n align-items: center;\n display: f'
What is the issue here?
Your payload has invalid Host, use real host, from your variable
Related
I am trying to scrape formularylookup.com, a site with information on the market for pharmaceuticals.
It requires a login:
username: -
password: -
I need the information for the medicine called Rybelsus.
When I look into the Inspect-> Network -> XHR I suspect there could be an easy way to get the required data form this page:
https://formularylookup.com/Formulary/Coverage?ProductId=237171&ProductName=Rybelsus&ChannelId=1&DrugTypeId=3&StateId=all&Options=SummaryCoverages
I identified this site, which might give an idea of how to connect to formularylookup.com, but I am very inexperienced with connecting to API's.
Here's my code:
import requests
from bs4 import BeautifulSoup
url ="https://api.mmitnetwork.com/Formulary/v1/Products?Name=rybelsus"
params = {
"ProductId":"237171",
"productSearch":"Rybelsus"}
headers = {
"authorization":"Bearer H-oa4ULGls2Cpu8U6hX4myixRoFIPxfj",
"Access-Token":"H-oa4ULGls2Cpu8U6hX4myixRoFIPxfj",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36",
"X-Requested-With": "XMLHttpRequest",
"Host": "formularylookup.com",
"X-NewRelic-ID": "XAYCVFZSGwcGU1lXBAI="
}
res = requests.get(url ,params=params ,headers = headers)
soup = BeautifulSoup(res.content, "lxml")
print(soup.prettify())
Which gives me the following response:
<!DOCTYPE html>
<html>
<head>
<title>
The resource cannot be found.
</title>
<meta content="width=device-width" name="viewport"/>
<style>
body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;}
p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
.marker {font-weight: bold; color: black;text-decoration: none;}
.version {color: gray;}
.error {margin-bottom: 10px;}
.expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
#media screen and (max-width: 639px) {
pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
}
#media screen and (max-width: 479px) {
pre { width: 280px; }
}
</style>
</head>
<body bgcolor="white">
<span>
<h1>
Server Error in '/' Application.
<hr color="silver" size="1" width="100%"/>
</h1>
<h2>
<i>
The resource cannot be found.
</i>
</h2>
</span>
<font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
<b>
Description:
</b>
HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable. Please review the following URL and make sure that it is spelled correctly.
<br/>
<br/>
<b>
Requested URL:
</b>
/Formulary/v1/Products
<br/>
<br/>
<hr color="silver" size="1" width="100%"/>
<b>
Version Information:
</b>
Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1590.0
</font>
</body>
</html>
<!--
[HttpException]: The controller for path '/Formulary/v1/Products' was not found or does not implement IController.
at System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(RequestContext requestContext, Type controllerType)
at System.Web.Mvc.DefaultControllerFactory.CreateController(RequestContext requestContext, String controllerName)
at System.Web.Mvc.MvcHandler.ProcessRequestInit(HttpContextBase httpContext, IController& controller, IControllerFactory& factory)
at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
-->
<!--
This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using <customErrors mode="Off"/>. Consider using <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> in production environments.-->
Update: I get an 404 error. Not sure why.
Below code will help you,
import requests
headers = {
'Accept': '*/*',
'X-Requested-With': 'XMLHttpRequest',
'Access-Token': '7Lq-KkDx2fCO_3kG90pLEpBS9Ssh62IQ',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36',
'Is-Session-Expired': 'false',
'Referer': 'https://formularylookup.com/',
}
response = requests.get('https://formularylookup.com/Formulary/Coverage?ProductId=237171&ProductName=Rybelsus&ChannelId=1&DrugTypeId=3&StateId=AL&Options=SummaryCoverages', headers=headers)
print(response.json())
Note: 'Is-Session-Expired': 'false' is very important in the header otherwise you'll get 404 error.
See it in action here
I'm trying to make an HTTPS proxy server, but I cant make a connection to any server.
Edit: this is the part of the code that, after a client connect to my server, I get the message and try to send to the web. This is the message of a FireFox client trying to connect to Google:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = socket.create_connection((google.com, 443))
ssl_sock = ssl.wrap_socket(sock)
fullData=b''
ssl_sock.send(b'CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n')
while 1:
# receive data from web server
data = ssl_sock.recv(4026)
print(data)
if (len(data) > 0):
fullData+=data
else:
break
clientSock.send(fullData)
Google should got me a ok message but its getting me an error
HTTP/1.1 405 Method Not Allowed
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1592
Date: Fri, 24 May 2019 05:28:17 GMT
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
Connection: close
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 405 (Method Not Allowed)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>405.</b> <ins>That\xe2\x80\x99s an error.</ins>
<p>The request method <code>CONNECT</code> is inappropriate for the URL <code>/</code>. <ins>That\xe2\x80\x99s all we know.</ins>
I am trying to stream a video using nginx-rtmp with rtmp protocol.which i setup on my vm(started headless). I created a index.html and embedded on it JWPlayer with the rtmp url. On my localhost,
how can I play the video from the browser
and is my configuration all right ?
I can't able to browse the video from my real machine
index.html
<!DOCTYPE HTML>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>VoD Example</title>
<script src="http://jwpsrv.com/library/s7iNvOAyEeSMdQ4AfQhyIQ.js"></script>
</head>
<body>
<div id='videotest'></div>
<script type='text/javascript'>
jwplayer('videotest').setup({
file: 'rtmp://192.168.x.xxx/vod/bird.mp4',
width: '50%',
aspectratio: '16:9'
});
</script>
</body>
</html>
here is my nginx.conf
#user nobody;
worker_processes 1;
error_log logs/error.log debug;
events {
worker_connections 1024;
}
http {
proxy_ignore_client_abort on;
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8080;
server_name 0.0.0.0;
# rtmp stat
location /stat {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}
location /stat.xsl {
# you can move stat.xsl to a different location
root /var/www/;
}
# rtmp control
location /control {
rtmp_control all;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
rtmp {
server {
listen 1935;
notify_method get;
chunk_size 8192;
application vod {
allow play all;
wait_video on;
play /var/www/html/video;
push rtmp://192.168.x.xxx/vod/;
}
}
}
I kept a videos /var/www/html/video/bird.mp4
access.log
192.168.x.xx - - [15/Sep/2018:12:25:21 +0530] "GET /stat.xsl HTTP/1.1" 304 0 "http://192.168.x.xxx:8080/stat" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/69.0.3497.81 Chrome/69.0.3497.81 Safari/537.36"
error.log
2018/09/15 12:24:49 [notice] 23499#0: nginx/1.15.1
2018/09/15 12:24:49 [notice] 23499#0: built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
2018/09/15 12:24:49 [notice] 23499#0: OS: Linux 4.9.0-6-amd64
2018/09/15 12:24:49 [notice] 23499#0: getrlimit(RLIMIT_NOFILE): 1024:1048576
2018/09/15 12:24:49 [notice] 23500#0: start worker processes
2018/09/15 12:24:49 [notice] 23500#0: start worker process 23501
2018/09/15 12:25:45 [info] 23501#0: *2 client closed connection while waiting for request, client: 192.168.x.xx, server: 0.0.0.0:8080
2018/09/15 12:26:15 [info] 23501#0: *3 client closed connection while waiting for request, client: 192.168.x.xx, server: 0.0.0.0:8080
salt-api returns HTTP/1.1 500 Internal Server Error when trying to log in:
curl -si localhost:8000/login -H "Accept: application/json" -d username='<notshown>' -d password='<notshown>' -d eauth='pam'
HTTP/1.1 500 Internal Server Error
Content-Length: 1607
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/3.2.2
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Thu, 14 Apr 2016 13:23:49 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 6a529d945c6654848c531337c1a1193f8635b482
Content-Type: text/html;charset=utf-8
Set-Cookie: session_id=6a529d945c6654848c531337c1a1193f8635b482; expires=Thu, 14 Apr 2016 23:23:49 GMT; Path=/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>500 Internal Server Error</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}
#traceback {
color: red;
}
</style>
</head>
<body>
<h2>500 Internal Server Error</h2>
<p>Configuration for external_auth could not be read.</p>
<pre id="traceback">Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.6/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 506, in hypermedia_handler
ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 1525, in POST
'Configuration for external_auth could not be read.')
HTTPError: (500, 'Configuration for external_auth could not be read.')
</pre>
<div id="powered_by">
<span>Powered by CherryPy 3.2.2</span>
</div>
</body>
</html>
When you enter the wrong credentials, the error is different so it definitely knows that the credentials entered before are correct:
curl -si localhost:8000/login -H "Accept: application/json" -d username='<notshown>' -d password='<notshown>' -d eauth='pam'
HTTP/1.1 401 Unauthorized
Content-Length: 1586
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/3.2.2
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Thu, 14 Apr 2016 13:23:34 GMT
Access-Control-Allow-Origin: *
Content-Type: text/html;charset=utf-8
Set-Cookie: session_id=b71f4afcc8f1caf3e0f5f33d0542c77bc1b9875a; expires=Thu, 14 Apr 2016 23:23:34 GMT; Path=/
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
<title>401 Unauthorized</title>
<style type="text/css">
#powered_by {
margin-top: 20px;
border-top: 2px solid black;
font-style: italic;
}
#traceback {
color: red;
}
</style>
</head>
<body>
<h2>401 Unauthorized</h2>
<p>Could not authenticate using provided credentials</p>
<pre id="traceback">Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.6/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 506, in hypermedia_handler
ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/salt/netapi/rest_cherrypy/app.py", line 1497, in POST
'Could not authenticate using provided credentials')
HTTPError: (401, 'Could not authenticate using provided credentials')
</pre>
<div id="powered_by">
<span>Powered by CherryPy 3.2.2</span>
</div>
</body>
</html>
/etc/salt/master:
rest_cherrypy:
port: 8000
host: 0.0.0.0
disable_ssl: true
debug: true
.....
external_auth:
pam:
<notshownhere>:
-.*
- '#runner'
- '#wheel'
versions used:
salt-api-2015.8.8-2.el6.noarch
salt-master-2015.8.8-2.el6.noarch
salt-minion-2015.8.8-2.el6.noarch
salt-2015.8.8-2.el6.noarch
Any idea why this is happening?
I had a typo in my master configuration:
I had:
external_auth:
pam:
<notshownhere>:
-.*
- '#runner'
- '#wheel'
Shoud be:
external_auth:
pam:
<notshownhere>:
- .*
- '#runner'
- '#wheel'
I am trying to invoke a node.js REST api (http://localhost:3000/users) from WSO2 Api Mgmt on-prem configuration but it doesn't work. The node.js REST api works fine from a browser/Postman. When called from WSO2, it returns empty response.
Endpoint exposed from WSO2 AM is http://localhost:801/api/v1/users and it is supposed to call the node.js REST api. I tried to use the debug features in WSO2 and found that the call never reaches the node.js endpoint due to some very long exception dump displayed in the console.
I am using Authorization header for the WSO2 front and it seems to work fine.
Though I have worked a bit in node.js, I am super-new to WSO2. So any help will be much appreciated.
Edit: I am using WSO2 AM 1.9.0
Response body from WSO2 API console given below -
<html>
<head>
<title>Apache Tomcat/7.0.55 - Error report</title>
<style>
<!
--H1
{
font-family: Tahoma,Arial,sans-serif;
color: white;
background-color: #525D76;
font-size: 22px;
}
H2
{
font-family: Tahoma,Arial,sans-serif;
color: white;
background-color: #525D76;
font-size: 16px;
}
H3
{
font-family: Tahoma,Arial,sans-serif;
color: white;
background-color: #525D76;
font-size: 14px;
}
BODY
{
font-family: Tahoma,Arial,sans-serif;
color: black;
background-color: white;
}
B
{
font-family: Tahoma,Arial,sans-serif;
color: white;
background-color: #525D76;
}
P
{
font-family: Tahoma,Arial,sans-serif;
background: white;
color: black;
font-size: 12px;
}
A
{
color: black;
}
A.name
{
color: black;
}
HR
{
color: #525D76;
}
-- ></style>
</head>
<body>
<h1>
HTTP Status 405 - HTTP method GET is not supported by this URL</h1>
<hr size="1" noshade="noshade">
<p>
<b>type</b> Status report</p>
<p>
<b>message</b> <u>HTTP method GET is not supported by this URL</u></p>
<p>
<b>description</b> <u>The specified HTTP method is not allowed for the requested resource.</u></p>
<hr size="1" noshade="noshade">
<h3>
Apache Tomcat/7.0.55</h3>
</body>
</html>
Edit : API Manager log
[2015-06-23 14:36:04,903] INFO - HandlerUtils Massage Info: Transaction id=1499
743816386524554259 Message direction=IN Server name=192.168.1.5:9763 Timestam
p=1435050364903 Service name=__SynapseService Operation Name=mediate
[2015-06-23 14:36:04,906] DEBUG - HTTPEndpoint Sending message through endpoint
: admin--GetUsers_APIproductionEndpoint_0 resolving to address = http://127.0.0.
1:3000/users
[2015-06-23 14:36:04,909] DEBUG - HTTPEndpoint SOAPAction: null
[2015-06-23 14:36:04,910] DEBUG - HTTPEndpoint WSA-Action: null
[2015-06-23 14:36:04,926] INFO - HandlerUtils Massage Info: Transaction id=1499
743816386524554259 Message direction=OUT Server name=192.168.1.5:9763 Timesta
mp=1435050364926 Service name=__SynapseService Operation Name=mediate
Edit : Remaining details from API log
Curl
curl -X GET --header "Accept: application/json" --header "Authorization: Bearer 6a931d4fb2f6c85be750ee7cea8bc094" "https://localhost:9443/api/v1/users"
Request URL
https://localhost:9443/api/v1/users
Response Code
405
Response Headers
{
"date": "Tue, 23 Jun 2015 09:05:49 GMT",
"server": "WSO2 Carbon Server",
"content-type": "text/html;charset=utf-8",
"content-length": "1065",
"content-language": "en"
}
You have to check the GET checkbox value while giving the endpoint address.
Use this API sequence
<?xml version="1.0" encoding="UTF-8"?>
<api xmlns="http://ws.apache.org/ns/synapse"
name="nodejsAPI"
context="/api">
<resource methods="GET" protocol="http" uri-template="/v1/users">
<inSequence>
<log level="full"/>
<send>
<endpoint>
<http method="GET"
uri-template="http://localhost:3000/users/"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</resource>
</api>
Your API creation has probably had some issues, and therefore it has not been mapped correctly to your nodeJS REST endpoint.
In the API creation wizard, you have the option to test whether the API-M can reach your endpoint. This step can verify that there is no issue in connecting.
I have written a blog post on step by step process for your use case. Please follow these steps, and hopefully, it should work.
Make sure the URL pattern field and production endpoint field have correct values.