I am struggling with Servant and the CORS configuration: I am exposing and API through Servant and I have the following configuration:
-- Wai application initialization logic
initializeApplication :: IO Application
initializeApplication = do
let frontCors = simpleCorsResourcePolicy { corsOrigins = Just ([pack "https://xxxx.cloudfront.net"], True)
, corsMethods = ["OPTIONS", "GET", "PUT", "POST"]
, corsRequestHeaders = simpleHeaders }
return
$ cors (const $ Just $ frontCors)
$ serve (Proxy #API)
$ hoistServer (Proxy #API) toHandler server
When I perform a query like this through Chromium (by copying and pasting):
curl 'https://api.xxx/' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Referer: https://xxx.cloudfront.net' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36' \
-H 'Authorization: Bearer XXX==' \
--compressed
It works but if I copy-paste the fetch query in the dev console:
fetch("https://api.xxx", {
"headers": {
"accept": "application/json, text/plain, */*",
"authorization": "Bearer XXX=="
},
"referrer": "https://xxx.cloudfront.net/",
"referrerPolicy": "no-referrer-when-downgrade",
"body": null,
"method": "GET",
"mode": "cors",
"credentials": "include"
});
I get:
> Access to fetch at 'https://api.xxx' from origin 'https://xxx.cloudfront.net' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
polyfills-es2015.3eb4283ca820c86b1337.js:1 GET https://api.xxx net::ERR_FAILED
e.fetch # polyfills-es2015.3eb4283ca820c86b1337.js:1
> (anonymous) # VM20:1
> x:1 Uncaught (in promise) TypeError: Failed to fetch
Any hints regarding that? Especially why it works in cUrl and not in Chromium?
Thanks in advance.
It was a basic CORS issue, in fact, sending Authorization without having it in the corsRequestHeaders makes the request rejected.
I should have written:
, corsRequestHeaders = ["Authorization", "Content-Type"]
Related
I'm using request module to send an http post request to a server which respond http 302 with the location redirect url "strangely" encoded:
'https:%3B//xxx.xxx.com?%3Bsrcext%3Dvalue&%3Berl=rrf
When i do the same request in chrome, chrome show me the location redirect URL as:
"https://xxx.xxx.com?srcext=value&erl=rrf"
Curl give me the same "response" as request module:
'https:%3B//xxx.xxx.com?%3Bsrcext%3Dvalue&%3Berl=rrf
Request options "followRedirect/followAllRedirects" fail to follow the link, curl option -L fail to follow the link.
No problem in chrome
I know i can apply "by hand" a method to decode the url "properly" but this is tricky as some parameters included need to stay encoded.
Is there an option in curl or request that i miss to directly "decode" the url and make the redirection option works ?
Thanks in advance
=========== EDIT =======
for request i'm using
request({
method: "POST",
url: "https://balbalab.com",
headers: {
'Cookie': "XXXXX",
'Accept-Encoding': 'gzip, deflate, fr',
'Accept-Language': 'fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36',
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
"Pragma": "no-cache",
"Content-Type": "application/x-www-form-urlencoded"
},
gzip: true,
form: formFields,
followRedirect: true,
followAllRedirects: true
}
for curl i'm using:
curl -X POST "https://url1.url1.com" \
-H 'Accept-Encoding: gzip, deflate, fr' \
-H 'Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Cache-Control: no-cache' \
-H 'Connection: keep-alive' \
-H "Pragma: no-cache" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d 'xx=ff' \
-L
How do you parse the URL? It seems like you are missing url-encoded flag in your parser.
I am trying this following curl request
curl 'https://www.nseindia.com/api/historical/cm/equity?symbol=COALINDIA&series=\[%22EQ%22\]&from=03-05-2020&to=03-05-2021&csv=true' \
-H 'authority: www.nseindia.com' \
-H 'accept: */*' \
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/88.0.4324.182 Safari/537.36' \
-H 'x-requested-with: XMLHttpRequest' \
-H 'sec-gpc: 1' \
-H 'sec-fetch-site: same-origin' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-dest: empty' \
-H 'referer: https://www.nseindia.com/get-quotes/equity?symbol=COALINDIA' \
-H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
-H 'cookie: ak_bmsc=2D5CCD6F330B77016DD02ADFD8BADB8A58DDD69E733C0000451A9060B2DF0E5C~pllIy1yQvFABwPqSfaqwV4quP8uVOfZBlZe9dhyP7+7vCW/YfXy32hQoUm4wxCSxUjj8K67PiZM+8wE7cp0WV5i3oFyw7HRmcg22nLtNY4Wb4xn0qLv0kcirhiGKsq4IO94j8oYTZIzN227I73UKWQBrCSiGOka/toHASjz/R10sX3nxqvmMSBlWvuuHkgKOzrkdvHP1YoLPMw3Cn6OyE/Z2G3oc+mg+DXe8eX1j8b9Hc=; nseQuoteSymbols=[{"symbol":"COALINDIA","identifier":null,"type":"equity"}]; nsit=X5ZCfROTTuLVwZzLBn7OOtf0; AKA_A2=A; bm_mi=6CE0B82205ACE5A1F72250ACDDFF563E~LZ4/HQ257rSMBPCrxy0uSDvrSxj4hHpLQqc8R5JZOzUZYo1OqZg5Q/GOt88XNtMbsWM8bB22vtCXzvksGwPcC/bH2nPFEZr0ci6spQ4GOpCa/TM7soc02HVf0tyDTkmg/ZdLZlWzond4r0vn+QpSB7f3fiVza1Gdx9OaFL1i3rvqe1OKmFONreHEue20PL0hlREVWeLcFM/5DxKArPwzCSopPp62Eea1510iivl7GmY=; nseappid=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcGkubnNlIiwiYXVkIjoiYXBpLm5zZSIsImlhdCI6MTYyMDA2MTQ5OSwiZXhwIjoxNjIwMDY1MDk5fQ.YBTQ0MqRayD3QBM3V6zUt5zbRRICkbIhWWNedkDYrdU; bm_sv=C49B743B48F174C77F3DDAD188AA6D87~bm5TD36snlaRLx9M5CS+FOUicUcbVV3OIKjZU2WLwd1PtHYUum7hnBfYeUCDv+5Xdb9ADklnmm1cwZGJJbiBstcA6c5vju53C7aTFBorl8SJZjBN/4ku61oz0ncrQYCaSxkFGkRRY9VMWm6SpQwHXfMsUzc/Qk7301zs7KZuGCY=' \
--compressed
This gives us the required response (example below)
"Date ","series ","OPEN ","HIGH ","LOW ","PREV. CLOSE ","ltp ","close ","vwap ","52W H","52W L ","VOLUME ","VALUE ","No of trades "
"03-May-2021","EQ","133.00","133.45","131.20","133.05","132.20","132.20","132.21","163.00","109.55",10262391,"1,356,811,541.80",59409
But if I use the following python script to get the data
import requests
headers = {
'authority': 'www.nseindia.com',
'accept': '*/*',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36',
'x-requested-with': 'XMLHttpRequest',
'sec-gpc': '1',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://www.nseindia.com/get-quotes/equity?symbol=COALINDIA',
'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8','cookie':'ak_bmsc=2D5CCD6F330B77016DD02ADFD8BADB8A58DDD69E733C0000451A9060B2DF0E5C~pllIy1yQvFABwPqSfaqwV4quP8uVOfZBlZe9dhyP7+7vCW/YfXy32hQoUm4wxCSxUjj8K67PiZM+8wE7cp0WV5i3oFyw7HRmcg22nLtNY4Wb4xn0qLv0kcirhiGKsq4IO94j8oYTZIzN227I73UKWQBrCSiGOka/toHASjz/R10sX3nxqvmMSBlWvuuHkgKOzrkdvHP1YoLPMw3Cn6OyE/Z2G3oc+mg+DXe8eX1j8b9Hc=; nseQuoteSymbols=[{"symbol":"COALINDIA","identifier":null,"type":"equity"}]; nsit=X5ZCfROTTuLVwZzLBn7OOtf0; AKA_A2=A; bm_mi=6CE0B82205ACE5A1F72250ACDDFF563E~LZ4/HQ257rSMBPCrxy0uSDvrSxj4hHpLQqc8R5JZOzUZYo1OqZg5Q/GOt88XNtMbsWM8bB22vtCXzvksGwPcC/bH2nPFEZr0ci6spQ4GOpCa/TM7soc02HVf0tyDTkmg/ZdLZlWzond4r0vn+QpSB7f3fiVza1Gdx9OaFL1i3rvqe1OKmFONreHEue20PL0hlREVWeLcFM/5DxKArPwzCSopPp62Eea1510iivl7GmY=; nseappid=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcGkubnNlIiwiYXVkIjoiYXBpLm5zZSIsImlhdCI6MTYyMDA2MTQ5OSwiZXhwIjoxNjIwMDY1MDk5fQ.YBTQ0MqRayD3QBM3V6zUt5zbRRICkbIhWWNedkDYrdU; bm_sv=C49B743B48F174C77F3DDAD188AA6D87~bm5TD36snlaRLx9M5CS+FOUicUcbVV3OIKjZU2WLwd1PtHYUum7hnBfYeUCDv+5Xdb9ADklnmm1cwZGJJbiBstcA6c5vju53C7aTFBorl8SJZjBN/4ku61oz0ncrQYCaSxkFGkRRY9VMWm6SpQwHXfMsUzc/Qk7301zs7KZuGCY=',}
params = (
('symbol', 'COALINDIA'),
('series', '/["EQ"/]'),
('from', '30-04-2021'),
('to', '03-05-2021'),
('csv', 'true'),
)
response = requests.get('https://www.nseindia.com/api/historical/cm/equity', headers=headers, params=params)
It gets stuck in the last line.
I am using python3.9 and urllib3.
Not sure what is the problem.
This url downloads a csv file from the website.
You have to jump through some loops with Python to get the file you're after. Mainly, you need to get the request header cookie part right, otherwise you'll keep getting 401 code.
First, you need to get the regular cookies from the authority www.nseindia.com. Then, you need to get the bm_sv cookie from the https://www.nseindia.com/json/quotes/equity-historical.json. Finally, add something that's called nseQuoteSymbols.
Glue all that together and make the request to get the file.
Here's how:
from urllib.parse import urlencode
import requests
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) '
'AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/88.0.4324.182 Safari/537.36',
'x-requested-with': 'XMLHttpRequest',
'referer': 'https://www.nseindia.com/get-quotes/equity?symbol=COALINDIA',
}
payload = {
"symbol": "COALINDIA",
"series": '["EQ"]',
"from": "04-04-2021",
"to": "04-05-2021",
"csv": "true",
}
api_endpoint = "https://www.nseindia.com/api/historical/cm/equity?"
nseQuoteSymbols = 'nseQuoteSymbols=[{"symbol":"COALINDIA","identifier":null,"type":"equity"}]; '
def make_cookies(cookie_dict: dict) -> str:
return "; ".join(f"{k}={v}" for k, v in cookie_dict.items())
with requests.Session() as connection:
authority = connection.get("https://www.nseindia.com", headers=headers)
historical_json = connection.get("https://www.nseindia.com/json/quotes/equity-historical.json", headers=headers)
bm_sv_string = make_cookies(historical_json.cookies.get_dict())
cookies = make_cookies(authority.cookies.get_dict()) + nseQuoteSymbols + bm_sv_string
connection.headers.update({**headers, **{"cookie": cookies}})
the_real_slim_shady = connection.get(f"{api_endpoint}{urlencode(payload)}")
csv_file = the_real_slim_shady.headers["Content-disposition"].split("=")[-1]
with open(csv_file, "wb") as f:
f.write(the_real_slim_shady.content)
Output -> a .csv file that looks like this:
I've created a Bash script to get the data from the url using rest API from a appliance using username, password and saving the Session ID into a Variable and then using the session ID to get the data into csv format which is working fine.
I want to change the bash code into python3 code as i'm parsing it using pandas.
Bash Code:
#!/bin/bash
sessionID=$(curl -k -H "accept: application/json" -H "content-type: application/json" -H "x-api-version: 120" -d '{"userName":"administrator","password":"adminpass"}' -X POST https://hpe.sysnergy.com/rest/login-sessions | jq -r ".sessionID")
curl -k -H 'accept: application/json' \
-H 'content-type: text/csv' \
-H 'x-api-version: 2' \
-H "auth: $sessionID" \
-X GET https://hpe.sysnergy.com/rest/resource-alerts
Python Version of tries code:
#!/usr/bin/python3
import requests
import json
url = "https://hpe.sysnergy.com/rest/login-sessions"
data = {'username': 'administrator', 'password': 'adminpass'}
headers = {'Content-type': 'text/csv', 'Accept': 'application/json', 'x-api-version': 2}
r = requests.post(url, data=json.dumps(data), headers=headers)
print(r)
I am getting below error:
Error:
requests.exceptions.InvalidHeader: Value for header {x-api-version: 2} must be of type str or bytes, not <class 'int'>
if i convert int to str as '2' then it gives another ssl error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='hpe.synerg.com', port=443): Max retries exceeded with url: /rest/login-sessions (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))
EDIT:
I have tried little different approach to get the same code format as bash in python but now it returns new error with new response code.
import os
import requests
sessionID = os.getenv('sessionID')
headers = {
'accept': 'application/json',
'content-type': 'text/csv',
'x-api-version': '2',
'auth': f"{sessionID}",
}
data = '{"userName":"administrator","password":"adminpassword"}'
response = requests.post('https://hpe.synergy.com/rest/login-sessions', headers=headers, data=data, verify=False)
print(response)
Error:
/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'hpe.synergy.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
<Response [415]>
Please help or suggest the way to achieve same function in the python.
You first need to make a POST request to get the sessionID, then you need to make a GET request. Also note the headers are slightly different for the 2 requests. Something like this should work:
import requests
session = requests.Session()
url = "https://hpe.sysnergy.com/rest/login-sessions"
credentials = {"userName": "administrator", "password": "adminpass"}
headers = {"accept": "application/json",
"content-type": "application/json",
"x-api-version": "120",
}
response = session.post(url, headers=headers, json=credentials, verify=False)
session_id = response.json()["sessionID"]
url = "https://hpe.sysnergy.com/rest/resource-alerts"
headers = {"accept": "application/json",
"content-type": "text/csv",
"x-api-version": "2",
"auth": session_id,
}
response = session.get(url, headers=headers, verify=False)
print(response)
#print(response.content) # returns bytes
#print(response.text) # returns string
I've found a lot of CORS-related questions, but none of them solved my problem. As I'm quite new to web-dev, I'd appreciate any help with getting my project infrastructure up and running.
My application is made of two parts:
Server part
A server made with Haskell, Servant. The most important endpoint I've been trying to connect to is:
type Unprotected =
"login"
:> ReqBody '[JSON] Credentials
:> Verb 'POST 204 '[JSON] (Headers '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie] NoContent)
It's based on the project described here: https://github.com/haskell-servant/servant-auth
I've already found a SO question about CORS: CORS header ‘Access-Control-Allow-Origin’ missing in servant, but that didn't help.
During development I launch the server on port 8081.
UI Part
I have also a UI project, that's hosted independently on port 8833. It uses WebPack. I've also found some SO posts regarding CORS, advising to use:
devServer: {
headers: {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Credentials": "true",
"Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, PATCH, OPTIONS",
"Access-Control-Allow-Headers": "X-Requested-With, content-type, Authorization"
}
That didn't help either.
Problem description:
When I make a "fetch" call in the UI project, trying to communicate with the Servant endpoint I get:
Access to fetch at 'http://localhost:8081/login' from origin 'http://localhost:8833'
has been blocked by CORS policy: Response to preflight request doesn't pass access control
check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an
opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource
with CORS disabled.
The way I use fetch:
let data = {
credentialsUserName: this.state.login,
credentialsPassword: this.state.password
};
let opts : RequestInit = {
method: "POST",
headers: new Headers({ "Content-Type": "application/json" }),
body: JSON.stringify(data)
};
fetch("http://localhost:8081/login", opts).then((value) => {
console.log(value);
// do something later
});
Questions
What shall I do in order to make the fetch work fine with the servant endpoint. Also, is there anything wrong with my application architecture ? Maybe there's a better way to design it.
Last thing: The entire code of my application is quite long, so I decided to put only to most important parts here. Feel free to request more if needed.
Edit 1
After sending the fetch request I can see two entries in "Network" tab in my browser.
The first one with the following headers:
content-type: application/json
Referer: http://localhost:8833/Login
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36
and json data I included in its body. It's status is "(failed) net::ERR_FAILED".
The other's status is 400 and contains more headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: POST
Connection: keep-alive
Host: localhost:8081
Origin: http://localhost:8833
Referer: http://localhost:8833/Login
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36
Interestingly, In "General" tab I can see:
Request URL: http://localhost:8081/login
Request Method: OPTIONS
Status Code: 400 Bad Request
Remote Address: 127.0.0.1:8081
Referrer Policy: no-referrer-when-downgrade
Request Method "OPTIONS" looks mysterious. Why is it there ?
As a quick guess I tried adding "OPTIONS" to CORS definition on Haskell side, but no luck this time either:
port :: Int
port = 8081
corsPolicy :: Middleware
corsPolicy = cors (const $ Just policy)
where
policy = simpleCorsResourcePolicy
{ corsMethods = [ "GET", "POST", "PUT", "OPTIONS" ]}
main :: IO ()
main = do
migrateDB
jwtConfig <- getJwtConfig
putStrLn $ "Serving endpoint " ++ (show port)
run port $ corsPolicy $ serveWithContext proxy (context cookieConfig jwtConfig) (appAPI cookieConfig jwtConfig)
EDIT 2
Thanks for the help - I've managed to configure Servant, here it is:
corsPolicy :: Middleware
corsPolicy = cors (const $ Just policy)
where
policy = simpleCorsResourcePolicy
{
corsMethods = [ "GET", "POST", "PUT", "OPTIONS" ],
corsOrigins = Just (["http://localhost:8833"], True),
corsRequestHeaders = [ "authorization", "content-type" ]
}
The remaining part of Main as in my previous listing.
By default Haskell CORS doesn't accept any headers. Not even "Content-Type", which I intend to send.
I'd like to pass a raw HTTP request like:
GET /foo/bar HTTP/1.1
Host: example.org
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept: */*
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: http://example.org/test
Cookie: foo=bar; lorem=ipsum;
And generate the python request such as:
import requests
burp0_url = "http://example.org:80/foo/bar"
burp0_cookies = {"foo": "bar", "lorem": "ipsum"}
burp0_headers = {"User-Agent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8", "Accept": "*/*", "Accept-Language": "fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3", "Accept-Encoding": "gzip,deflate", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Keep-Alive": "115", "Connection": "keep-alive", "Content-Type": "application/x-www-form-urlencoded", "X-Requested-With": "XMLHttpRequest", "Referer": "http://example.org/test"}
requests.get(burp0_url, headers=burp0_headers, cookies=burp0_cookies)
Is there a library for that?
I could not find an existing library that does this conversion, but there is a Python library to convert curl commands to python requests code.
https://github.com/spulec/uncurl
e.g.
import uncurl
print(uncurl.parse('curl --header "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" --compressed --header "Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3" --header "Connection: keep-alive" --header "Content-Type: application/x-www-form-urlencoded" --cookie "foo=bar; lorem=ipsum;" --header "Keep-Alive: 115" --header "Referer: http://example.org/test" --user-agent "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; fr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" --header "X-Requested-With: XMLHttpRequest" https://example.org/foo/bar '))
I haven't found a Python library to transform raw HTTP into such a curl command. However, this Perl program does it.
Like this:
$ cat basic
GET /index.html HTTP/2
Host: example.com
Authorization: Basic aGVsbG86eW91Zm9vbA==
Accept: */*
$ ./h2c < basic
curl --http2 --header User-Agent: --user "hello:youfool" https://example.com/index.html
You could either call it from your python script, use a Python-Perl bridge or try to port it.
Postman also allows you to convert raw HTTP requests directly to python requests code, using its Code snippet generator. Although, it seems this can only be done via the GUI. It's also not Open-Source, so you can't access the code that does this transformation.
I needed something that can generate a request and couldn't find it so ended up writing it in gist:
class RequestParser(object):
def __parse_request_line(self, request_line):
request_parts = request_line.split(' ')
self.method = request_parts[0]
self.url = request_parts[1]
self.protocol = request_parts[2] if len(request_parts) > 2 else DEFAULT_HTTP_VERSION
def __init__(self, req_text):
req_lines = req_text.split(CRLF)
self.__parse_request_line(req_lines[0])
ind = 1
self.headers = dict()
while ind < len(req_lines) and len(req_lines[ind]) > 0:
colon_ind = req_lines[ind].find(':')
header_key = req_lines[ind][:colon_ind]
header_value = req_lines[ind][colon_ind + 1:]
self.headers[header_key] = header_value
ind += 1
ind += 1
self.data = req_lines[ind:] if ind < len(req_lines) else None
self.body = CRLF.join(self.data)
def __str__(self):
headers = CRLF.join(f'{key}: {self.headers[key]}' for key in self.headers)
return f'{self.method} {self.url} {self.protocol}{CRLF}' \
f'{headers}{CRLF}{CRLF}{self.body}'
def to_request(self):
req = requests.Request(method=self.method,
url=self.url,
headers=self.headers,
data=self.data, )
return req