replicate curl command python 3 urllib request API - python-3.x

This problem is kind of driving me crazy.
I'm doing a very simple python 3 script to manage an API in a public website.
I am able to do it with curl, but not in pyhton.
Can't use either requests library or curl in my real environment, just for tests
This is working:
curl -d "credential_0=XXXX&credential_1=XXXXXX" -c cookiefile.txt https://XXXXXXXXXXXXXXX/LOGIN
curl -d 'json={"devices" : ["00:1A:1E:29:73:B2","00:1A:1E:29:73:B2"]}' -b cookiefile.txt -v https://XXXXXXXXX/api-path --trace-ascii /dev/stdout
and we can see this in the curl debug:
Send header, 298 bytes (0x12a)
0000: POST /api-path HTTP/1.1
0034: Host: XXXXXXXXXXXXXXXX
0056: User-Agent: curl/7.47.0
006f: Accept: /
007c: Cookie: csrf_token=751b6bd9-0290-496b-820e-XXXXXXXX; session
00bc: =XXXXXX-6d29-4cf9-8907-XXXXXXXXXXXX
00e3: Content-Length: 60
00f7: Content-Type: application/x-www-form-urlencoded
0128:
=> Send data, 60 bytes (0x3c)
0000: json={"devices" : ["00:1A:1E:29:73:B2","00:1A:1E:29:73:B2"]}
== Info: upload completely sent off: 60 out of 60 bytes
This is the python code to replicate the second request, which is the problematic one
string_query={"devices" : [ "34:FC:B9:CE:14:7E","00:1A:1E:29:73:B2" ]}
jsonbody_url=urllib.parse.urlencode(string_query)
jsonbody_url=jsonbody_url.encode("utf-8")
req=urllib.request.Request(url,data=jsonbody_url,headers={"Cookie" :
cookie,"Content-Type": "application/x-www-form-urlencoded","User-
Agent":"curl/7.47.0","charset":"UTF-8","Content-
length":len(jsonbody_url),
"Connection": "Keep-Alive"},method='POST')
And the server is completely ignoring the Json content.
Everything else is working, login and other url parameters from the same API
Any ideas?

Try this:
import requests
string_query={"devices" : [ "34:FC:B9:CE:14:7E","00:1A:1E:29:73:B2" ]}
headers={
"Cookie" : cookie,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent":"curl/7.47.0",
"charset":"UTF-8",
"Connection": "Keep-Alive"
}
response = requests.post(url,data=string_query,headers=headers)
print(response.content)

Related

DocuSign, Node SDK, JWT auth, Envelope fetch, "The custom error module does not recognize this error."

[FINAL UPDATE]
Fixed per this thread: https://github.com/docusign/docusign-esign-node-client/issues/295
Apparently related to security additions made to DocuSign's API.
I haven't encountered this issue before today. I gave a demo to a client just last week and the same code was working fine. Today I get the following:
{
"status": 444,
"response": {
"req": {
"method": "GET",
"url": "https://demo.docusign.net/restapi...",
"data": {},
"headers": {
"user-agent": "node-superagent/3.8.2",
"x-docusign-sdk": "Node",
"authorization": "Bearer ABC...",
"content-type": "application/json",
"accept": "application/json"
}
},
"header": {
"content-type": "text/html",
"x-docusign-tracetoken": "2eeb8caa-8865-4898-bef9-d3611bfaa3f7",
"x-docusign-node": "DA2DFE5",
"date": "Fri, 17 Jun 2022 01:02:02 GMT",
"content-length": "54",
"connection": "close",
"strict-transport-security": "max-age=31536000; includeSubDomains"
},
"status": 444,
"text": "The custom error module does not recognize this error."
}
}
Through the node SDK I fetch a token using requestJWTUserToken, and set the apiclient's auth header, then I new-up an EnvelopesAPI instance. When I make a call to getEnvelope() or listStatusChanges(), then I get the error above. None of this code has changed in months, and I'm using the same integration key, account, private key, everything. I've demo'd it all a few times now - no issues.
An interesting observation: the error above gives me a URL and token. The token is valid, and if I make a request to the URL (the envelopes endpoint) via Postman using said token, then the request succeeds. So my calls through the SDK seem to be failing for some reason.
I can't seem to crack this one, and now I can't get around it given a couple demo systems that worked just last week.
I'm using the docusign-esign 5.17 module - upgraded from 5.15 in an attempt to fix the issue. No luck.
Where is this error coming from?
[Update 1]
I'm running my node app that is making requests through the DocuSign Node SDK against a proxy so I can see what the failing request actually look like:
They fail the same way.
HTTP/1.1 444
Content-Type: text/html
X-DocuSign-TraceToken: 338534c6-c8c3-4b01-9b66-35d697cd0053
X-DocuSign-Node: DA1DFE4
Date: Fri, 17 Jun 2022 03:55:07 GMT
Content-Length: 54
Vary: Accept-Encoding
Connection: close
Strict-Transport-Security: max-age=31536000; includeSubDomains
The custom error module does not recognize this error.
I'm using Proxyman to catch the request, and like Chrome or Firefox it will let you copy a request as a cURL command. If I copy the failing request as cURL, then run in at the terminal, it succeeds.
[MacBookPro0020]~/source/docusign/jwt-smoke-test:0 (master)
$ curl 'https://demo.docusign.net/restapi/v2.1/accounts/a0a4c81f-.../envelopes?envelope_ids=e750526f-...&envelope_ids=a38b794b...&envelope_ids=a5d8c586-...' \
-H 'Host: demo.docusign.net' \
-H 'User-Agent: node-superagent/3.8.2' \
-H 'X-DocuSign-SDK: Node' \
-H 'Node-Ver: v14.18.3' \
-H 'Authorization: Bearer ABCD...' \
-H 'Accept: application/json' \
-H 'Connection: close' \
-H 'Content-Type: application/json' \
--proxy http://localhost:9090
{"resultSetSize":"1","startPosition":"0","endPosition":"0","totalSetSize":"1","nextUri":"","previousUri":"","envelopes":[{"status":"created","documentsUri":"/envelopes/d97565c8...purgeState":"unpurged","envelopeIdStamping":"true","autoNavigation":"true","isSignatureProviderEnvelope":"false","allowComments":"true","anySigner":null,"envelopeLocation":"current_site"}]}
I'm using a JWT auth token, so again, I'm getting a valid token. Calls through the SDK consistently fail, but cURL'ing and manually requesting through Postman both succeed.
I'm at a loss.
Additional details: I see this same issue on MacOS and Windows (i.e., node app hosting docusign-esign). I'm using auth code grant to send envelopes and query envelope statuses and that works fine. I've used JWT Grant without issue up until this week (just demo'd automated functionality last week and it worked.) I haven't made any code changes to my DocuSign functionality, nor have my colleagues, at least according to the repo's history.
I don't recall ever encountering the error above before. I'd love to know why cURL'ing the same request succeeds. I'd rather not ditch the SDK and roll my own requests, but it wouldn't be difficult.
[Update 2]
Here's a simple repro - it's a quick and dirty copy of the QuickStart demo project for the node SDK. I'm using only docusign-esign.
Exact. same. issue.
Again, I can take that token and drop it into cURL or postman and the request will succeed. There's not a lot of code here. The token is valid.
async function main() {
// Data used
// dsConfig.dsClientId
// dsConfig.impersonatedUserGuid
// dsConfig.privateKey
// dsConfig.dsOauthServer
let dsConfig = dsConfig_customer; // defined globally
const jwtLifeSec = 10 * 60, // requested lifetime for the JWT is 10 min
dsApi = new docusign.ApiClient();
dsApi.setOAuthBasePath(dsConfig.dsOauthServer.replace('https://', '')); // it should be domain only.
const results = await dsApi.requestJWTUserToken(dsConfig.dsClientId,
dsConfig.impersonatedUserGuid, 'signature impersonation', dsConfig.privateKey,
jwtLifeSec);
console.log( results.body.access_token );
const userInfo = await dsApi.getUserInfo(results.body.access_token);
dsApi.setBasePath(userInfo.accounts[0].baseUri + '/restapi');
dsApi.addDefaultHeader( 'Authorization', 'Bearer ' + results.body.access_token );
const envelopesAPI = new docusign.EnvelopesApi(dsApi);
const res = await envelopesAPI.getEnvelope( dsConfig.accountID, 'e1917111-2900-48e8-9054-799169379c8a', null );
console.log(res);
return {
accessToken: results.body.access_token,
tokenExpirationTimestamp: expiresAt,
userInfo,
account: userInfo.accounts[0]
};
}
main().then(result => console.log(result)).catch(err=>console.error(err));
...
header: {
'content-type': 'text/html',
'x-docusign-tracetoken': '685b6226-a0d3-4547-94c7-df0216d884a3',
'x-docusign-node': 'DA2DFE188',
date: 'Fri, 17 Jun 2022 05:20:12 GMT',
'content-length': '54',
vary: 'Accept-Encoding',
connection: 'close',
'strict-transport-security': 'max-age=31536000; includeSubDomains'
},
statusCode: 444,
status: 444,
statusType: 4,
info: false,
ok: false,
redirect: false,
clientError: true,
serverError: false,
error: Error: cannot GET /restapi/v2.1/accounts/49754554-ABCD-.../envelopes/e1917111-2900-48e8-9054-799169379c8a (444)
...
I finally manage to resolve the issue by downgrading my Docusign SDK NodeJS version from 5.15.0 to 5.7.0 in my package.json file.
An issue relative to this can be find there, I really hope this issue will be resolved anytime soon.
Edit :
The Docusign support actually took action on this (the github's issue from above is now closed), it might be interesting to test again with the latest version of Docusign SDK NodeJS (or the one you were using previously).

Python call rest api to get data from url

I've created a Bash script to get the data from the url using rest API from a appliance using username, password and saving the Session ID into a Variable and then using the session ID to get the data into csv format which is working fine.
I want to change the bash code into python3 code as i'm parsing it using pandas.
Bash Code:
#!/bin/bash
sessionID=$(curl -k -H "accept: application/json" -H "content-type: application/json" -H "x-api-version: 120" -d '{"userName":"administrator","password":"adminpass"}' -X POST https://hpe.sysnergy.com/rest/login-sessions | jq -r ".sessionID")
curl -k -H 'accept: application/json' \
-H 'content-type: text/csv' \
-H 'x-api-version: 2' \
-H "auth: $sessionID" \
-X GET https://hpe.sysnergy.com/rest/resource-alerts
Python Version of tries code:
#!/usr/bin/python3
import requests
import json
url = "https://hpe.sysnergy.com/rest/login-sessions"
data = {'username': 'administrator', 'password': 'adminpass'}
headers = {'Content-type': 'text/csv', 'Accept': 'application/json', 'x-api-version': 2}
r = requests.post(url, data=json.dumps(data), headers=headers)
print(r)
I am getting below error:
Error:
requests.exceptions.InvalidHeader: Value for header {x-api-version: 2} must be of type str or bytes, not <class 'int'>
if i convert int to str as '2' then it gives another ssl error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='hpe.synerg.com', port=443): Max retries exceeded with url: /rest/login-sessions (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))
EDIT:
I have tried little different approach to get the same code format as bash in python but now it returns new error with new response code.
import os
import requests
sessionID = os.getenv('sessionID')
headers = {
'accept': 'application/json',
'content-type': 'text/csv',
'x-api-version': '2',
'auth': f"{sessionID}",
}
data = '{"userName":"administrator","password":"adminpassword"}'
response = requests.post('https://hpe.synergy.com/rest/login-sessions', headers=headers, data=data, verify=False)
print(response)
Error:
/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'hpe.synergy.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
<Response [415]>
Please help or suggest the way to achieve same function in the python.
You first need to make a POST request to get the sessionID, then you need to make a GET request. Also note the headers are slightly different for the 2 requests. Something like this should work:
import requests
session = requests.Session()
url = "https://hpe.sysnergy.com/rest/login-sessions"
credentials = {"userName": "administrator", "password": "adminpass"}
headers = {"accept": "application/json",
"content-type": "application/json",
"x-api-version": "120",
}
response = session.post(url, headers=headers, json=credentials, verify=False)
session_id = response.json()["sessionID"]
url = "https://hpe.sysnergy.com/rest/resource-alerts"
headers = {"accept": "application/json",
"content-type": "text/csv",
"x-api-version": "2",
"auth": session_id,
}
response = session.get(url, headers=headers, verify=False)
print(response)
#print(response.content) # returns bytes
#print(response.text) # returns string

How do I redirect all of the cURL command with POST to output towards stdout

Apologies if this has been asked before but I've trawled through a lot of similar questions but wasn't able to figure out so here goes:
I have a cURL command that does a HTTP POST.
How do I make the output to be redirected to standard output.
The command I am using inside a docker-container is the following:
curl -v -X POST "http://username:pass123#data-service:8081/api" -H "Content-Type: application/json" -d #postBody
This should give an output something like:
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 172.18.0.7...
* TCP_NODELAY set
* Connected to data-service (172.18.0.7) port 8081 (#0)
* Server auth using Basic with user 'username'
> POST /v1/sources HTTP/1.1
> Host: data-service:8081
> Authorization: Basic Z38JEsJ65JI9128hhtJlZW21XQ==
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 192
>
* upload completely sent off: 192 out of 192 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Server: Jetty(8.1.8.v20121106)
<
{
"action" : "GO"
* Curl_http_done: called premature == 0
* Connection #0 to host data-service left intact
}
How do I redirect all of this to stdout including the returned body.
I have tried the following but it doesn't redirect everything:
curl -vs POST "http://username:pass123#data-service:8081/api" -H "Content-Type: application/json" -d #postBody 2> dev/console
Any ideas?!
Thank you.

Why is PHP cURL on Linux changing the Content-Type request header?

It seems PHP's built-in cURL module changes the header fields before sending them.
I developed a small class to communicate with an encoder device trough HTTP requests, using cURL to do the task. The code works fine under Windows, however, when I run it under Debian, the device responds with an HTTP 406 error.
The error code indicates that the server cannot respond in the requested format. (More info)
This is strange, since the response type is determined by the extension of the URL (and the possible modes are xml and json), and I didn't set explicitly the Accept parameter in the header.
Using the CURLOPT_VERBOSE parameter, it dumps the following data:
* Hostname was NOT found in DNS cache
* Trying 172.19.0.9...
* Connected to 172.19.0.9 (172.19.0.9) port 1080 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* subject: C=US; ST=Illinois; L=Lake Forest; O=Haivision Network Video, Inc.; OU=PRODUCT DEVELOPMENT; CN=localhost.localdomai n; emailAddress=support#haivision.com
* start date: 2016-01-22 14:40:48 GMT
* expire date: 2026-01-19 14:40:48 GMT
* issuer: C=US; ST=Illinois; L=Lake Forest; O=Haivision Network Video, Inc.; OU=PRODUCT DEVELOPMENT; CN=localhost.localdomain ; emailAddress=support#haivision.com
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> POST /ecs/auth.xml HTTP/1.1
Host: 172.19.0.9:1080
Accept: */*
Content-Length: 86
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 86 out of 86 bytes
< HTTP/1.1 406 Not Acceptable
* Server nginx is not blacklisted
< Server: nginx
< Date: Fri, 01 Apr 2016 08:45:30 GMT
< Content-Type: application/xml
< Content-Length: 135
< Connection: keep-alive
<
* Connection #0 to host 172.19.0.9 left intact
It looks like the Content-Type: application/xml changed to application/x-www-form-urlencoded, and I think this is the main reason why the request fails so miserably.
The array being passed to the curl_setopts() function looks like this:
array(11) {
[19913]=>
bool(true)
[64]=>
bool(false)
[52]=>
bool(false)
[68]=>
int(10)
[10023]=>
array(5) {
["Authorization"]=>
string(10) "Basic ==Og"
["Cache-Control"]=>
string(8) "no-cache"
["Content-Type"]=>
string(15) "application/xml"
["Connection"]=>
string(10) "keep-alive"
["Content-Length"]=>
int(86)
}
[20079]=>
array(2) {
[0]=>
object(Pest)#43 (6) {
["curl_opts"]=>
array(9) {
[19913]=>
bool(true)
[64]=>
bool(false)
[52]=>
bool(false)
[68]=>
int(10)
[10023]=>
array(0) {
}
[20079]=>
*RECURSION*
[81]=>
int(0)
[84]=>
int(2)
[41]=>
bool(true)
}
["base_url"]=>
string(23) "https://172.19.0.9:1080"
["last_response"]=>
NULL
["last_request"]=>
NULL
["last_headers"]=>
NULL
["throw_exceptions"]=>
bool(true)
}
[1]=>
string(13) "handle_header"
}
[81]=>
int(0)
[84]=>
int(2)
[41]=>
bool(true)
[10036]=>
string(4) "POST"
[10015]=>
string(86) "<?xml version="1.0" encoding="UTF-8"?>
<user username="#########" password="########"/>
"
As you can see, there's no Accept tag, and the content type is set to application/xml.
So here's my question: why does curl changing the request's header? If the roots of the problem are something else, then what's the reason it works on Win10 and not on Debian Jessie?
Update (16. 04. 04.):
Fun thing, that the same version of cURL library doesn't work in PHP, but it does in cli:
curl -X POST -H "Authorization: Basic aGFpYWRtaW46bWFuYWdlcg==" -H "Content-Type: application/xml" -H "Cache-Control: no-cache" -H "Postman-Token: 760f1aac-619f-4b64-ec06-0146554fcecf" -d '<?xml version="1.0"?><user username="########" password="#######" />' "https://172.19.0.9:1080/ecs/auth.xml"
<?xml version="1.0"?>
<sessionid value="fd7b8fd0-ac5e-4f72-a01c-142082de24f1"/>
CURL version on the linux box is 7.26.0 (x86_64-pc-linux-gnu) libcurl/7.26.0.
Thanks in advance, and sorry for the wall of text.

How to query jayson (nodejs) with curl

I am trying to figured out how to query using CURL on the "jayson" npm package for nodejs available from here https://github.com/tedeh/jayson. I am using the test program below on the server side. Node is running properly and respond to Curl with an error output. no way to find what should be passed in Curl or what should be changed on the node program:
var jayson = require(__dirname + '/../..');
var server = jayson.server({
echo: function(msg, callback) {
if(msg != null)
callback(null, msg);
},
add: function(a, b, callback) {
if( (a!= null) && (b!= null) )
callback(null, a + b);
}
});
server.http().listen(90);
this is the CURL command:
$ curl -v -i -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id":"curltest", "method":"echo", "params": ["hello"] }' http://localhost:90
and this is the nodejs answer received by Curl if the curl command is not compliant to jayson:
* About to connect() to localhost port 90 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 90 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.28.1
> Host: localhost:90
> Accept: */*
> Content-Type: application/json
> Content-Length: 16
>
* upload completely sent off: 16 out of 16 bytes
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 79
Content-Length: 79
< Content-Type: application/json
Content-Type: application/json
< Date: Tue, 25 Mar 2014 19:26:21 GMT
Date: Tue, 25 Mar 2014 19:26:21 GMT
< Connection: keep-alive
Connection: keep-alive
<
* Connection #0 to host localhost left intact
{"jsonrpc":"2.0","id":null,"error":{"code":-32600,"message":"Invalid request"}}* Closing connection #0
Many thanks for the help,
RĂ©mi
You are running the curl from windows. That's why the single quote around the parameters doesn't work for you. Change them into double quotes:
curl -i -X POST -H "Content-Type: application/json" -d "{\"echo\": \"Name\"}" http://localhost:90/?
Also, try to run the command with -v appending at the end. It will show you the debug msg of curl. Show us the msg if it doesn't work for you.
I found the trick and I have corrected the error above. We should be jsonrpc 2.0 compliant as Jayson is considering jsonrpc2 only.
This is the correct curl command line for the "echo" script:
$ curl -v -i -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id":"curltest", "method":"echo", "params": ["hello"] }' localhost:90
This is the correct curl command line for the "add" script:
$ curl -v -i -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id":"curltest", "method":"add", "params": [1, 2] }' http://localhost:90

Resources