404 Downloading OneDrive content from Microsoft Graph - sharepoint

I am trying to download the content of a onedrive item via the microsoft api. However, no matter the method I use, I get 404 responses. Here is a reproduction of the problem in python/requests
import requests
import json
root_url = "https://graph.microsoft.com"
base_path = "/v1.0/<tenant_id>/users/<principal_name>/drive/"
token = "ALONGTOKEN"
headers = {"Authorization": "Bearer %s" % token}
r = requests.get(root_url + base_path + "/root/children", headers=headers)
listing = json.loads(r.text)
target = listing["value"][0]
print("Target node:")
print(json.dumps(target))
print("Target node id:")
print(target["id"])
r = requests.get(root_url + base_path + "items/" + target["id"], headers=headers)
print("Target metadata:")
print(r.text)
resp = json.loads(r.text)
download_url = resp["#microsoft.graph.downloadUrl"]
print("Target download url:")
print(download_url)
r = requests.get(download_url, headers=headers)
print("Download response code:")
print(r.status_code)
print("Download response headers:")
print(r.headers)
print("Download response cookies:")
print(r.cookies)
print("Download response redirect history:")
print(r.history)
outputs the following:
Target node:
{"parentReference": {"driveId": "drive_id", "path": "/drive/root:", "id": "parent_id"}, "cTag": "\"c:{tag},1\"", "lastModifiedDateTime": "2016-08-24T17:32:45Z", "name": "birds.png", "createdDateTime": "2016-08-24T17:32:45Z", "image": {}, "webUrl": "https://org-my.sharepoint.com/personal/principal_name/Documents/birds.png", "lastModifiedBy": {"user": {"displayName": "User Name", "id": "user_id"}}, "eTag": "\"{etag},1\"", "createdBy": {"user": {"displayName": "User Name", "id": "user_id"}}, "#microsoft.graph.downloadUrl": "https://org-my.sharepoint.com/personal/principal_name/_layouts/15/download.aspx?guestaccesstoken=access_token&docid=did&expiration=2016-09-01T17%3a12%3a14.000Z&userid=uid&authurl=True&NeverAuth=True", "file": {"hashes": {}}, "id": "01L4SXJGJ2LR2PGPKJMVGZPHIADCAYJEFE", "size": 34038}
Target node id:
01L4SXJGJ2LR2PGPKJMVGZPHIADCAYJEFE
Target metadata:
{"#odata.context":"https://graph.microsoft.com/v1.0/$metadata#users('principal_name')/drive/items/$entity","#microsoft.graph.downloadUrl":"https://org-my.sharepoint.com/personal/principal_name/_layouts/15/download.aspx?guestaccesstoken=accesstoken&docid=docid&expiration=2016-09-01T17%3a12%3a15.000Z&userid=uid&authurl=True&NeverAuth=True","createdBy":{"user":{"id":"user_id","displayName":"User Name"}},"createdDateTime":"2016-08-24T17:32:45Z","eTag":"\"{etag},1\"","id":"01L4SXJGJ2LR2PGPKJMVGZPHIADCAYJEFE","lastModifiedBy":{"user":{"id":"user_id","displayName":"User Name"}},"lastModifiedDateTime":"2016-08-24T17:32:45Z","name":"birds.png","webUrl":"https://org-my.sharepoint.com/personal/principal_name/Documents/birds.png","cTag":"\"c:{ctag},1\"","file":{"hashes":{}},"image":{},"parentReference":{"driveId":"drive_id","id":"parent_id","path":"/drive/root:"},"size":34038}
Target download url:
https://org-my.sharepoint.com/personal/principal_name/_layouts/15/download.aspx?guestaccesstoken=accesstoken&docid=docid&expiration=2016-09-01T17%3a12%3a15.000Z&userid=uid&authurl=True&NeverAuth=True
Download response code:
404
Download response headers:
{'Content-Length': '13702', 'SPIisLatency': '4', 'X-Content-Type-Options': 'nosniff', 'X-AspNet-Version': '4.0.30319', 'request-id': '288b9f9d-c04a-2000-133b-ebab2f6f332b', 'Strict-Transport-Security': 'max-age=31536000', 'MicrosoftSharePointTeamServices': '16.0.0.5625', 'X-Powered-By': 'ASP.NET', 'SPRequestGuid': '288b9f9d-c04a-2000-133b-ebab2f6f332b', 'Server': 'Microsoft-IIS/8.5', 'X-MS-InvokeApp': '1; RequireReadOnly', 'X-SharePointHealthScore': '0', 'SPRequestDuration': '297', 'SharePointError': '0', 'Cache-Control': 'private', 'Date': 'Thu, 01 Sep 2016 16:12:14 GMT', 'P3P': 'CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"', 'Content-Type': 'text/html; charset=utf-8'}
Download response cookies:
<RequestsCookieJar[]>
Download response redirect history:
[]
Which is to say that immediately following the link results in a 404, though it is supposed to download the file bytes. I have reproduced this in java, python, bash/curl, and in the browser. Can anybody help point out what I am doing wrong, or is this a problem with the microsoft graph api?
EDIT:
I can also reproduce the same 404 using the /drive/items/{item-id}/content endpoint described here. The request to this endpoint results in a 302 redirect (as described in the documentation), which, when followed, results in the same 404 behavior as described above.
EDIT2:
Here are all the request-ids I could find in the response headers that looked useful for debugging from Microsoft's side.
For the 200 request on the item object: 'request-id': 'adfa3492-4825-439d-8e59-022f32e78244', 'client-request-id': 'adfa3492-4825-439d-8e59-022f32e78244'
For the 404 request on the download url: 'request-id': '33e09e9d-b0c2-2000-133c-304585c15000', 'SPRequestGuid': '33e09e9d-b0c2-2000-133c-304585c15000',
Additionally, the actual HTML returned from the 404 includes Correlation ID: a8e09e9d-a0bb-2000-133b-ef6fc8ac7015

File download is currently only supported with delegated permissions (e.g. File.Read scope) as documented here. Your request was made with application permissions Files.Read.All and Files.ReadWrite.All, which we're gradually adding support for, but they're not yet fully functional and are not listed here.

Can you check that item_id is in fact the ID of an item? If you are working off the collection returned by GET /v1.0/users//drive/items/, the collection will return an array of folder and item metadata. If you try your request against a folder, you will get the 404 as you've described. That's the only way that I can repro your issue. If that is not the issue, please provide the request/response trace so that we can see the error details.

Related

Error 500 : Anyone can guess why i am getting this if sending this request from robot framework python and its working using postman

Code:
Make request for getBookingOptions with valid user
#[Arguments] ${VALID_USER} ${VALID_PASSWORD} ${VALID_EMAIL} ${MEETING_DATE} ${MEETING_TIME} ${MEETING_DURATION} ${TIMEZONE_OFFSET} ${ROOMS} ${REQUIRED_INVITEES}
${body} = Create Dictionary userId=45646546456 password=fgdfgdfg email=mohammednasir.ali#istrbc.com meetingDate=2022-08-30 meetingTime=2022-08-30T16:00:00.000 meetingDuration=30 timeZoneOffset=-14400
${header} = Create Dictionary Content-Type=application/json
${body} Evaluate json.dumps(${body}) json
${response} = Post request CEA /getBookingOptions json=${body} headers=${header}
log ${RESPONSE_DATA}
set test variable ${response} ${response}
set global variable ${RESPONSE_DATA} ${response.json()}
Robot framework generated:
POST Request using
: uri=/getBookingOptions, params=None, files=None, allow_redirects=True, timeout=None
headers={'User-Agent': 'python-requests/2.23.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json'}
data=None
json={"userId": "64554646", "password": "Disdcovdgery7", "email": "nasir#google.com", "meetingDate": "2022-08-30", "meetingTime": "2022-08-30T16:00:00.000", "meetingDuration": "30", "timeZoneOffset": "-14400"}
Output:
{'message': 'invalid request', 'status': 'failure'}
${response} = <Response [500]>
but same kind of json is working fine using postman. I am unable to find the reason. Can anyone guess please?

DocuSign, Node SDK, JWT auth, Envelope fetch, "The custom error module does not recognize this error."

[FINAL UPDATE]
Fixed per this thread: https://github.com/docusign/docusign-esign-node-client/issues/295
Apparently related to security additions made to DocuSign's API.
I haven't encountered this issue before today. I gave a demo to a client just last week and the same code was working fine. Today I get the following:
{
"status": 444,
"response": {
"req": {
"method": "GET",
"url": "https://demo.docusign.net/restapi...",
"data": {},
"headers": {
"user-agent": "node-superagent/3.8.2",
"x-docusign-sdk": "Node",
"authorization": "Bearer ABC...",
"content-type": "application/json",
"accept": "application/json"
}
},
"header": {
"content-type": "text/html",
"x-docusign-tracetoken": "2eeb8caa-8865-4898-bef9-d3611bfaa3f7",
"x-docusign-node": "DA2DFE5",
"date": "Fri, 17 Jun 2022 01:02:02 GMT",
"content-length": "54",
"connection": "close",
"strict-transport-security": "max-age=31536000; includeSubDomains"
},
"status": 444,
"text": "The custom error module does not recognize this error."
}
}
Through the node SDK I fetch a token using requestJWTUserToken, and set the apiclient's auth header, then I new-up an EnvelopesAPI instance. When I make a call to getEnvelope() or listStatusChanges(), then I get the error above. None of this code has changed in months, and I'm using the same integration key, account, private key, everything. I've demo'd it all a few times now - no issues.
An interesting observation: the error above gives me a URL and token. The token is valid, and if I make a request to the URL (the envelopes endpoint) via Postman using said token, then the request succeeds. So my calls through the SDK seem to be failing for some reason.
I can't seem to crack this one, and now I can't get around it given a couple demo systems that worked just last week.
I'm using the docusign-esign 5.17 module - upgraded from 5.15 in an attempt to fix the issue. No luck.
Where is this error coming from?
[Update 1]
I'm running my node app that is making requests through the DocuSign Node SDK against a proxy so I can see what the failing request actually look like:
They fail the same way.
HTTP/1.1 444
Content-Type: text/html
X-DocuSign-TraceToken: 338534c6-c8c3-4b01-9b66-35d697cd0053
X-DocuSign-Node: DA1DFE4
Date: Fri, 17 Jun 2022 03:55:07 GMT
Content-Length: 54
Vary: Accept-Encoding
Connection: close
Strict-Transport-Security: max-age=31536000; includeSubDomains
The custom error module does not recognize this error.
I'm using Proxyman to catch the request, and like Chrome or Firefox it will let you copy a request as a cURL command. If I copy the failing request as cURL, then run in at the terminal, it succeeds.
[MacBookPro0020]~/source/docusign/jwt-smoke-test:0 (master)
$ curl 'https://demo.docusign.net/restapi/v2.1/accounts/a0a4c81f-.../envelopes?envelope_ids=e750526f-...&envelope_ids=a38b794b...&envelope_ids=a5d8c586-...' \
-H 'Host: demo.docusign.net' \
-H 'User-Agent: node-superagent/3.8.2' \
-H 'X-DocuSign-SDK: Node' \
-H 'Node-Ver: v14.18.3' \
-H 'Authorization: Bearer ABCD...' \
-H 'Accept: application/json' \
-H 'Connection: close' \
-H 'Content-Type: application/json' \
--proxy http://localhost:9090
{"resultSetSize":"1","startPosition":"0","endPosition":"0","totalSetSize":"1","nextUri":"","previousUri":"","envelopes":[{"status":"created","documentsUri":"/envelopes/d97565c8...purgeState":"unpurged","envelopeIdStamping":"true","autoNavigation":"true","isSignatureProviderEnvelope":"false","allowComments":"true","anySigner":null,"envelopeLocation":"current_site"}]}
I'm using a JWT auth token, so again, I'm getting a valid token. Calls through the SDK consistently fail, but cURL'ing and manually requesting through Postman both succeed.
I'm at a loss.
Additional details: I see this same issue on MacOS and Windows (i.e., node app hosting docusign-esign). I'm using auth code grant to send envelopes and query envelope statuses and that works fine. I've used JWT Grant without issue up until this week (just demo'd automated functionality last week and it worked.) I haven't made any code changes to my DocuSign functionality, nor have my colleagues, at least according to the repo's history.
I don't recall ever encountering the error above before. I'd love to know why cURL'ing the same request succeeds. I'd rather not ditch the SDK and roll my own requests, but it wouldn't be difficult.
[Update 2]
Here's a simple repro - it's a quick and dirty copy of the QuickStart demo project for the node SDK. I'm using only docusign-esign.
Exact. same. issue.
Again, I can take that token and drop it into cURL or postman and the request will succeed. There's not a lot of code here. The token is valid.
async function main() {
// Data used
// dsConfig.dsClientId
// dsConfig.impersonatedUserGuid
// dsConfig.privateKey
// dsConfig.dsOauthServer
let dsConfig = dsConfig_customer; // defined globally
const jwtLifeSec = 10 * 60, // requested lifetime for the JWT is 10 min
dsApi = new docusign.ApiClient();
dsApi.setOAuthBasePath(dsConfig.dsOauthServer.replace('https://', '')); // it should be domain only.
const results = await dsApi.requestJWTUserToken(dsConfig.dsClientId,
dsConfig.impersonatedUserGuid, 'signature impersonation', dsConfig.privateKey,
jwtLifeSec);
console.log( results.body.access_token );
const userInfo = await dsApi.getUserInfo(results.body.access_token);
dsApi.setBasePath(userInfo.accounts[0].baseUri + '/restapi');
dsApi.addDefaultHeader( 'Authorization', 'Bearer ' + results.body.access_token );
const envelopesAPI = new docusign.EnvelopesApi(dsApi);
const res = await envelopesAPI.getEnvelope( dsConfig.accountID, 'e1917111-2900-48e8-9054-799169379c8a', null );
console.log(res);
return {
accessToken: results.body.access_token,
tokenExpirationTimestamp: expiresAt,
userInfo,
account: userInfo.accounts[0]
};
}
main().then(result => console.log(result)).catch(err=>console.error(err));
...
header: {
'content-type': 'text/html',
'x-docusign-tracetoken': '685b6226-a0d3-4547-94c7-df0216d884a3',
'x-docusign-node': 'DA2DFE188',
date: 'Fri, 17 Jun 2022 05:20:12 GMT',
'content-length': '54',
vary: 'Accept-Encoding',
connection: 'close',
'strict-transport-security': 'max-age=31536000; includeSubDomains'
},
statusCode: 444,
status: 444,
statusType: 4,
info: false,
ok: false,
redirect: false,
clientError: true,
serverError: false,
error: Error: cannot GET /restapi/v2.1/accounts/49754554-ABCD-.../envelopes/e1917111-2900-48e8-9054-799169379c8a (444)
...
I finally manage to resolve the issue by downgrading my Docusign SDK NodeJS version from 5.15.0 to 5.7.0 in my package.json file.
An issue relative to this can be find there, I really hope this issue will be resolved anytime soon.
Edit :
The Docusign support actually took action on this (the github's issue from above is now closed), it might be interesting to test again with the latest version of Docusign SDK NodeJS (or the one you were using previously).

Robot Framework - Unsupported Media Type error

I have the following json file:
{
"company":[
{
"name":"My company",
"security":"WPA2-PSK"
}
],
"name":"one name"
}
I'm trying to send a POST based on this json file but it's getting an error.
My code looks like this:
Create new company
${headers} = Create Dictionary Accept=application/json Content-Type=application/json
${body} = Get file create.json
Create Session http_session ${url} disable_warnings=${True}
${response} = POST On Session http_session ${company} data=${body} headers=${headers}
But I get the following error:
POST Response:
headers={'content-type': 'application/json', 'vary': 'Accept, Origin, Cookie', 'allow': 'GET, POST, HEAD, OPTIONS', 'x-frame-options': 'DENY', 'content-length': '52', 'x-content-type-options': 'nosniff', 'referrer-policy': 'same-origin', 'set-cookie': '64bc588b56c7114f53411b945693e29ba=55323dcbd60f99b6aa8cfcb0f44f578e; path=/; HttpOnly; Secure; SameSite=None'}
body={"detail":"Unsupported media type \"\" in request."}
HTTPError: 415 Client Error: Unsupported Media Type for url: myUrl
So I did:
Create new company
${headers} = Create Dictionary Accept=application/json Content-Type=application/json
${json1} = Get file create.json
${json} = Evaluate json.dumps(${json1})
${body} = Evaluate json.loads('''${json}''') json
Create Session http_session ${url} disable_warnings=${True}
${response} = POST On Session http_session ${company} data=${body} headers=${headers}
so i got the same error.
What did I do wrong?
It was a problem in converting json. I did it like this:
${body} = Evaluate json.loads("""${json}""")
Now it worked.

How to change axios content from "text/html" to application/json type in Nestjs

I discovered that axios is returning a string instead of a valid json.
headers:Object {date: "Tue, 02 Jun 2020 08:44:06 GMT", server: "Apache", connection: "close", …}
connection:"close"
content-type:"text/html; charset=UTF-8"
date:"Tue, 02 Jun 2020 08:44:06 GMT"
server:"Apache"
transfer-encoding:"chunked"
how do I change the content-type to application/json in NestJs application?
I tried this but didnt not work
const meterInfo = await this.httpService.get(url, { headers: { "Content-Type": "application/json" } }).toPromise();
Here is the invalid json returned.
"{"status":"00","message":"OK","access_token":"2347682423567","customer":{"name":"John Doe","address":"Mr. John Doe 34 Tokai, leaflet. 7999.","util":"Demo Utility","minimumAmount":"13897"},"response_hash":"c43c9d74480f340f55156f6r5c56487v8w"}"
Instead of sending a Content-Type header, you should send an Accept header with the same MIME type. This tells the server what you are expecting to receive, and if Content Negotiation is set up properly, it will allow you to get a JSON back instead of that weird string.
this.httpService.get(
url,
{
headers: {
'Accept': 'application/json',
},
},
).toPromise();
If that doesn't work, you'll need to provide your own serializer to take the string from that wonky format to JSON, or get in touch with the server admins and see if they can provide you better documentation about how to consume their API.

How to make all binary files (images) download using requests() and open()?

When I try to download a image from one url, the code works, but when I try another URL, it dosen't work.
This Doesn't Work. It only creates the filename.
# This Doesn't Work.
import requests
url = 'https://ryanspressurewashing.com/wp-content/uploads/2017/06/metal-
roof-after-pressure-washing.jpg'
r = requests.get(url, stream=True)
with open('image3.jpg', 'wb') as my_file:
# Read by 4KB chunks
for byte_chunk in r.iter_content(chunk_size=4096):
my_file.write(byte_chunk)
# This Works?
import requests
url = 'http://www.webscrapingfordatascience.com/files/kitten.jpg'
r = requests.get(url, stream=True)
with open('image.jpg', 'wb') as my_file:
# Read by 4KB chunks
for byte_chunk in r.iter_content(chunk_size=4096):
my_file.write(byte_chunk)
Different portals may have different security systems to block script/bots.
When you open image3.jpg in text editor you will see
<head>
<title>Not Acceptable!</title>
</head>
<body>
<h1>Not Acceptable!</h1>
<p>An appropriate representation of the requested resource could not be found on this server.
This error was generated by Mod_Security.</p>
</body>
</html>
Some servers may need correct headerse, cookies, session-id, etc. to give access to data.
This portal needs correct header user-agent
import requests
url = 'https://ryanspressurewashing.com/wp-content/uploads/2017/06/metal-roof-after-pressure-washing.jpg'
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'
}
r = requests.get(url, stream=True, headers=headers)
with open('image3.jpg', 'wb') as my_file:
# Read by 4KB chunks
for byte_chunk in r.iter_content(chunk_size=4096):
my_file.write(byte_chunk)
requests as default uses user-agent: python-requests/2.21.0 so portals can easily recognize script and block it.
You can see this header using https://httpbin.org/get
import requests
r = requests.get('https://httpbin.org/get')
print(r.text)
Result:
{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.21.0"
},
"origin": "83.23.39.165, 83.23.39.165",
"url": "https://httpbin.org/get"
}
See more functions on httpbin.org

Resources