I've been using reqwest for a few months now; solid features, reuse of types defined in http (and the like) and documentation.
I have the following http post request that works in Postman and curl that for reasons I can't figure out, generates a http::StatusCode "400 Bad Request".
Staring at and comparing the debug output of the request (below) over the past couple of days hasn't helped (ok, and a solid amount of reading the docs and playing with code too :)); thus the post.
The request spec that works with other clients
POST /2/files/list_folder HTTP/1.1
Host: api.dropboxapi.com
Authorization: Bearer <secret>
Content-Type: application/json
Content-Length: 65
{
"path": "",
"include_non_downloadable_files": false
}
The failing reqwest request code
let client = reqwest::Client::new();
let request = client
.request(
http::Method::from_str(method).unwrap(),
format!(
"{host}{endpoint}{query}",
host = &drive_server,
endpoint = &endpoint,
query = &query_ls
),
)
.header(AUTHORIZATION, format!("Bearer {}", access_token.secret()))
.header(CONTENT_TYPE, "application/json") // redundant when setting body with .json
.header(ACCEPT, "application/json")
.version(Version::HTTP_11)
.json(&req_body);
... and the code that builds the req_body:
// dropbox WIP
let json = r#"
{
"path": "",
"include_non_downloadable_files": false
}
"#;
#[derive(Deserialize, Serialize)]
struct ReqBody {
path: String,
include_non_downloadable_files: bool,
}
let req_body = serde_json::from_str::<ReqBody>(json)
.map_err(|err| AuthError::JsonParsingError(err.into()))?;
Debug output of the request
RequestBuilder {
method: POST,
url: Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"api.dropboxapi.com",
),
),
port: None,
path: "/2/files/list_folder",
query: None,
fragment: None,
},
headers: {
"authorization": "Bearer <secret>",
"content-type": "application/json",
"accept": "application/json",
},
}
Aside but to know what to expect from the diagnostic: Is the output expected to exclude the request body when present? I have not found a way to echo the request body once "theoretically" set :))
A bit more context
Finally, a bit more background, the explicit call to the version is a failed attempt on my part to override google drive forcing a switch to http 2 when it sees a request body (a separate use of this client). I'll have to figure out another way that includes some sort of conditional call to .json(<body>) without violating ownership issues, when services like dropbox require the body but not google. When I disable the .json on the reqwest RequestBuilder for a service such as google, the request works as expected.
Plainly put, the question
How do I build the http post request described at the top of this post using reqwest?
Related
I have been bashing my head against the wall for the last 2 days with the following problem.
This is the scenario: When I make a GET request by browsing to a particular website, this website sends a cookie called PHPSESSION="xyz" it then prompts the user to enter a password and subsequently makes a post request to the same URL sending this particular cookie and a hidden form element alongside for verification and upon success sends a pdf.
I can successfully replicate this in Postman.
I make a get request - it sets the cookie - I have password filled into my form-data responds body and manually add the secret string that is added to the form for verification -> send... and I get the pdf - so far so good.
However, I would like to automate this process so that I don't have to painstakingly extract the value of the hidden form by hand but use node.js to make these requests so I wrote the following code:
// making the get request to the URL above
// extract the cookie PHPSESSION value
const sessionString = String(response.headers.get('set-cookie')).substring(10,36)
// parse the body
const htmlBody = await response.text()
let doc = new DOMParser().parseFromString(htmlBody)
// extract the verification token from the form
const formToken = await doc.getElementById('verification__token').getAttribute('value')
let formData = new FormData();
formData.append('verification[char_1]',0)
formData.append('verification[char_2]',6)
formData.append('verification[char_3]',4)
formData.append('verification[char_4]',5)
formData.append('verification[char_5]',8)
formData.append('verification[char_6]',1)
formData.append('verification[char_7]',7)
formData.append('verification[char_8]',6)
formData.append('verification[_token]',formToken)
const obj = {
headers:{
"Cookie" : `PHPSESSID=${sessionString};`,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "PostmanRuntime/7.29.2",
"Accept-Encoding": "gzip, deflate, br",
"credentials": "include"
},
method: "POST",
body: formData
}
const postResponse = await fetch("https://url...",obj)
const r = await postResponse.text()
Unfortunately, the post requests fails in node.js - the website is simply redirecting me to back to the form in which I have to type in the password.
I am suspecting it has something to do with the headers / cookie but I simply don't know.
Does anyone spots an obvious mistake?
Thank you
Solved... after sacrificing the entire weekend to this lovely task.
If anyone comes across a similar problem here is the solution - or at lest what helped me.
https://reqbin.com/curl
https://curlconverter.com
So basically make your request work with curl and then port it.
In my case that looked like this:
const x = await fetch('https://yourURL', {
method: 'POST',
headers: {
'Cookie': 'PHPSESSID=lfjdd2uba1bmecr064rt7chvu3; Path=/; Secure; HttpOnly;',
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'verification[_token]=5d5e4d8783daf952d5.UZ661yMyOUtJSQeG1Td7cUtxWqnI2Oaot-xMQevly4o.acH9hXR9SRkwGm30kE9WIggDNpqdl6Ln2rQnOIG9pcEp1tOiYnNLJggZcA&verification[char_1]=0&verification[char_2]=6&verification[char_3]=4&verification[char_4]=5&verification[char_5]=8&verification[char_6]=1&verification[char_7]=7&verification[char_8]=6'
});
I want to send multiple files to Github repository via nodejs. Tried several approaches and end up using node-rest-client module. Tried below code send a sample file to repository called 'metadata'. But after post I am getting error message "Request forbidden by administrative rules. Please make sure your request has a User-Agent header"...please let me know if anyone faced this error before and get rid of it.
convertval = "somedata";
var dataObj = {
"message": "my commit message",
"committer": {
"name": "Scott Chacon",
"email": "ravindra.devagiri#gmail.com"
},
"content": "bXkgbmV3IGZpbGUgY29udGVudHM="
}
debugger;
var Client = require('node-rest-client').Client;
var client = new Client()
var args = {
data: dataObj,
headers: { 'Content-Type': 'application/json' },
};
client.post("https://api.github.com/repos/metadata/contents", args, function (data, response) {
console.log("file send: True : " + data);
});
According to the REST API:
All API requests MUST include a valid User-Agent header. Requests with
no User-Agent header will be rejected.
First of all, you need to define 'User-Agent' with value 'request' in your request header. Refer to this link.
Second, endpoint you are trying to call might require authentication. Generate a personal token from here, add that token in your request header, 'Authorization': 'token '.
If you're using Git extensively in your code, I suggest you to use this - Nodegit.
Edit:
I don't think sending multiple files in a single request is possible in 'Contents' endpoints group (link).
You can checkout Git Data API (as discussed here).
I've been playing around with some web scraping but I've run into an issue I can't figure out; Using a nodejs server (on my local computer) I cannot get passed a permission error barring me from accessing the data. What is confusing to me most is that using the chrome extension "Postman" I don't run into the permission errors, but using the code generated by postman, I do (as well as fiddling with variations of my own scratch code).
Do I have to be using a live server? Do I need to include some extra items in the headers that aren't being put there by Postman? Is there some layer of security around the API that for some reason Postman has access do that a local machine doesnt?
Any light that can be shed would be of use. Note that there is no public documentation of the SmithsFoodAndDrug API (that I can find), so there aren't necessarily APIKeys that are going to be used. But the fact that Postman can access the information makes me think I should be able to on a node server without any special authentication set up.
In Summary:
I'm looking at SmithsFoodAndDrug product information, and found the API where they are grabbing information from.
I figured out the headers needed in order to get local price information on products (on top of the json body format for the POST request)
Using postman I can generate the POST request and retrieve the desired API results
Using nodejs (and the code generated by postman to replicate the request) with both 'request' module and standard 'http' module request module I receive permission errors from the server.
Details: (assume gathering data on honeycrisp apples (0000000003283) with division-id of 706 and store-id of 00144)
http://www.smithsfoodanddrug.com/products/api/products/details
Headers are 'division-id' and 'store-id'. Body is in format of {"upcs":["XXX"],"filterBadProducts":false} where XXX is the specific product code.
Here are the Request Headers in postman. Here are the Request Body settings in postman. The following is a portion of the json response (which is what I want).
{"products": [
{
"brandName": null,
"clickListItem": true,
"countryOfOrigin": "Check store for country of origin details",
"customerFacingSize": "price $2.49/lb",
...
"calculatedPromoPrice": "2.49",
"calculatedRegularPrice": "2.99",
"calculatedReferencePrice": null,
"displayTemplate": "YellowTag",
"division": "706",
"minimumAdvertisedPrice": null,
"orderBy": "Unit",
"regularNFor": "1",
"referenceNFor": "1",
"referencePrice": null,
"store": "00144",
"endDate": "2018-09-19T00:00:00",
"priceNormal": "2.55",
"priceSale": "2.12",
"promoDescription": "About $2.12 for each",
"promoType": null,
...
"upc": "0000000003283",
...
}
],
"coupons": {},
"departments": [],
"priceHasError": false,
"totalCount": 1 }
When using the code given by postman to replicate the request, I get the error saying 'You don't have permission to access "http://www.smithsfoodanddrug.com/products/api/products/details" on this server.
Reference #18.1f3de93f.1536955806.1989a2b1.' .
// Code given by postman
var request = require("request");
var options = { method: 'POST',
url: 'http://www.smithsfoodanddrug.com/products/api/products/details',
headers:
{ 'postman-token': 'ad9638c1-1ea5-1afc-925e-fe753b342f91',
'cache-control': 'no-cache',
'store-id': '00144',
'division-id': '706',
'content-type': 'application/json' },
body: { upcs: [ '0000000003283' ], filterBadProducts: false },
json: true };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
change headers
headers:
{
'store-id': '00144',
'division-id': '706'
//'content-type': 'application/json'
}
I am trying to import a json array into arangodb using the http api from various node modules like needle, http, request. Each time i get the following error or similar:
{ error: true,
errorMessage: 'expecting a JSON array in the request',
code: 400,
errorNum: 400 }
The code is below (similar for most modules listed above with minor variations). Various scenarios (single document import, etc.) all seem to point to the post body not being correctly recognized for some reason.
var needle = require('needle');
var data = [{
"lastname": "ln",
"firstname": "fn",
},
{
"lastname": "ln2",
"firstname": "fn2"
}];
var options = { 'Content-Type': 'application/json; charset=utf-8' };
needle.request('POST', 'http://ip:8529/_db/mydb/_api/import?type=array&collection=accounts&createCollection=false', data, options, function(err, resp) {
console.log(resp.body);
});
While i am able to upload the documents using curl and browser dev tools, I have not been able to get it working in node.js. What am i doing wrong? This is driving me crazy. Any help would be appreciated. Thank you very much.
You can use ngrep (or wireshark) to quickly find out whats wrong:
ngrep -Wbyline port 8529 -d lo
T 127.0.0.1:53440 -> 127.0.0.1:8529 [AP]
POST /_db/mydb/_api/import?type=array&collection=accounts& createCollection=true HTTP/1.1.
Accept: */*.
Connection: close.
User-Agent: Needle/0.9.2 (Node.js v1.8.1; linux x64).
Content-Type: application/x-www-form-urlencoded.
Content-Length: 51.
Host: 127.0.0.1:8529.
.
##
T 127.0.0.1:53440 -> 127.0.0.1:8529 [AP]
lastname=ln&firstname=fn&lastname=ln2&firstname=fn2
The body to be sent to ArangoDB has to be json (as you try to achieve by setting the content type).
Making needle to actualy post json works this way: (see https://github.com/tomas/needle#request-options )
var options = {
Content-Type: 'application/json; charset=utf-8',
json: true
};
which produces the proper reply:
{ error: false,
created: 2,
errors: 0,
empty: 0,
updated: 0,
ignored: 0 }
I am using nodejs and the REST API to interact with bigquery. I am using the google-oauth-jwt module for JWT signing.
I granted a service account write permission. So far I can list projects, list datasets, create a table and delete a table. But when it comes to upload a file via multipart POST, I ran into two problems:
gzipped json file doesn't work, I get an error saying "end boundary missing"
when I use uncompressed json file, I get a 401 unauthorized error
I don't think this is related to my machine's time being out of sync since other REST api calls worked as expected.
var url = 'https://www.googleapis.com/upload/bigquery/v2/projects/' + projectId + '/jobs';
var request = googleOauthJWT.requestWithJWT();
var jobResource = {
jobReference: {
projectId: projectId,
jobId: jobId
},
configuration: {
load: {
sourceFormat: 'NEWLINE_DELIMITED_JSON',
destinationTable: {
projectId: projectId,
datasetId: datasetId,
tableId: tableId
},
createDisposition: '',
writeDisposition: ''
}
}
};
request(
{
url: url,
method: 'POST',
jwt: jwtParams,
headers: {
'Content-Type': 'multipart/related'
},
qs: {
uploadType: 'multipart'
},
multipart: [
{
'Content-Type':'application/json; charset=UTF-8',
body: JSON.stringify(jobResource)
},
{
'Content-Type':'application/octet-stream',
body: fileBuffer.toString()
}
]
},
function(err, response, body) {
console.log(JSON.parse(body).selfLink);
}
);
Can anyone shine some light on this?
P.S. the documentation on bigquery REST api is not up to date on many things, wish the google guys can keep it updated
Update 1:
Here is the full HTTP request:
POST /upload/bigquery/v2/projects/239525534299/jobs?uploadType=multipart HTTP/1.1
content-type: multipart/related; boundary=71e00bd1-1c17-4892-8784-2facc6998699
authorization: Bearer ya29.AHES6ZRYyfSUpQz7xt-xwEgUfelmCvwi0RL3ztHDwC4vnBI
host: www.googleapis.com
content-length: 876
Connection: keep-alive
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/json
{"jobReference":{"projectId":"239525534299","jobId":"test-upload-2013-08-07_2300"},"configuration":{"load":{"sourceFormat":"NEWLINE_DELIMITED_JSON","destinationTable":{"projectId":"239525534299","datasetId":"performance","tableId":"test_table"},"createDisposition":"CREATE_NEVER","writeDisposition":"WRITE_APPEND"}}}
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/octet-stream
{"practiceId":2,"fanCount":5,"mvp":"Hello"}
{"practiceId":3,"fanCount":33,"mvp":"Hello"}
{"practiceId":4,"fanCount":71,"mvp":"Hello"}
{"practiceId":5,"fanCount":93,"mvp":"Hello"}
{"practiceId":6,"fanCount":92,"mvp":"Hello"}
{"practiceId":7,"fanCount":74,"mvp":"Hello"}
{"practiceId":8,"fanCount":100,"mvp":"Hello"}
{"practiceId":9,"fanCount":27,"mvp":"Hello"}
--71e00bd1-1c17-4892-8784-2facc6998699--
You are most likely sending duplicate content-type headers to the Google API.
I don't have the capability to effortlessly make a request to Google BigQuery to test, but I'd start with removing the headers property of your options object to request().
Remove this:
headers: {
'Content-Type': 'multipart/related'
},
The Node.js request module automatically detects that you have passed in a multipart array, and it adds the appropriate content-type header. If you provide your own content-type header, you most likely end up with a "duplicate" one, which does not contain the multipart boundary.
If you modify your code slightly to print out the actual headers sent:
var req = request({...}, function(..) {...});
console.log(req.headers);
You should see something like this for your original code above (I'm using the Node REPL):
> req.headers
{ 'Content-Type': 'multipart/related',
'content-type': 'multipart/related; boundary=af5ed508-5655-48e4-b43c-ae5be91b5ae9',
'content-length': 271 }
And the following if you remove the explicit headers option:
> req.headers
{ 'content-type': 'multipart/related; boundary=49d2371f-1baf-4526-b140-0d4d3f80bb75',
'content-length': 271 }
Some servers don't deal well with multiple headers having the same name. Hopefully this solves the end boundary missing error from the API!
I figured this out myself. This is one of those silly mistakes that would have you stuck for the whole day and at the end when you found the solution you would really knock on your own head.
I got the 401 by typing the selfLink URL in the browser. Of course it's not authorized.