Arangodb import via http api using node.js - node.js

I am trying to import a json array into arangodb using the http api from various node modules like needle, http, request. Each time i get the following error or similar:
{ error: true,
errorMessage: 'expecting a JSON array in the request',
code: 400,
errorNum: 400 }
The code is below (similar for most modules listed above with minor variations). Various scenarios (single document import, etc.) all seem to point to the post body not being correctly recognized for some reason.
var needle = require('needle');
var data = [{
"lastname": "ln",
"firstname": "fn",
},
{
"lastname": "ln2",
"firstname": "fn2"
}];
var options = { 'Content-Type': 'application/json; charset=utf-8' };
needle.request('POST', 'http://ip:8529/_db/mydb/_api/import?type=array&collection=accounts&createCollection=false', data, options, function(err, resp) {
console.log(resp.body);
});
While i am able to upload the documents using curl and browser dev tools, I have not been able to get it working in node.js. What am i doing wrong? This is driving me crazy. Any help would be appreciated. Thank you very much.

You can use ngrep (or wireshark) to quickly find out whats wrong:
ngrep -Wbyline port 8529 -d lo
T 127.0.0.1:53440 -> 127.0.0.1:8529 [AP]
POST /_db/mydb/_api/import?type=array&collection=accounts& createCollection=true HTTP/1.1.
Accept: */*.
Connection: close.
User-Agent: Needle/0.9.2 (Node.js v1.8.1; linux x64).
Content-Type: application/x-www-form-urlencoded.
Content-Length: 51.
Host: 127.0.0.1:8529.
.
##
T 127.0.0.1:53440 -> 127.0.0.1:8529 [AP]
lastname=ln&firstname=fn&lastname=ln2&firstname=fn2
The body to be sent to ArangoDB has to be json (as you try to achieve by setting the content type).
Making needle to actualy post json works this way: (see https://github.com/tomas/needle#request-options )
var options = {
Content-Type: 'application/json; charset=utf-8',
json: true
};
which produces the proper reply:
{ error: false,
created: 2,
errors: 0,
empty: 0,
updated: 0,
ignored: 0 }

Related

Trouble building a valid request for dropbox that succeeds with curl

I've been using reqwest for a few months now; solid features, reuse of types defined in http (and the like) and documentation.
I have the following http post request that works in Postman and curl that for reasons I can't figure out, generates a http::StatusCode "400 Bad Request".
Staring at and comparing the debug output of the request (below) over the past couple of days hasn't helped (ok, and a solid amount of reading the docs and playing with code too :)); thus the post.
The request spec that works with other clients
POST /2/files/list_folder HTTP/1.1
Host: api.dropboxapi.com
Authorization: Bearer <secret>
Content-Type: application/json
Content-Length: 65
{
"path": "",
"include_non_downloadable_files": false
}
The failing reqwest request code
let client = reqwest::Client::new();
let request = client
.request(
http::Method::from_str(method).unwrap(),
format!(
"{host}{endpoint}{query}",
host = &drive_server,
endpoint = &endpoint,
query = &query_ls
),
)
.header(AUTHORIZATION, format!("Bearer {}", access_token.secret()))
.header(CONTENT_TYPE, "application/json") // redundant when setting body with .json
.header(ACCEPT, "application/json")
.version(Version::HTTP_11)
.json(&req_body);
... and the code that builds the req_body:
// dropbox WIP
let json = r#"
{
"path": "",
"include_non_downloadable_files": false
}
"#;
#[derive(Deserialize, Serialize)]
struct ReqBody {
path: String,
include_non_downloadable_files: bool,
}
let req_body = serde_json::from_str::<ReqBody>(json)
.map_err(|err| AuthError::JsonParsingError(err.into()))?;
Debug output of the request
RequestBuilder {
method: POST,
url: Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"api.dropboxapi.com",
),
),
port: None,
path: "/2/files/list_folder",
query: None,
fragment: None,
},
headers: {
"authorization": "Bearer <secret>",
"content-type": "application/json",
"accept": "application/json",
},
}
Aside but to know what to expect from the diagnostic: Is the output expected to exclude the request body when present? I have not found a way to echo the request body once "theoretically" set :))
A bit more context
Finally, a bit more background, the explicit call to the version is a failed attempt on my part to override google drive forcing a switch to http 2 when it sees a request body (a separate use of this client). I'll have to figure out another way that includes some sort of conditional call to .json(<body>) without violating ownership issues, when services like dropbox require the body but not google. When I disable the .json on the reqwest RequestBuilder for a service such as google, the request works as expected.
Plainly put, the question
How do I build the http post request described at the top of this post using reqwest?

postman POST request doesn't work in nodejs (accessing 3rd party API)

I've been playing around with some web scraping but I've run into an issue I can't figure out; Using a nodejs server (on my local computer) I cannot get passed a permission error barring me from accessing the data. What is confusing to me most is that using the chrome extension "Postman" I don't run into the permission errors, but using the code generated by postman, I do (as well as fiddling with variations of my own scratch code).
Do I have to be using a live server? Do I need to include some extra items in the headers that aren't being put there by Postman? Is there some layer of security around the API that for some reason Postman has access do that a local machine doesnt?
Any light that can be shed would be of use. Note that there is no public documentation of the SmithsFoodAndDrug API (that I can find), so there aren't necessarily APIKeys that are going to be used. But the fact that Postman can access the information makes me think I should be able to on a node server without any special authentication set up.
In Summary:
I'm looking at SmithsFoodAndDrug product information, and found the API where they are grabbing information from.
I figured out the headers needed in order to get local price information on products (on top of the json body format for the POST request)
Using postman I can generate the POST request and retrieve the desired API results
Using nodejs (and the code generated by postman to replicate the request) with both 'request' module and standard 'http' module request module I receive permission errors from the server.
Details: (assume gathering data on honeycrisp apples (0000000003283) with division-id of 706 and store-id of 00144)
http://www.smithsfoodanddrug.com/products/api/products/details
Headers are 'division-id' and 'store-id'. Body is in format of {"upcs":["XXX"],"filterBadProducts":false} where XXX is the specific product code.
Here are the Request Headers in postman. Here are the Request Body settings in postman. The following is a portion of the json response (which is what I want).
{"products": [
{
"brandName": null,
"clickListItem": true,
"countryOfOrigin": "Check store for country of origin details",
"customerFacingSize": "price $2.49/lb",
...
"calculatedPromoPrice": "2.49",
"calculatedRegularPrice": "2.99",
"calculatedReferencePrice": null,
"displayTemplate": "YellowTag",
"division": "706",
"minimumAdvertisedPrice": null,
"orderBy": "Unit",
"regularNFor": "1",
"referenceNFor": "1",
"referencePrice": null,
"store": "00144",
"endDate": "2018-09-19T00:00:00",
"priceNormal": "2.55",
"priceSale": "2.12",
"promoDescription": "About $2.12 for each",
"promoType": null,
...
"upc": "0000000003283",
...
}
],
"coupons": {},
"departments": [],
"priceHasError": false,
"totalCount": 1 }
When using the code given by postman to replicate the request, I get the error saying 'You don't have permission to access "http://www.smithsfoodanddrug.com/products/api/products/details" on this server.
Reference #18.1f3de93f.1536955806.1989a2b1.' .
// Code given by postman
var request = require("request");
var options = { method: 'POST',
url: 'http://www.smithsfoodanddrug.com/products/api/products/details',
headers:
{ 'postman-token': 'ad9638c1-1ea5-1afc-925e-fe753b342f91',
'cache-control': 'no-cache',
'store-id': '00144',
'division-id': '706',
'content-type': 'application/json' },
body: { upcs: [ '0000000003283' ], filterBadProducts: false },
json: true };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
change headers
headers:
{
'store-id': '00144',
'division-id': '706'
//'content-type': 'application/json'
}

Making a Post request to Github API for creating issue is not working

I have been trying to make this post request to the github api for the last couple of days, but unfortunately the response is coming back as "bad message"
here is the piece of code we are sending in the post request using https request in node -
This is the post data
var issueData = JSON.stringify({
"title":title,
"body":comment
});
This is the options file
var options = {
host: 'api.github.com',
path: '/repos/sohilpandya/katasohil/issues?access_token='+sessions.token,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0',
},
method: 'POST'
};
This is the https request
var requestaddIssue = https.request(options, function(responseFromIssues){
responseFromIssues.setEncoding('utf8');
responseFromIssues.on('data', function(chunk){
console.log('>>>>chunk>>>>>',chunk);
issueBody += chunk;
});
responseFromIssues.on('end',function(issueBody){
console.log(issueBody);
});
});
requestaddIssue.write(issueData);
requestaddIssue.end();
I have tried another approach where the authentication token for the user is in the header as
'Authentication': 'OAuth '+ sessions.token (where we are storing token inside sessions)
But the chunk response always seems to come back with the following in the console log.
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3/issues/#create-an-issue"
}
I have tried the same in apigee and it seems to work ok and returns to correct response. Hoping someone can find the minor error in the code above that is causing this bad message error.
Except the issueBody variable is not defined in the snippets you posted, the code is correct. I tried it using a personal access token.
The error you get appears because you need to add a scope with power to open issues.
I tried the repo and public_repo scopes and they are both working. Note that repo has access to private repositories. Here you can see the list of scopes.
If you're using OAuth, then you you should have an url looking like this:
https://github.com/login/oauth/authorize?client_id=<client-id>&scope=public_repo&redirect_uri=<redirect-uri>

How to send a Single-Part POST Request in Node.js?

PhantomJs's webserver does not support multipart requests, so I'm trying to send a single-part request from NodeJs.
Unfortunatly the nodejs example looks to be multipart. is there any way of doing this with NodeJs?
http://nodejs.org/api/http.html#http_http_request_options_callback
edit:
in the nodejs docs it mentions:
Sending a 'Content-length' header will disable the default chunked encoding.
but unfortunatly it's still multi-part, just not multi-multipart :P
edit2: for showing code, it's a bit hard to show a distilled example, but here goes:
node.js code (it's Typescript code):
```
//send our POST body (our clientRequest)
var postBody = "hello";
var options : __node_d_ts.IRequestOptions = {
host: host,
port: port,
path: "/",
method: "POST",
headers: {
"Content-Type": "application/json",
"Content-length": postBody.length
}
};
//logger.assert(false);
var clientRequest = http.request(options,(response: http.ServerResponse) => {
//callback stuff here
});
clientRequest.on("error", (err) => {
thisObj.abort("error", "error,request error", err);
});
//clientRequest.write();
clientRequest.end(postBody);
```
when i read the results from PhantomJS, the post/postRaw fields are null.
when I use a tool like the Chrome "Advanced REST Client" extension to send a POST body, phantomjs gets it no problem.
i don't have a network sniffer, but as described here, it says phantomjs doesnt work with multipart so I think that's a good guesss: How can I send POST data to a phantomjs script
EDIT3:
indeed, here's the request phantomjs gets from my chrome extension (valid post)
//cookie, userAgent, and Origin headers removed for brevity
{"headers":{"Accept":"*/*","Accept-Encoding":"gzip,deflate,sdch","Accept-Language":"en-US,en;q=0.8,ko;q=0.6","Connection":"keep-alive","Content-Length":"5","Content-Type":"application/json","DNT":"1","Host":"localhost:41338", "httpVersion":"1.1","method":"POST","post":"hello","url":"/"}
and here's the request phantomjs gets from the nodejs code i show above:
//full request, nothing omitted!
{"headers":{"Connection":"keep-alive","Content-Type":"application/json","Content-length":"5","Host":"10.0.10.15:41338"},"httpVersion":"1.1","method":"POST","url":"/"}

Getting 401 uploading file into a table with a service account

I am using nodejs and the REST API to interact with bigquery. I am using the google-oauth-jwt module for JWT signing.
I granted a service account write permission. So far I can list projects, list datasets, create a table and delete a table. But when it comes to upload a file via multipart POST, I ran into two problems:
gzipped json file doesn't work, I get an error saying "end boundary missing"
when I use uncompressed json file, I get a 401 unauthorized error
I don't think this is related to my machine's time being out of sync since other REST api calls worked as expected.
var url = 'https://www.googleapis.com/upload/bigquery/v2/projects/' + projectId + '/jobs';
var request = googleOauthJWT.requestWithJWT();
var jobResource = {
jobReference: {
projectId: projectId,
jobId: jobId
},
configuration: {
load: {
sourceFormat: 'NEWLINE_DELIMITED_JSON',
destinationTable: {
projectId: projectId,
datasetId: datasetId,
tableId: tableId
},
createDisposition: '',
writeDisposition: ''
}
}
};
request(
{
url: url,
method: 'POST',
jwt: jwtParams,
headers: {
'Content-Type': 'multipart/related'
},
qs: {
uploadType: 'multipart'
},
multipart: [
{
'Content-Type':'application/json; charset=UTF-8',
body: JSON.stringify(jobResource)
},
{
'Content-Type':'application/octet-stream',
body: fileBuffer.toString()
}
]
},
function(err, response, body) {
console.log(JSON.parse(body).selfLink);
}
);
Can anyone shine some light on this?
P.S. the documentation on bigquery REST api is not up to date on many things, wish the google guys can keep it updated
Update 1:
Here is the full HTTP request:
POST /upload/bigquery/v2/projects/239525534299/jobs?uploadType=multipart HTTP/1.1
content-type: multipart/related; boundary=71e00bd1-1c17-4892-8784-2facc6998699
authorization: Bearer ya29.AHES6ZRYyfSUpQz7xt-xwEgUfelmCvwi0RL3ztHDwC4vnBI
host: www.googleapis.com
content-length: 876
Connection: keep-alive
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/json
{"jobReference":{"projectId":"239525534299","jobId":"test-upload-2013-08-07_2300"},"configuration":{"load":{"sourceFormat":"NEWLINE_DELIMITED_JSON","destinationTable":{"projectId":"239525534299","datasetId":"performance","tableId":"test_table"},"createDisposition":"CREATE_NEVER","writeDisposition":"WRITE_APPEND"}}}
--71e00bd1-1c17-4892-8784-2facc6998699
Content-Type: application/octet-stream
{"practiceId":2,"fanCount":5,"mvp":"Hello"}
{"practiceId":3,"fanCount":33,"mvp":"Hello"}
{"practiceId":4,"fanCount":71,"mvp":"Hello"}
{"practiceId":5,"fanCount":93,"mvp":"Hello"}
{"practiceId":6,"fanCount":92,"mvp":"Hello"}
{"practiceId":7,"fanCount":74,"mvp":"Hello"}
{"practiceId":8,"fanCount":100,"mvp":"Hello"}
{"practiceId":9,"fanCount":27,"mvp":"Hello"}
--71e00bd1-1c17-4892-8784-2facc6998699--
You are most likely sending duplicate content-type headers to the Google API.
I don't have the capability to effortlessly make a request to Google BigQuery to test, but I'd start with removing the headers property of your options object to request().
Remove this:
headers: {
'Content-Type': 'multipart/related'
},
The Node.js request module automatically detects that you have passed in a multipart array, and it adds the appropriate content-type header. If you provide your own content-type header, you most likely end up with a "duplicate" one, which does not contain the multipart boundary.
If you modify your code slightly to print out the actual headers sent:
var req = request({...}, function(..) {...});
console.log(req.headers);
You should see something like this for your original code above (I'm using the Node REPL):
> req.headers
{ 'Content-Type': 'multipart/related',
'content-type': 'multipart/related; boundary=af5ed508-5655-48e4-b43c-ae5be91b5ae9',
'content-length': 271 }
And the following if you remove the explicit headers option:
> req.headers
{ 'content-type': 'multipart/related; boundary=49d2371f-1baf-4526-b140-0d4d3f80bb75',
'content-length': 271 }
Some servers don't deal well with multiple headers having the same name. Hopefully this solves the end boundary missing error from the API!
I figured this out myself. This is one of those silly mistakes that would have you stuck for the whole day and at the end when you found the solution you would really knock on your own head.
I got the 401 by typing the selfLink URL in the browser. Of course it's not authorized.

Resources