how to do this curl operation in Node.js using request library - node.js

How can I convert this curl operation using request Node.js library:
curl -L -X GET -H "Content-Type:application/json" -H "Authorization: authorization..." -H "Scope: 11111111" https://url/download >> file.gz
/*the -L is curl option which means --location Follow redirects (H)
--location-trusted Like '--location', and send auth to other hosts (H)*/

If you just want to download to a file, you can use a head request type.
The request will look like so:
request.head({
url: "https://url/download",
followAllRedirects: true,
headers: {
'Content-Type': 'application/json',
'Authorization':'authorization...',
'Scope': '11111111'
}
}, (err, res, body) => {
request('https://url/download')
.pipe(fs.createWriteStream(`tmp/${res.path}`)).on('close', (data, err) => {
if(err) console.log(`Unable to write to file ${err}`)
console.log('Done')
})
})
I have used a similar snippet which worked well
Use Postmans code generator
Click code on the top left
Paste your curl request
Select Node.js Request from dropdown on top left of popup
You should then get JS snippet converted from your working cURL request

Here is the solution we have to put request inside another because: so the first request returns the url and the second one will download the file
const options = {
url: url,
headers: {
Authorization: `auth`,
'scope': profileId,
'content-type': 'application/json'
},
};
const r = request.get(options, async (err, res, body) => {
const fileStream = fs.createWriteStream(`${path}.gz`);
request(res.request.uri.href).pipe(fileStream);
// path exists unless there was an error
});

Related

How to upload an image to Cloudflare Images from Node using fetch?

I'm trying to upload an image from my computer to Cloudflare Images using Node and the fetch API. They provided me with a curl command that works fine.
curl -X POST -F file=#./<file_name> -H "Authorization: Bearer <api_token>" https://api.cloudflare.com/client/v4/accounts/<domain_id>/images/v1
When I tried to convert it to fetch, Cloudflare keeps sending me a 400 in their response.
const cloudflarePostBody = new FormData();
cloudflarePostBody.append("file", fs.createReadStream("testing.jpeg"));
const cloudflareResponse = await fetch("https://api.cloudflare.com/client/v4/accounts/<my_id>/images/v1", {
method: "POST",
headers: {
Authorization: `Bearer ${cloudflareApiToken}`,
"Content-Type": "multipart/form-data"
},
body: cloudflarePostBody,
});
My guess is that I'm doing something wrong with how I'm reading the file with createReadStream, but a lot of the examples I've looked up showed exactly that. Does anyone have any advice? Thanks in advance.
Try fs.readFile() instead of fs.createReadStream():
fs.readFile("testing.jpeg", (err, data) => {
if (err) throw err;
cloudflarePostBody.append("file", new Blob([data]), "testing.jpeg");
});

Making a node-fetch POST request with file as payload

first time poster, excited to have a community to collaborate with here!
So here's my situation. I'm using Node to put together an automation for my company's compliance - taking reports from our MDM server and posting them to our compliance platform (Tugboat Logic). The idea is to have it deployed on a recurring basis via AWS Lambda. The basic logic is this: getToken fetches an auth token which is then passed to getReports. getReports loops through an array of endpoints to get reports from the MDM and then passes those along to fileReport - which then posts that data to the endpoint.
The problem is that the final endpoint needs a file as the payload (sample POST request below). I managed to get the whole fetch chain working by using fs writeFile/readFile (and a delay), and while that worked, it doesn't translate well into a Lambda environment. Ideally, I want to just take the payload from getReports (which comes through as JSON but can also be accepted as text) and push it straight to the endpoint. Any help on how I could clean up this code would be appreciated!
Here's the bit giving me the most trouble (from the last file)
form.append('file', x, `${reportsArray[i].name}.json`);
// Sample post request for final endpoint
curl -v --user <provided-username>:<given-password> \
-H "X-API-KEY: <given-x-api-key>" \
-F "collected=<date-of-evidence>" -F "file=#<local_filename_csv>;type=text/csv" \
<given-collector-url>
//getReports.js accepts a token from an earlier function and takes fileReport as the cb
function getReports(token, cb) {
const headers = {
method: 'GET',
headers: {
'accept': 'application/json',
'Authorization': `Bearer ${token}`
},
redirect: 'follow'
}
for (let i = 0; i < reportsArray.length; i++) {
fetch(reportsArray[i].source, headers)
.then(res => res.json())
// writeFile leftover from successful deploy
/*.then(data => fs.writeFile(`./reports/${reportsArray[i].name}.json`, data, function (err) {
if (err) throw err;
}))*/
.then(res => cb(i, res))
.catch(error => console.log('error', error))
}
};
//fileReport.js - i identifies the right endpoint from the imported array and sets filename. x is the JSON payload passed down from getReports
function fileReport(i, x) {
const form = new FormData();
form.append('collected', getTimestamp());
form.append('file', x, `${reportsArray[i].name}.json`);
fetch(`${reportsArray[i].dest}`, {
method: 'POST',
headers: {
'X-API-KEY': `${process.env.TUGBOAT_X_API_KEY}`,
'Authorization': 'Basic ' + btoa(`${process.env.TUGBOAT_USERNAME}:${process.env.TUGBOAT_PASSWORD}`)
},
body: form
});
};

How to send file using request?

I am trying to hit an API which I got as a curl request. But When I am trying to upload it throwing an error that file is not uploaded. This is my curl request which is working fine.
curl -v -u admin:geoserver -XPUT -H "Content-type: application/zip" --data-binary #/Users/aitplap24/Desktop/WebGIS/uploads/users/61adbbe3ececf2068d6eb862/1638776276622.zip http://localhost:8080/geoserver/rest/workspaces/61adbbe3ececf2068d6eb862/datastores/layer1/61a604021467d3152d5720fa.shp
Here is my NodeJS code for doing the same using requests
const url = process.env.GEOSERVER_URL;
var headers = {
'Content-type': 'application/zip'
};
var dataString = '#/Users/aitplap24/Desktop/WebGIS/uploads/users/61adbbe3ececf2068d6eb862/1638776276622.zip';
var options = {
url: 'http://localhost:8080/geoserver/rest/workspaces/61adbbe3ececf2068d6eb862/datastores/test1/file.shp',
method: 'PUT',
headers: headers,
body: dataString,
formData : {
"file" : fs.createReadStream("/Users/aitplap24/Desktop/WebGIS/uploads/users/61adbbe3ececf2068d6eb862/1638776276622.zip")
},
auth: {
'user': 'admin',
'pass': 'geoserver'
}
};
request(options, (error, response, body) => {
console.log(error, response, body);
});
Error which I am getting
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
I am sure something I am doing wrong with file uploading. Any suggestions is of great help!

sending multiform data with DUPLICATE KEYS in nodejs

I need to send a POST request with NodeJS to an API that requires the same multiform key be used more than once.
This is a CURL example of the required action:
curl -H "Authorization: Bearer MY_ACCESS_TOKEN" -i -X POST -F "whitespace=1" \
-F "terms[]=lait" -F "definitions[]=milk" -F "terms[]=petits pois" \
-F "definitions[]=peas" -F "title=My first set via the API" \
-F "lang_terms=fr" -F "lang_definitions=en" \
https://api.quizlet.com/2.0/sets
As you can see, the keys "terms[]" and "definitions[]" are used more than once in the same request.
I've tried using the nodejs request/http/multi-form libraries with no success, as most of them require a JavaScript object to define the form data, which of course cannot accept duplicate keys.
Other than resorting to an exec() command to cURL, is there any nodejs library that will enable me to send a request with duplicate multiform keys?
I'm really banging my head against a wall with this one..
Try this its an example with request library
let options = { method: 'POST',
url:url,
headers:
{
'cache-control': 'no-cache',
authorization: 'Bearer '+accessToken ,
'content-type': 'application/json'
},
body:
{ //your array here
terms:['terms'] ,
definitions:['milk']
},
json: true
};
request(options, function (error, response, body) {
if(error){
console.log("Error ",error);
}
console.log("Response",body);
})
With unirest:
var unirest = require('unirest');
var req = unirest('POST', 'YOUR URL')
.headers({
'Content-Type': 'multipart/form-data; boundary=--------------------------846713359653092950719061',
'Authorization': 'YOUR AUTH'
})
.field('AAA', 'VAL1')
.field('AAA', 'VAL2')
.field('AAA', 'VAL3')
.end(function (res) {
if (res.error) throw new Error(res.error);
console.log(res.raw_body);
});

What is the equivalent of curl --upload-file in node-request

I'm using node-request and trying to send a file to IBM's HDFS, as per this documentation.
Passing this JSON object to request successfully uploads:
var options = {
method: 'put',
body: 'foo',
headers: {
'Content-Type': 'application/octet-stream',
'Transfer-Encoding': 'chunked'
}
};
And running this CURL command successfully uploads a file as well:
curl -v -X PUT -L -b cookie.jar "https://host:port/webhdfs/v1/tmp/myLargeFile.zip?op=CREATE&data=true" --header "Content-Type:application/octet-stream" --header "Transfer-Encoding:chunked" -T "file.txt"
However, trying to specify a file stream like so:
var options = {
method: 'put',
headers: {
'Content-Type': 'application/octet-stream',
'Transfer-Encoding': 'chunked'
},
multipart : [
{ body: fs.createReadStream(localFile) }
]
};
fails, and I don't know where I'm going wrong. How do I reproduce the '--upload-file' argument from CURL using node-request?
I threw up an example on Github that works.
The gist is that you need to pipe the file into Request:
fs.createReadStream(filePath).pipe(request.put(putURL,options,function(err, httpsResponse, body){
if ( err ) {
console.log('err', err);
} else {
console.log(body);
}
}));
Still not sure how to pass in a file stream as an option in the options params but this will do for me!
-- UPDATE --
Wow, I feel like an absolute idiot. The option would be file. Yes, it's that simple.

Resources