Making a node-fetch POST request with file as payload - node.js

first time poster, excited to have a community to collaborate with here!
So here's my situation. I'm using Node to put together an automation for my company's compliance - taking reports from our MDM server and posting them to our compliance platform (Tugboat Logic). The idea is to have it deployed on a recurring basis via AWS Lambda. The basic logic is this: getToken fetches an auth token which is then passed to getReports. getReports loops through an array of endpoints to get reports from the MDM and then passes those along to fileReport - which then posts that data to the endpoint.
The problem is that the final endpoint needs a file as the payload (sample POST request below). I managed to get the whole fetch chain working by using fs writeFile/readFile (and a delay), and while that worked, it doesn't translate well into a Lambda environment. Ideally, I want to just take the payload from getReports (which comes through as JSON but can also be accepted as text) and push it straight to the endpoint. Any help on how I could clean up this code would be appreciated!
Here's the bit giving me the most trouble (from the last file)
form.append('file', x, `${reportsArray[i].name}.json`);
// Sample post request for final endpoint
curl -v --user <provided-username>:<given-password> \
-H "X-API-KEY: <given-x-api-key>" \
-F "collected=<date-of-evidence>" -F "file=#<local_filename_csv>;type=text/csv" \
<given-collector-url>
//getReports.js accepts a token from an earlier function and takes fileReport as the cb
function getReports(token, cb) {
const headers = {
method: 'GET',
headers: {
'accept': 'application/json',
'Authorization': `Bearer ${token}`
},
redirect: 'follow'
}
for (let i = 0; i < reportsArray.length; i++) {
fetch(reportsArray[i].source, headers)
.then(res => res.json())
// writeFile leftover from successful deploy
/*.then(data => fs.writeFile(`./reports/${reportsArray[i].name}.json`, data, function (err) {
if (err) throw err;
}))*/
.then(res => cb(i, res))
.catch(error => console.log('error', error))
}
};
//fileReport.js - i identifies the right endpoint from the imported array and sets filename. x is the JSON payload passed down from getReports
function fileReport(i, x) {
const form = new FormData();
form.append('collected', getTimestamp());
form.append('file', x, `${reportsArray[i].name}.json`);
fetch(`${reportsArray[i].dest}`, {
method: 'POST',
headers: {
'X-API-KEY': `${process.env.TUGBOAT_X_API_KEY}`,
'Authorization': 'Basic ' + btoa(`${process.env.TUGBOAT_USERNAME}:${process.env.TUGBOAT_PASSWORD}`)
},
body: form
});
};

Related

Node.js? API Authentication problems

This is what my "dev" sent me. Someone help please
I'm trying my best, but their API doesn't respond to our methods. This authentication is the root of the problem. I'm right now using Axios(the most popular and only method for making API requests for web apps) but it's not accepting request
and then i told him i would ask for help*
You can ask this question- ` How do I make requests for creating order API in my express app? I've tried to make the request by getting my form data from my EJS form using the request.body. But still, it is saying error 400.
Here is his code:
app.post('/order-labels', checkAuthenticated, (req, res) => {
const data = JSON.stringify(req.body);
console.log(data)
const config = {
method: 'post',
url: 'https://labelsupply.io/api/order',
headers: {
'X-Api-Auth': '32854090-03dd-a3c1-Deleted some for safety',
'Content-Type': 'application/x-www-form-urlencoded'
},
data: data
};
axios(config)
.then(function(response) {
console.log(response.data);
})
.catch(function(error) {
console.log(error);
});
})
by console.logging we are getting the data, but the API doesn't accepting
The API Docs are here.
you may need an account to view just put junk
The API calls for url encoded string.
const data = JSON.stringify(req.body);
console.log(data)
data = new URLSearchParams(Object.entries(data)).toString();
console.log(data); // now should be URL encoded
const config = {
method: 'post',
url: 'https://labelsupply.io/api/order',
headers: {
'X-Api-Auth': '32854090-03dd-a3c1-Deleted some for safety',
'Content-Type': 'application/x-www-form-urlencoded'
},
data: data
};
See if the API likes the new encoding?

Walmart Seller API "Bulk Item Setup" doesn't work

I tried to use Walmart API v4.2 to publish some items. I used "Bulk Item Setup" API method to create some feed. I used some types of ways to did it:
Send binary file (in the request body, for-data) with header "multipart/form-data" (this way was described in the Walmart API docs)
Send stringified object in the request body with header 'Content-Type': 'application/json',
Walmart API correctly returns me feedId.
But all of these ways didn't work! After feed creating, I saw "Submitted" status at the Walmart Seller Center. But this status was changed after few minutes to "Error". At the error column I see "ERROR TYPE: Data Error" with a description "Malformed data. Please check data file to ensure it is formatted properly.java.lang.NullPointerException".
I use my backend NodeJs app to do it. I use Axios for making a request.
My code example:
async createFeed(wdpId, wdpSecret, accessToken, feedsData) {
try {
const string = JSON.stringify(feedsData);
const file = Buffer.from(string);
const formData = new FormData();
formData.append('file', file);
const baseToken = WalmartService.getBaseAuthToken(wdpId, wdpSecret);
const options = {
params: {
feedType: 'MP_WFS_ITEM',
},
headers: {
Authorization: baseToken,
'WM_SEC.ACCESS_TOKEN': accessToken,
'WM_QOS.CORRELATION_ID': uuidv4(),
'WM_SVC.NAME': 'Walmart Marketplace',
Accept: 'application/json',
'Content-Type': 'application/json',
...formData.getHeaders(),
},
};
return (
axios
.post(`${process.env.WALMART_API_BASEURL}/feeds`, formData, options)
.then((response) => {
return response.data;
})
.catch((error) => {
console.error(error.message);
throw new BadRequestException('Walmart error, ', error.message);
})
);
} catch (error) {
throw new BadRequestException('Can not create listing');
}
}
It is difficult to identify the exact issue based on the information you provided. Few things that you might want to check
If you are appending/attaching a file (as I see it in the code), use Content-Type header as "multipart/form-data. Also, make sure the file name has a .json extension if you are sending data as a json string. If you don't use this, it might default to xml and you will get the same error as what you see.
Try invoking the API using a rest client like Postman and verify if that call is successful.
If you do want to send the data as HTTP body (instead of a file), that should work too with Content-Type as application/json. This has not been documented on their developer portal, but it works.

how to do this curl operation in Node.js using request library

How can I convert this curl operation using request Node.js library:
curl -L -X GET -H "Content-Type:application/json" -H "Authorization: authorization..." -H "Scope: 11111111" https://url/download >> file.gz
/*the -L is curl option which means --location Follow redirects (H)
--location-trusted Like '--location', and send auth to other hosts (H)*/
If you just want to download to a file, you can use a head request type.
The request will look like so:
request.head({
url: "https://url/download",
followAllRedirects: true,
headers: {
'Content-Type': 'application/json',
'Authorization':'authorization...',
'Scope': '11111111'
}
}, (err, res, body) => {
request('https://url/download')
.pipe(fs.createWriteStream(`tmp/${res.path}`)).on('close', (data, err) => {
if(err) console.log(`Unable to write to file ${err}`)
console.log('Done')
})
})
I have used a similar snippet which worked well
Use Postmans code generator
Click code on the top left
Paste your curl request
Select Node.js Request from dropdown on top left of popup
You should then get JS snippet converted from your working cURL request
Here is the solution we have to put request inside another because: so the first request returns the url and the second one will download the file
const options = {
url: url,
headers: {
Authorization: `auth`,
'scope': profileId,
'content-type': 'application/json'
},
};
const r = request.get(options, async (err, res, body) => {
const fileStream = fs.createWriteStream(`${path}.gz`);
request(res.request.uri.href).pipe(fileStream);
// path exists unless there was an error
});

Isomorphic fetch (REDUX / nodeJs) - POST request with username / password

I am using the below code to get the data back from API and it works fine. I have another API which is secured one and that requires a user name / password to access the content and I am not sure how to pass in the credentials when using isomorphic fetch module. Can someone help?
I am using this module: https://www.npmjs.com/package/isomorphic-fetch
I need to pass in username and password as below (Sample curl command)
curl -u admin:hello123 http://test:8765/select?q=*
Code:
fetch(
url
).then(function (response) {
if (response.status != 200) {
dispatch(setError(response.status + '===>' + response.statusText + '===>' + response.url))
}
return response.json();
}).then(function (json) {
dispatch(setData(json, q))
}).catch(function(err){
});
Most APIs would used a POST request for authentication. They expect as well to receive the data to validate (user/password). Also, they usually require extra information in the header like to specify the format (e.g. application/json) of the data (the user/password data) you are sending. You are not passing any of that. Check below something that might work, but it all depends of what the API you are hitting is expecting (check its documentation).
fetch(url, {
method: 'POST',
headers: {
// Check what headers the API needs. A couple of usuals right below
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
// Validation data coming from a form usually
email: email,
password: password
}
}).then(function (response) {
if (response.status != 200) {
dispatch(setError(response.status + '===>' + response.statusText + '===>' + response.url))
}
return response.json();
}).then(function (json) {
dispatch(setData(json, q))
}).catch(function(err){
console.log(err);
};

Node JS upload file streams over HTTP

I'm switching one of my projects from request over to something a bit more light-weight (such as got, axios, or fetch). Everything is going smoothly, however, I'm having an issue when attempting to upload a file stream (PUT and POST). It works fine with the request package, but any of the other three return a 500 from the server.
I know that a 500 generally means an issue on the server's end, but it is consistent only with the HTTP packages that I'm testing out. When I revert my code to use request, it works fine.
Here is my current Request code:
Request.put(`http://endpoint.com`, {
headers: {
Authorization: `Bearer ${account.token.access_token}`
},
formData: {
content: fs.createReadStream(localPath)
}
}, (err, response, body) => {
if (err) {
return callback(err);
}
return callback(null, body);
});
And here is one of the attempts using another package (in this case, got):
got.put(`http://endpoint.com`, {
headers: {
'Content-Type': 'multipart/form-data',
Authorization: `Bearer ${account.token.access_token}`,
},
body: {
content: fs.createReadStream(localPath)
}
})
.then(response => {
return callback(null, response.body);
})
.catch(err => {
return callback(err);
});
Per the got documentation, I've also tried using the form-data package in conjunction with it according to its example and I still get the same issue.
The only difference between these 2 I can gather is with got I do have to manually specify the Content-Type header otherwise the endpoint does give me a proper error on that. Otherwise, I'm not sure how the 2 packages are constructing the body with the stream, but as I said, fetch and axios are also producing the exact same error as got.
If you want any of the snippets using fetch or axios I'd be happy to post them as well.
I know this question was asked a while ago, but I too am missing the simple pipe support from the request package
const request = require('request');
request
.get("https://res.cloudinary.com/demo/image/upload/sample.jpg")
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
and had to do some experimenting to find similar features from current libraries.
Unfortunately, I haven't worked with "got" but I hope the following 2 examples help someone else that are interested in working with the Native http/https libraries or the popular axios library
HTTP/HTTPS
Supports piping!
const http = require('http');
const https = require('https');
console.log("[i] Test pass-through: http/https");
// Note: http/https must match URL protocol
https.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
(imageStream) => {
console.log(" [i] Received stream");
imageStream.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
);
}
);
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
)
Axios
Note the usage of imageStream.data and that it's being attached to data in the Axios config.
const axios = require('axios');
(async function selfInvokingFunction() {
console.log("[i] Test pass-through: axios");
const imageStream = await axios.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
{
responseType: "stream", // Important to ensure axios provides stream
}
);
console.log(" [i] Received stream");
const upload = await axios({
method: "post",
url: "http://127.0.0.1:8000/api/upload/stream/",
data: imageStream.data,
headers: {
"Content-Type": imageStream.headers["content-type"],
},
});
console.log("Upload response", upload.data);
})();
Looks like this was a headers issue. If I use the headers directly from FormData (i.e., headers: form.getHeaders()) and just add in my additional headers afterwards (Authorization), then this ends up working just fine.
For me just works when I added other parameters on FormData.
before
const form = new FormData();
form.append('file', fileStream);
after
const form = new FormData();
form.append('file', fileStream, 'my-whatever-file-name.mp4');
So that way I can send stream from my backend to another backend in node, waiting a file in multipart/form-data called 'file'

Resources