External API responds with a multipart/mixed content type, and in my case, one part is some JSON and the second one is an octet-stream data of a PDF file. I'm building a POC with NodeJS 18 where fetch API is native and expect the response to have formData() method as per specs (https://developer.mozilla.org/en-US/docs/Web/API/Response/formData).
The sample code looks like this (apologies for mixing the promise and await):
fetch(url, {
method: 'GET',
headers,
}).then(async res => {
try {
const parts = await res.formData();
for(const entry of parts.entries()) {
writeFileSync(`./${entry[0]}`, entry[1]);
}
} catch (error) {
console.error('Error:', error);
}
process.exit();
})
.catch(error => {
console.error('Error:', error);
process.exit();
});
what I'm getting is a plain error
Error: TypeError: Response.formData: undefined
Which is quite unexpected. I couldn't find anything related to this behaviour and my first assumption is that the reason is the feature is still experimental.
Another reason I consider is that while API response has a proper Content-Type header it doesn't have Content-Length.
While I can parse the response body using some external dependency I'd rather stick with the native way. Any ideas about what the reason could be for this behavior?
Related
I'm having a bit of trouble taking a local file on the NodeJS server and uploading it to a web endpoint. I keep getting error:
AxiosError: Request failed with status code 400
at settle (D:\myproject\node_modules\axios\dist\node\axios.cjs:1855:12)
at IncomingMessage.handleStreamEnd (D:\myproject\node_modules\axios\dist\node\axios.cjs:2712:11)
at IncomingMessage.emit (node:events:539:35)
at endReadableNT (node:internal/streams/readable:1345:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) { code: 'ERR_BAD_REQUEST',
I'm not entirely sure what I'm doing wrong. I've poured over a lot of posts and seem to be structuring my AXIOS post request correctly?
try {
const fileStream = await fs.createReadStream('./pathtofile.ext');
let formData = new FormData();
formData.append('myFile', fileStream, 'pathtofile.ext');
axios.post('https://my.endpoint.com', formData, {
headers: {
...formData.getHeaders(),
}
})
.then(res => res.data)
.catch( err => console.log(err))
} catch(err) {
console.error(err)
}
I'm at a bit of a loss, as I can't seem to figure out what I'm doing wrong with such a simple task? The headers also appear to be correct when I call "getHeaders()", it's showing "'content-type': 'multipart/form-data;"
I appreciate any and all assistance!
I'm not sure why you're trying to read the file with a ReadStream, since you need the whole file to post it to your endpoint.
The benefits of using a ReadStream is to perform operations as you read the file. So with every chunk of the file you do something.
In this case, i believe all you need to do is just read the file and then firing your POST request.
You can do this by using readFileSync instead of createReadStream.
Your code should maybe go on this direction:
try {
const fileContents = await fs.readFileSync('./pathtofile.ext',{encoding: 'the-right-encoding'});
let formData = new FormData();
formData.append('myFile', fileContents, 'pathtofile.ext');
axios.post('https://my.endpoint.com', formData, {
headers: {
...formData.getHeaders(),
}
})
.then(res => res.data)
.catch( err => console.log(err));
} catch(err) {
console.error(err)
}
That way, the await keyword would have more sense, since you'll be waiting for the readFileSync promise to resolve.
Be sure to choose the right encoding for reading the file.
Also, since you're using Axios, you can check for the status of the upload by providing a callback function to the post method.
const onUploadProgress = (event) => {
const percentage = Math.round((100 * event.loaded) / event.total);
console.log(percentage);
};
Usage: https://www.bezkoder.com/axios-file-upload/
Firstly, you should go to a site like Reqbin and simulate the request to ensure your web endpoint is not the issue. I took the trouble to test your code with my own endpoint and it works perfectly. Most of the time, such issues are due to Authorization, Rate-limiting or something of that sort. Perhaps I could help out if I knew the kind of endpoint. Good luck.
On my webpage user can enter text and press send. this causes the server to append the message to a json object stored in a file. when this file is altered it then sends the new json to the client.
app.post("/recieve",function(req,res){
watcher = fs.watch(__dirname+"/msgs/msg.json", (eventName, filename) => {
watcher.close();
fs.readFile(__dirname+"/msgs/msg.json", (err,data) => {
return res.send(data);
});
});
})
here is the client side
async function recieveMSG(){
$.ajax({
url: "recieve",
type: "POST",
contentType: "text; charset=utf-8"
}).done(function(data){
$("#msgbox").html("<br>"+data+"<br>");
recieveMSG();
});
}
recieveMSG();
As shown in the code above, the client sends a POST request to the server. Next after the json file is changed the server responds to the POST request with the json. I know this may be the completely wrong way to do it, but I want to know why res.send(data) is being called twice on the same res object.
It seems after the first refresh the recieve POST request just doesnot do anything
app.post("/recieve",async function(req,res){
try{
watcher.close();
}
catch(e){
console.log("WatcherUndefined --first execution");
}
watcher = fs.watch(__dirname+"/msgs/msg.json", (eventName, filename) => {
watcher.close();
fs.readFile(__dirname+"/msgs/msg.json", (err,data) => {
return res.send(data);
});
});
})
The problem was that the watcher wasn't getting closed after the client refreshed/disconnected. After the client refreshed the res object generated by their stale request is unusable. I believe that the watcher's callback was never redefined with the new res object (after refresh). I do not know if my assumption is correct, and would like to hear other's thoughts on this as I am new to nodejs.
I am using the following JS in a webpage to send information to a Node.js server upon 'clicking' on an image in the webpage, I am having trouble with the 'redirect' once the 'fetch' is executed:
fetch('/members/pages/callup', {
method: 'post',
headers: {
'Accept': 'application/json, text/plain, */*',
'Content-Type': 'application/json'
},
body: JSON.stringify({name: splits[1], presence: available, str: 'Some string: &=&'})
})
.then(function(res) {res.json()})
.then(function(res) {
if(res.response) {
redirect: window.location.replace("/members/pages/" + splits[1]);
} else {
alert("Error in the response");
}
})
.catch(function(err) {
alert("Error in the fetch call..." + err);
})
The fetch seems to properly send the 'body' data to the server. However I am getting the following error: "Error in the fetch call...TypeError: Cannot read property 'response' of undefined"...
The server performs a database call using the information sent by the frontend, and I thought all I needed to do was to send a "200 (OK)" response back...here is the server code:
app.post('/member/pages/callup', jsonParser, function (req, res) {
console.log("I RECEIVED FROM CLIENT THE FOLLOWING:");
console.log(req.body); //works fine, prints output from frontend 'fetch' to console...
db.lookupMember(req.body.name)
.then(function(foundUser) {
console.log('Async success!', foundUser); //works fine, prints database info to console...
if (typeof foundUser != "undefined") {
res.sendStatus(200); //trying this to 'reply' back to 'fetch' in frontend...is this not correct?
} //'foundUser' is NOT'undefined'...
})
.catch(function(error) {
console.log('UNABLE TO RETRIEVE MEMBER INFORMATION FROM THE DATABASE...' + error);
res.redirect('/'); //route to splash page...
});
})
Any suggestions appreciated, this has gone from a minor irritant to a major problem. I thank you in advance.
There are few issues in the code. If fixed, code should work fine.
You forgot to return res.json() from the function at one place. Make it return res.json() and it will work fine (Inside fetch, 1st then). Due to not returning, res is undefined which is giving the error
You are trying to use res.response but res is not send as a proper json from node server. This will fail at res.json(). You should be doing something like res.send({response: true})
After the if loop in server there is syntax error. It needs to be redirect = instead of redirect:. Also redirect is not declared anywhere which. (Note: you might not need redirect variable here, simply window.lo... should also work)
Note: Have updated the original answer after having the discussion with OP
On firebase function I need to get data from Paypal and do 4 things :
1. returns an empty HTTP 200 to them.
2. send the complete message back to PayPal using `HTTPS POST`.
3. get back "VERIFIED" message from Paypal.
4. *** write something to my Firebase database only here.
What I do now works but i am having a problem with (4).
exports.contentServer = functions.https.onRequest((request, response) => {
....
let options = {
method: 'POST',
uri: "https://ipnpb.sandbox.paypal.com/cgi-bin/webscr",
body: verificationBody
};
// ** say 200 to paypal
response.status(200).end();
// ** send POST to paypal back using npm request-promise
return rp(options).then(body => {
if (body === "VERIFIED") {
//*** problem is here!
return admin.firestore().collection('Users').add({request.body}).then(writeResult => {return console.log("Request completed");});
}
return console.log("Request completed");
})
.catch(error => {
return console.log(error);
})
As you can see when I get final VERIFIED from Paypal I try to write to the db with admin.firestore().collection('Users')..
I get a warning on compile :
Avoid nesting promises
for the write line.
How and where should I put this write at that stage of the promise ?
I understand that this HTTPS Cloud Function is called from Paypal.
By doing response.status(200).end(); at the beginning of your HTTP Cloud Function you are terminating it, as explained in the doc:
Important: Make sure that all HTTP functions terminate properly. By
terminating functions correctly, you can avoid excessive charges from
functions that run for too long. Terminate HTTP functions with
res.redirect(), res.send(), or res.end().
This means that in most cases the rest of the code will not be executed at all or the function will be terminated in the middle of the asynchronous work (i.e. the rp() or the add() methods)
You should send the response to the caller only when all the asynchronous work is finished. The following should work:
exports.contentServer = functions.https.onRequest((request, response) => {
let options = {
method: 'POST',
uri: "https://ipnpb.sandbox.paypal.com/cgi-bin/webscr",
body: verificationBody
};
// ** send POST to paypal back using npm request-promise
return rp(options)
.then(body => {
if (body === "VERIFIED") {
//*** problem is here!
return admin.firestore().collection('Users').add({ body: request.body });
} else {
console.log("Body is not verified");
throw new Error("Body is not verified");
}
})
.then(docReference => {
console.log("Request completed");
response.send({ result: 'ok' }); //Or any other object, or empty
})
.catch(error => {
console.log(error);
response.status(500).send(error);
});
});
I would suggest you watch the official Video Series on Cloud Functions from Doug Stevenson (https://firebase.google.com/docs/functions/video-series/) and in particular the first video on Promises titled "Learn JavaScript Promises (Pt.1) with HTTP Triggers in Cloud Functions".
I'm working on my first CLI project and I'm having trouble getting it to execute API requests. I have tried fetch, axios, express, and a couple of npm packages, but I just can't figure out what's wrong. The project will console.log and gather user data from the command line, but will not retrieve API data. I'm using a fake API data url at this point just to be sure it works. Here is the code:
const axios = require('axios');
let apiResponse;
axios.get('https://jsonplaceholder.typicode.com/posts')
.then(function(response) {
apiResponse = response;
console.log('Does this work?')
})
.catch(function (error) {
console.log(error, 'Error');
});
console.log('apiResponse: ', apiResponse);
In the command line I get 'apiResponse: undefined' when I run the file. Again, I've tried using several different libraries so I must be doing something fundamentally wrong. The console.log OUTSIDE of the function prints, but neither console.logs INSIDE are printing. Any help would be greatly appreciated!
I'm guessing in your console you see
undefined
Does this work?
The .get method is asynchronous, which means any assignment outside of then will most likely always be what you initialize it as, in this case nothing, or undefined.
Here's a high level of how things are actually happening:
1) Create undefined var apiResponse
2) axios.get(...)
3) console.log(apiResponse)
4) #2 completes, assigns to `apiResponse`
5) End execution
Here's one of many resources about Promises.
Move the log statement inside the .then() block.
const axios = require('axios');
let apiResponse;
axios.get('https://jsonplaceholder.typicode.com/posts')
.then(function(response) {
apiResponse = response;
console.log('Does this work?')
console.log('apiResponse: ', apiResponse);
})
.catch(function (error) {
console.log(error, 'Error');
});