How to upload file from the back-end to another back-end? - node.js

I am trying to upload file from another back-end to currently existing API.
For doing so, I am trying to replicate what's being done in the front-end.
To create FormData and then send it to the API.
But the request seems to hang...
And if I would try to replicate the request express would console log error
Error: Request aborted
at IncomingMessage.<anonymous> (C:\Users\TCP_BCP\node_modules\formidable\lib\incoming_form.js:122:19)
at IncomingMessage.emit (events.js:314:20)
at IncomingMessage.EventEmitter.emit (domain.js:506:15)
at abortIncoming (_http_server.js:533:9)
at socketOnEnd (_http_server.js:549:5)
at Socket.emit (events.js:326:22)
at Socket.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
I understand that if I am trying to perform new request the old one gets aborted probably.
But why the request could hang? The target file is small in size (500kb +-).
In the front-end there would be used $(".form").ajaxSubmit
In the back-end I am trying to create form with FormData.
What could be wrong?

So the issues was with headers.
Formidable at other API would hang without passing FormData headers for axios.
Code bellow allowed me to successfully upload files to a other API.
const data = await fs.readFile(zipPath);
const form = new FormData();
form.append("file", data, zipName);
const headers = form.getHeaders();
Axios.defaults.headers.cookie = cookies;
await Axios({
method: "POST",
url: endpoint,
headers: {
...headers,
cookie: cookies,
},
data: form,
});

Related

AxiosError: maxContentLength size of -1 exceeded

I am trying to download some images from a website using axios, but I get this error every time I try. I tried setting the maxContentLength, but it didn't fix anything.
This is a part of my code
const resp = await axios({
method: 'GET',
url: `${host}/data/${chapterHash}/${page}`,
maxContentLength: 20000,
responseType: 'arraybuffer'
this is the error
const err = new AxiosError(
^
AxiosError: maxContentLength size of -1 exceeded
at IncomingMessage.handlerStreamAborted (C:\Users\alexou\Documents\node-reader\node_modules\axios\dist\node\axios.cjs:2912:23)
at IncomingMessage.emit (node:events:513:28)
at IncomingMessage._destroy (node:_http_incoming:224:10)
at _destroy (node:internal/streams/destroy:102:25)
at IncomingMessage.destroy (node:internal/streams/destroy:64:5)
at TLSSocket.socketCloseListener (node:_http_client:441:11)
at TLSSocket.emit (node:events:525:35)
at node:net:757:14
at TCP.done (node:_tls_wrap:584:7) {
code: 'ERR_BAD_RESPONSE'
what I ended up doing was just adding a try{ }catch. The code isn't crashing anymore, but I still get the errors. If you have a better way of handling this, please tell me.

https.get not working to access a public API with Nodejs

I'm currently trying to access the JokeAPI using Node's module HTTPS, where below is my code:
const https = require('https');
let url = 'https://v2.jokeapi.dev/joke/Any';
https.get(url, function (res) {
console.log(res);
})
However, somehow I keep getting this error:
node:events:498
throw er; // Unhandled 'error' event
^
Error: self signed certificate in certificate chain
at TLSSocket.onConnectSecure (node:_tls_wrap:1530:34)
at TLSSocket.emit (node:events:520:28)
at TLSSocket._finishInit (node:_tls_wrap:944:8)
at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:725:12)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:520:28)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
code: 'SELF_SIGNED_CERT_IN_CHAIN'
Is it because I don't have a server set up? I'm a beginner with JS and Node, so any help would be very appreciated. Thank you!
Edit: I actually added 'process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';' to the top of the code and it worked, but now I'm getting an 'undefined' for a response body. And I'm not sure why that is happening :(
My system does not show the self signed certificate in certificate chain error, but it sounds like you have a work-around for that.
However, now I'm getting an 'undefined' for a response body. I'm not too sure what's causing this.
https.get() just sends the request and reads the headers. It does not, by itself, read the whole response. To do that, you need to listen for the data event on the res object. Installing the listener for the data event will cause the stream to start reading data from the incoming request. This data will not necessarily come all at once. It can come broken into multiple data events so you would really need to accumulate all the data and then in the end event, you know you have all the data. You can see a full example for how to implement that here in the http.get() doc.
Here's a simpler version that will just show you what's in each data event.
const https = require('https');
let url = 'https://v2.jokeapi.dev/joke/Any';
https.get(url, function(res) {
res.on('data', data => {
console.log(data.toString());
});
});
FYI, I no longer use plain https.get() because all the higher level libraries listed here are more convenient to use and all support promises which is generally an easier way to program with asynchronous requests.
My favorite library from that list is got(), but you can choose whichever one you like the API and/or features for. Here's an implementation using got():
const got = require('got');
const url = 'https://v2.jokeapi.dev/joke/Any';
got(url).json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});

Duplicate Content-Length with Axios running in NodeJS/JEST

I face really weird problem with Axios / nanoexpress. I started to write JEST test and copied working piece of code from Vuex that calls my backend. This code works reliably in Vue but here it fails with a network error. The Axios version is the same on both SPA and JEST test.
JEST:
let response = await axios.get(`${API}/users/1234`, getAuthHeader());
function getAuthHeader() {
const config = { headers: { } };
config.headers.Authorization = "bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxZTQwdjBiMWo1Iiwibmlja25hbWUiOiJsaXRlcmFrbCIsInB3ZFRpbWVzdGFtcCI6IjIwMjAtMDMtMjJUMTE6MTA6NDkuNDg2WiIsInJvbGVzIjpbImFkbWluOnBvbGwiXSwiaWF0IjoxNTg2NjkwNzc2LCJleHAiOjE1ODkzNjkxNzZ9.N5MfpZ9i9Sjv-izdYItR4gXCmzVkqkuVcVSEL_6Q89c";
return config;
}
When I remove the config with Authorization header, the code works well:
let response = await axios.get(`${API}/users/1234`, getAuthHeader());
This is the error:
Network Error
at createError (node_modules/axios/lib/core/createError.js:16:15)
at XMLHttpRequest.handleError (node_modules/axios/lib/adapters/xhr.js:83:14)
at XMLHttpRequest.<anonymous> (node_modules/jsdom/lib/jsdom/living/helpers/create-event-accessor.js:33:32)
at innerInvokeEventListeners (node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:316:27)
at invokeEventListeners (node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:267:3)
at XMLHttpRequestEventTargetImpl._dispatch (node_modules/jsdom/lib/jsdom/living/events/EventTarget-impl.js:214:9)
at fireAnEvent (node_modules/jsdom/lib/jsdom/living/helpers/events.js:17:36)
at requestErrorSteps (node_modules/jsdom/lib/jsdom/living/xhr-utils.js:121:3)
at Object.dispatchError (node_modules/jsdom/lib/jsdom/living/xhr-utils.js:51:3)
at EventEmitter.<anonymous> (node_modules/jsdom/lib/jsdom/living/xmlhttprequest.js:675:20)
at Request.<anonymous> (node_modules/jsdom/lib/jsdom/living/xhr-utils.js:384:47)
at Request.onRequestError (node_modules/request/request.js:877:8)
console.error node_modules/jsdom/lib/jsdom/virtual-console.js:29
Error: Error: Parse Error: Duplicate Content-Length
at Object.dispatchError (C:\Users\leos\WebstormProjects\nano-options\node_modules\jsdom\lib\jsdom\living\xhr-utils.js:54:19)
at EventEmitter.<anonymous> (C:\Users\leos\WebstormProjects\nano-options\node_modules\jsdom\lib\jsdom\living\xmlhttprequest.js:675:20)
at EventEmitter.emit (events.js:327:22)
at Request.<anonymous> (C:\Users\leos\WebstormProjects\nano-options\node_modules\jsdom\lib\jsdom\living\xhr-utils.js:384:47)
at Request.emit (events.js:315:20)
at Request.onRequestError (C:\Users\leos\WebstormProjects\nano-options\node_modules\request\request.js:877:8)
at ClientRequest.emit (events.js:315:20)
at Socket.socketOnData (_http_client.js:486:9)
at Socket.emit (events.js:315:20)
at addChunk (_stream_readable.js:297:12) undefined
The service is nothing fancy and it is never called when Authorization header is present and it is called when the Authorization header is missing.
app.get('/v1/users/:userId', async (req, res) => {
console.log("getUser handler starts");
const response = {};
res.send(response);
return res;
});
I have spent several hours trying to see the raw network communication but failed. It must be something with Axios because I can run successfuly exactly the same request with the Authorization header from Postman.
Update:
it seems to be related to NodeJS version. I submitted an issue: https://github.com/axios/axios/issues/2889
Update 2:
I switched from Axios to Got and the test is working properly. https://github.com/literakl/mezinamiridici/commit/e667c5661429fc8cba273716dfced2244e211abf

React Native/Node API: Can't set headers after they are sent

I am having an issue with my react native app interacting with my Node API.
When I try to post data to update some values in the database through node (which then goes to a stored procedure), I get a 500 - Can't set headers after they are sent in Node.
Everything I can find says that this could be due to sending responses twice. I don't think this is the case here. I've tested in postman and things work fine, it returns a status of 200 with the correct returned data.
I am trying to post data to the API like so:
const myHeaders = new Headers();
myHeaders.append('Content-Type', 'application/json');
fetch(`http://localhost:3000/user/preferences`, {
method: 'POST',
headers: myHeaders,
mode: 'cors',
cache: 'default',
body: JSON.stringify({
id: '1',
minage: this.state.minageValue,
maxage: this.state.maxageValue
})
})
.then(response => response.json())
.then(body => console.log(body))
.catch(err => console.log(err));
I am receiving it on the API side and passing the data to a stored procedure with this:
function updatePreferences(req, res, next) {
console.log(req);
var userid = req.body[0].id;
var minage = req.body[0].minage;
var maxage = req.body[0].maxage;
db.func('___spsavepreferences', [userid, minage, maxage])
.then(function(data){
res.status(200)
.json({
status: 'success',
preferences: data
});
});
}
Any ideas?
EDIT:
Don't know if this tells me anything, but I have logged the error message to the console:
Error: Can't set headers after they are sent.
at validateHeader (_http_outgoing.js:504:11)
at ServerResponse.setHeader (_http_outgoing.js:511:3)
at ServerResponse.header (/Users/user/Projects/Project_node/node_modules/express/lib/response.js:730:10)
POST /user/preferences 500 410.809 ms - 189
at ServerResponse.send (/Users/user/Projects/Project_node/node_modules/express/lib/response.js:170:12)
at done (/Users/user/Projects/Project_node/node_modules/express/lib/response.js:967:10)
at Object.exports.renderFile (/Users/user/Projects/Project_node/node_modules/jade/lib/index.js:374:12)
at View.exports.__express [as engine] (/Users/user/Projects/Project_node/node_modules/jade/lib/index.js:417:11)
at View.render (/Users/user/Projects/Project_node/node_modules/express/lib/view.js:128:8)
at tryRender (/Users/user/Projects/Project_node/node_modules/express/lib/application.js:640:10)
at Function.render (/Users/user/Projects/Project_node/node_modules/express/lib/application.js:592:3)
at ServerResponse.render (/Users/user/Projects/Project_node/node_modules/express/lib/response.js:971:7)
at /Users/user/Projects/Project_node/app.js:51:7
at Layer.handle_error (/Users/user/Projects/Project_node/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/Users/user/Projects/Project_node/node_modules/express/lib/router/index.js:315:13)
at /Users/user/Projects/Project_node/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/Users/user/Projects/Project_node/node_modules/express/lib/router/index.js:335:12)
EDIT 2: Got it. I needed to recieve the values as req.body.id, req.body.minage, etc... not as req.body[0].id.
Weird to get an error like that but that solved the error.
If you’re seeing different behavior in postman, that’s because postman just does the POST directly while the browser instead does a CORS preflight OPTIONS request before trying the POST.
So it sounds like the request that’s causing the “Can't set headers after they are sent” you’re seeing is that OPTIONS request, not the POST request.
You can confirm that by sending a OPTIONS request with postman, and making sure to also supply an Origin request header as part of the request. With curl you can do that like this:
curl -X OPTIONS -i -H 'Origin: http://x.y' http://localhost:3000/user/preferences
That will likely produce the same 500 error you’re seeing from the browser request.
As far as how to fix it, I think there’s no way anybody else here could tell you without seeing a lot more of your server side code. But what it would seem to come down to is, somewhere in your code you have one or more places with logic for handling OPTIONS requests, and somewhere in that OPTIONS-handling code, it’s trying to send headers after they’ve already been sent.
To track down where that is, it seems like you want to look through the server logs on the server side and find any message that’s getting logged when that 500 failure occurs.
And by the way, in this case the thing that triggers the browser to do a COR preflight OPTIONS request to begin with is the myHeaders.append('Content-Type', 'application/json') part.

issues deleting an image using Cloud Functions for Firebase and #google-cloud/storage

I'm trying to create a script in Cloud Functions for Firebase that will react to a db event and remove an image that has its path in one of the params ("fullPath").
this is the code i'm using:
'use strict';
const functions = require('firebase-functions');
const request = require('request-promise');
const admin = require('firebase-admin');
const gcs = require('#google-cloud/storage')({
projectId: 'XXXXXXX',
credentials: {
// removed actual credentials from here
}});
admin.initializeApp(functions.config().firebase);
// Deletes the user data in the Realtime Datastore when the accounts are deleted.
exports.removeImageOnNodeRemoval = functions.database
.ref("images/{imageId}")
.onWrite(function (event) {
// exit if we are creating a new record (when no previous data exists)
if (!event.data.previous.exists()) {
console.log("a new image added");
return;
}
// exit if we are just trying to update the image
if (event.data.exists()) {
console.log("image is been modified");
return;
}
let previousData = event.data.previous.val();
if(!previousData || !previousData.fullPath){
console.log("no data in the previous");
return;
}
let bucketName = 'XXXXXXX';
console.log("default bucketName", gcs.bucket(bucketName));
let file = gcs.bucket(bucketName).file(previousData.fullPath);
console.log('the file /'+previousData.fullPath, file);
file.exists().then(function(data) {
let exists = data[0];
console.info("file exists", exists);
});
file.delete().then(function() {
// File deleted successfully
console.log("image removed from project", previousData.fullPath);
}).catch(function(error) {
// Uh-oh, an error occurred!
console.error("failed removing image from project", error, previousData);
});
});
the error i'm getting:
failed removing image from project { ApiError: Not Found
at Object.parseHttpRespBody (/user_code/node_modules/#google-cloud/storage/node_modules/#google-cloud/common/src/util.js:192:30)
at Object.handleResp (/user_code/node_modules/#google-cloud/storage/node_modules/#google-cloud/common/src/util.js:132:18)
at /user_code/node_modules/#google-cloud/storage/node_modules/#google-cloud/common/src/util.js:465:12
at Request.onResponse [as _callback] (/user_code/node_modules/#google-cloud/storage/node_modules/retry-request/index.js:120:7)
at Request.self.callback (/user_code/node_modules/#google-cloud/storage/node_modules/request/request.js:188:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request.<anonymous> (/user_code/node_modules/#google-cloud/storage/node_modules/request/request.js:1171:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage.<anonymous> (/user_code/node_modules/#google-cloud/storage/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:291:16)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
code: 404,
errors: [ { domain: 'global', reason: 'notFound', message: 'Not Found' } ],
response: undefined,
message: 'Not Found' } { contentType: 'image/png',
fullPath: 'images/1491162408464hznsjdt6oaqtqmukrzfr.png',
name: '1491162408464hznsjdt6oaqtqmukrzfr.png',
size: '44.0 KB',
timeCreated: '2017-04-02T19:46:48.855Z',
updated: '2017-04-02T19:46:48.855Z' }
i have tried with and without credentials to google-cloud/storage (thinking they might get auto filled while im in firebase.functions - do i need them?). i have tried adding a slash to the file's path. i have validated that the file actually exists in the bucket (even tho file.exists() returns false). the credentials i provided are for an iam i created with admin privileges for the storage service.
i have also enable the billing account on the free plan.
any ideas?
ok, so i got this solved. here are my conclusions:
you need to add to your buckets name the ".appspot.com". Its not written anywhere in the docs and getting your bucket name in firebase is hard enough for someone that is not familiar in the google cloud. I hope they will add this little peace of information into their docs or make it clear what is your bucket's name in firebase.
use the environment variable process.env.GCLOUD_PROJECT, it should have your project id that is identical to your bucket id in firebase. again, remember to add the .appspot.com suffix to it.
regarding the credentials to GCS, you don't need to provide them when using cloud functions for firebase. you seem to be authenticated already.
Make sure your bucket name doesn't include gs://
Eg. instead of gs://my-project-id.appspot.com use my-project-id.appspot.com
let bucket = gcs.bucket('my-project-id.appspot.com')
This may happen to you (as it did to me) if you copy your bucket name from, for instance, Android code, where you can use full URL to connect with storage, ie. storage.getReferenceFromUrl('gs://my-proj...
.
Also, it seems the projectId variable you use to initialise gcs doesn't need the appspot.com suffix (but it shouldn't break if you have it included). Ie. projectId:'my-project-id' should be suffice.
Lastly, the GCS node package docs states you need to generate separate JSON to pass as keyFilename to test things locally. Good news is - you can use the Firebase Admin SDK key, as described in the Get Started Server/Admin docs instead.

Resources