having issues uploading file from NodeJs server side, found 100 posts and reasearches but nothing works, would appreciate any help.
Structure of the App
Front App - React Admin framework receving file and i encode in base64 the content of the image to send to API
Backend - NestJS App - receving base64 image in API
From my backend API need to send file to an external backend (Python API) to upload - here is the problem
Please see below my code, something wrong with the file from JS
i have tried several methods and all of them ends in same error
1 solution
converting base64 image in buffer and send to external backend to upload the file
have tried to pass as well cleanImageBuffer but no changes
import axios from 'axios';
import FormData from 'form-data';
export async function upload(
fileBase64: string,
filename: string
): Promise<any> {
const buffer = Buffer.from(fileBase64, 'base64')
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const cleanImageBuffer = Buffer.from(cleanBase64, 'base64')
const formData = new FormData();
// have tried to pass as well cleanImageBuffer but no changes
formData.append('file', buffer);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', JSON.stringify(formData), {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
result 1 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
2 solution
from base64 image received saving on my disk
after creating a stream and sending the image
export async function upload (
fileBase64: string,
filename: string
): Promise<any> {
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const TMP_UPLOAD_PATH = '/tmp'
if (!fs.existsSync(TMP_UPLOAD_PATH)) {
fs.mkdirSync(TMP_UPLOAD_PATH);
}
fs.writeFile(TMP_UPLOAD_PATH + '/' + filename + '.' + extension, cleanBase64, 'base64', function(err) {
console.log(err);
})
const fileStream = fs.createReadStream(TMP_UPLOAD_PATH + '/' + filename + '.' + extension)
const formData = new FormData();
formData.append('file', fileStream, filename + '.' + extension);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', formData, {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
}
result 2 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
other solution that ended in same result
tried to use fetch from node-fetch - same result
found out that some people had an outdated version of axios and having this issues, i have installed latest axios version 1.1.3 but same result
best scenario that i need
from base64 image received
convert in buffer and send file to external Python API so to avoid saving the file on local disk
would appreciate any help
below is a python example that works but not JS (JS nothing works)
import requests
url = "http://127.0.0.1:8000/external_api"
payload={'namespace': 'test'}
files=[
('file',('lbl-pic.png',open('/local/path/lbl-pic.png','rb'),'image/png'))
]
headers = {
'Authorization': 'Bearer token'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
print(response.text)
Just a suggestion:
Here's a line which returns mentioned error https://github.com/tiangolo/fastapi/blob/41735d2de9afbb2c01541d0f3052c718cb9f4f30/fastapi/datastructures.py#L20, you might find it useful.
First see if you can make it work with regular HTML file input (don't complicate with Base64 yet), as described here https://stackoverflow.com/a/70824288/2347084
If (1) works, then try converting base64 into a File object as suggested here https://stackoverflow.com/a/47497249/2347084
Combine (2) and (1)
I want to post my solution that worked, because as i can see in internet everybody have issues with FormData on nodejs
i was using axios to send the buffer for uploading file
issue is with axios and specially with FormData, it does not add Content-Lenght in headers, any version of axios does not do this
python API had required Content-Lenght
So kindly was asking to make this header optionally in python API and the code started to work
The solution is if anybody goes in similar issues
axios does not add Content-Lenght when working with FormData (could not find any version of axios that works)
if you work with a buffer without having the file on local disk than will be issue because of Content-Lenght
if u have the file locally than using the module fs u are able to read the file and add all hedears and Content-Lenght
on axios GitHub issue is saying that this bug is fixed in latest axios, but it was still not working in my case
below is a code by using buffer and Content-Lenght is not required in 3rd API
function upload (image: {imageBase64: string, fileName: string}) {
const { imageBase64, fileName } = image;
const cleanBase64 = imageBase64.substr(imageBase64.indexOf(',') + 1);
// buffer should be clean base64
const buffer = Buffer.from(cleanBase64, 'base64');
const formData = new FormData();
// filename as option is required, otherwise will not work, will say that received file is string and UploadFile
formData.append('file', buffer, { filename: fileName });
return client
.post('url', formData, {
headers: {
...formData.getHeaders(),
},
})
.then((response) => response.data)
.catch((error) => {
return {
status: 'error',
error,
};
});
}
Related
What I'm trying to accomplish is using a Firebase Cloud Function (Node.js) to:
First download an image from an url (f.eg. from unsplash.com) using an axios.get() request
Secondly take that image and upload it to a Wordpress site using the Wordpress Rest API
The problem seems (to me) to be that the formData doesnt actually append any data, but the axios.get() request actually does indeed retrieve a buffered image it seems. Maybe its something wrong I'm doing with the Node.js library form-data or maybe I get the image in the wrong encoding? This is my best (but unsuccessfull) attempt:
async function uploadMediaToWordpress() {
var FormData = require("form-data");
var formData = new FormData();
var response = await axios.get(
"https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80"
{ responseType: "arraybuffer" }
);
formData.append("file", response.data);
try {
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
} catch (error) {
console.log(error);
throw new functions.https.HttpsError("failed-precondition", "WP media upload failed");
}
return uploadedMedia.data;
}
I have previously successfully uploaded an image to Wordpress with Javascript in a browser like this:
async function uploadMediaToWordpress() {
let formData = new FormData();
const response = await fetch("https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80");
const blob = await response.blob();
const file = new File([blob], "image.jpeg", { type: blob.type });
formData.append("file", file);
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
return uploadedMedia.data;
},
I have tried the last couple of days to get this to work but cannot for the life of me seem to get it right. Any pointer in the right direction would be greatly appreciated!
The "regular" JavaScript code (used in a browser) works because the image is sent as a file (see the new File in your code), but your Node.js code is not really doing that, e.g. the Content-Type value is wrong which should be multipart/form-data; boundary=----...... Nonetheless, instead of trying (hard) with the arraybuffer response, I suggest you to use stream just as in the axios documentation and form-data documentation.
So in your case, you'd want to:
Set stream as the responseType:
axios.get(
'https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80',
{ responseType: 'stream' }
)
Use formData.getHeaders() in the headers of your file upload request (to the /wp/v2/media endpoint):
axios.post( 'https://wordpresssite.com/wp-json/wp/v2/media', formData, {
headers: {
...formData.getHeaders(),
Authorization: 'Bearer ...'
},
} )
And because the remote image from Unsplash.com does not use a static name (e.g. image-name.jpg), then you'll need to set the name when you call formData.append():
formData.append( 'file', response.data, 'your-custom-image-name.jpeg' );
I hope that helps, which worked fine for me (using the node command for Node.js version 14.15.4, the latest release as of writing).
I have a code as shown below that uploads files from the browser and saves in the server, once it has been saved to the server, I want the server to connect to the Pinata API so the file can also be saved to the IPFS node.
let data = new FormData();
const fileBuffer = Buffer.from(`./public/files/${fileName}`, 'utf-8');
data.append('file', fileBuffer, `${fileName}`);
axios.post('https://api.pinata.cloud/pinning/pinJSONToIPFS',
data,
{
headers: {
'Content-Type': `multipart/form-data; boundary= ${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
console.log("FILE UPLOADED TO IPFS NODE", fileName);
console.log(response);
}).catch(function (error) {
console.log("FILE WASNT UPLOADED TO IPFS NODE", fileName);
console.log(error);
});
The issue i'm having is that after creating a buffer of my file and wrapping it in a formdata, the pinata API returns an error :
data: {
error: 'This API endpoint requires valid JSON, and a JSON content-type'
}
If i convert the data to string like JSON.stringify(data) and change the content-type to application/json, the file buffer will be uploaded successfully as string.
I hope explained it well to get a solution. Thanks.
It looks like you're attempting to upload a file to the pinJSONToIPFS endpoint, which is intended to purely be used for JSON that is passed in via a request body.
In your situation I would recommend using Pinata's pinFileToIPFS endpoint
Here's some example code based on their documentation that may be of help:
//imports needed for this function
const axios = require('axios');
const fs = require('fs');
const FormData = require('form-data');
export const pinFileToIPFS = (pinataApiKey, pinataSecretApiKey) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`;
//we gather a local file for this example, but any valid readStream source will work here.
let data = new FormData();
data.append('file', fs.createReadStream('./yourfile.png'));
return axios.post(url,
data,
{
maxContentLength: 'Infinity', //this is needed to prevent axios from erroring out with large files
headers: {
'Content-Type': `multipart/form-data; boundary=${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
//handle response here
}).catch(function (error) {
//handle error here
});
};
The proper code to pin any file to IPFS is as below.
Apparently, even Pinata support staff didn't know this.
You need to set an object with the property name filepath as your last parameter. The name doesn't matter, it can be a duplicate, it can be the same as others, or it can be unique.
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
const fileContents = Buffer.from(bytes);
const data = new FormData();
data.append("file", fileContents, {filepath: "anyname"});
const result = await axios
.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
"pinata_api_key": userApiKey,
"pinata_secret_api_key": userApiSecret,
"path": "somename"
}
});
Code to upload a file on IPFS using Pinata.
There are two methods available to upload files/images on Pinata. One is with Pinata SDK and the second is the pinFileToIPFS endpoint.
If you are uploading files from Next.js then you cannot convert your image into binary using fs.createReadStream or Buffer.from. These packages support the Node side. So if you want to upload the file with Next.js on Pinata then you can use this code.
// convert file into binary
const data = new FormData();
data.append("title", file.name);
data.append("file", file);
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
// pass binary data into post request
const result = await axios.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
pinata_api_key: "your_pinata_key",
pinata_secret_api_key:
"your_pinata_secret",
path: "somename",
},
});
console.log("RESULT", result);
this will upload a file to ipfs under the path ipfs://{cid}/images/{fileId}
const PINATA_BASE_URL = "https://api.pinata.cloud";
const PINATA_PIN_URI = "/pinning/pinFileToIPFS";
const fileExt = file.type.split("/")[1];
let nftId = 1
// creates a 64byte string '0000...0001' to follow ERC-1155 standard
const paddedId = createPaddedHex(nftId);
const ipfsFileId = `${paddedId}.${fileExt}`;
const ipfsImageFilePath = `/images/${ipfsFileId}`;
const fileUploadData = new FormData();
// this uploads the file and renames the uploaded file to the path created above
fileUploadData.append("file", file, ipfsImageFilePath);
fileUploadData.append(
"pinataOptions",
'{"cidVersion": 1, "wrapWithDirectory": true}'
);
fileUploadData.append(
"pinataMetadata",
`{"name": "${ipfsImageFilePath}", "keyvalues": {"company": "Pinata"}}`
);
const pinataUploadRes = await axios.post(
PINATA_BASE_URL + PINATA_PIN_URI,
fileUploadData,
{
headers: {
Authorization: `Bearer ${PINATA_JWT}`,
},
}
);
const ipfsCID = pinataUploadRes.data.IpfsHash;
i got a problem here.
Lets start from we have API, this API returns data with next headers:
Content-Type image/png
Transfer-Encoding chunked
Connection keep-alive
And response body where only file is.
When im try to write this body data into file - it's always broken. I mean i've binary data, his mime, body.length is match original filesize, but this image could not be opened in any viewer after i save it.
What i'm do:
public userFile(req, res: Response) {
const data = {
fileId: parseInt(req.body.fileId),
};
let params = {
headers: {
'Authorization': keys.token,
},
};
axios.post('/api/getfile/', data, params,)
.then((response: AxiosResponse) => {
const fs = require('fs');
const dir = require('path').resolve(__dirname + '../../../files/storage');
const ext = {
'image/png': '.png'
};
fs.writeFile(dir + '/' + img + ext[response.headers['content-type']], response.data, (er) => {
res.send(response.data);
});
})
.catch((err) => {
logger.error("AXIOS ERROR: ", err)
})
}
BUT! When i get this file with postman... here it is!
So, i need your help - what i do wrong?
You must specifically declare which response type it is in params as
responseType: 'stream'
then save stream to a file using pipe
response.data.pipe(fs.createWriteStream("/dir/xyz.png"))
I'm trying to send an image to my server using axios with react-native.
For doing this, I'm passing the image data (the base 64 encoded image data) directly to an uploadPicture function which uses axios this way:
const uploadPicture = async (data): Promise<AxiosResponse<string>> => {
const response = publicApi.post(
`${API_URL}/upload`,
{
image: data,
},
{
headers: { 'Content-Type': 'multipart/form-data' },
transformRequest: [transformToFormData],
}
);
return response;
};
const transformToFormData: AxiosTransformer = data => {
const formData = new FormData();
for (const key in data) {
formData.append(key, data[key]);
}
return formData;
};
The issue I face :
I get an internal error, like if my image was not correctly transmitted through my request.
If I'm doing the exact same request using Postman, it works fine, setting the body like this :
Which make me think that the issue doesn't come from my server but from my axios request.
Any idea of what I could be doing wrong ? Am I missing any axios option somewhere ?
I managed to find a solution:
I used the fetch function from javascript instead of axios
I send a file object instead of the data directly
I had to disable the react-native network inspect, otherwise the upload won't work
My working solution below, image being the response of react native image picker:
const file = {
uri: image.uri,
name: image.fileName,
type: image.type,
size: image.fileSize,
slice: () => new Blob(),
};
const body = new FormData();
body.append('image', file);
const response = await fetch(`${API_URL}/upload`, {
method: 'POST',
body,
});
I tried 400 combinations of syntaxes and headers, I can't figure out how to make a HTTP call from Angular to retrieve a file from my NodeJS server.
Found on Stackoverflow and tried, to no avail :
Download file from http post request - Angular 6
How download a file from HttpClient
Download a file from NodeJS Server using Express
How do I download a file with Angular2
It can't be a simple <a download> tag, or a public express.static() folder, because access to the file is restricted and I need to pass a JWT token along (in Node, I have an Express authentication middleware that will reject the request if no token is provided in the headers or if it is invalid).
The file is a GZIP : ./dumps/dump.gz and weighs 812 Kb.
I do manage to download the file, but whatever I try, it weighs 1.4 MB or 94 bytes (wrong size) and can't be opened (7zip can't open file downloads/dump.gz as archive).
What I have tried Angular-side (multiple attempts) :
import { saveAs } from 'file-saver';
let headers = new Headers({
"Authorization": "Bearer " + user.jwt, // I need this in the headers
"Content-Type" : "application/octet-stream", // Tried with and without, "application/gzip", "application/json", no difference
"responseType": "blob" as "json", // Tried with and without, "text", "json", no difference
"Access-Control-Expose-Headers" : "Content-Disposition" // Tried with and without, no difference
})
this.http
.get("/download/dump", { headers })
.toPromise()
.then(res => {
const blob = new Blob([res["_body"]] , { type: "application/octet-stream;"} ); // Error : body is not a blob or an array buffer
// const blob = new Blob([res["_body"]]); // Same result
// const blob = new Blob([res.blob()]); // Error : body is not a blob or an array buffer
saveAs(blob, "dump.gz"); // Saves a big corrupted file
// window.URL.createObjectURL(new Blob(blob, {type: 'blob'})); Saves a 94 byte corrupted file. Tried {type: 'gzip'}, same thing
})
.catch(err => console.error("download error = ", err))
What I have tried Node-side (multiple attempts) :
EDIT
Node has been innocented as I could retrieve the file directly from Chrome after disabling authentication. So, the back-end works and the issue is in Angular.
app.get( "/download/dump", authenticate, (req:Request, res:Response) => {
const file = path.resolve(__dirname, `./dumps/dump.gz`);
res
.set({ // Tried with and without headers, doesn't seem to do anything
"Content-Disposition" : "attachment", // Tried with and without
"filename" : "dump.gz", // Tried with and without
"filename*" : "dump.gz", // Tried with and without
"Content-Encoding" : "gzip", // Tried with and without
"Content-Type" : "application/gzip" // Tried with and without, "application/text", "application/json", no difference
})
.sendFile(file); // getting a big corrupted file
// .download(file); // Same result (big corrupted file)
})
Assuming that you are using the new HttpClient from angular (available since angular 4), this should work
front
import { saveAs } from 'file-saver';
import {HttpHeaders} from "#angular/common/http";
let headers = new HttpHeaders({
"Authorization": "Bearer " + user.jwt, // Auth header
//No other headers needed
});
this.http
.get("/download/dump", { headers, responseType: "blob" }) //set response Type properly (it is not part of headers)
.toPromise()
.then(blob => {
saveAs(blob, "dump.gz");
})
.catch(err => console.error("download error = ", err))
backend
app.get( "/download/dump", authenticate, (req:Request, res:Response) => {
const file = path.resolve(__dirname, `./dumps/dump.gz`);
//No need for special headers
res.download(file);
})