IPFS Pinata service not accepting file - node.js

I have a code as shown below that uploads files from the browser and saves in the server, once it has been saved to the server, I want the server to connect to the Pinata API so the file can also be saved to the IPFS node.
let data = new FormData();
const fileBuffer = Buffer.from(`./public/files/${fileName}`, 'utf-8');
data.append('file', fileBuffer, `${fileName}`);
axios.post('https://api.pinata.cloud/pinning/pinJSONToIPFS',
data,
{
headers: {
'Content-Type': `multipart/form-data; boundary= ${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
console.log("FILE UPLOADED TO IPFS NODE", fileName);
console.log(response);
}).catch(function (error) {
console.log("FILE WASNT UPLOADED TO IPFS NODE", fileName);
console.log(error);
});
The issue i'm having is that after creating a buffer of my file and wrapping it in a formdata, the pinata API returns an error :
data: {
error: 'This API endpoint requires valid JSON, and a JSON content-type'
}
If i convert the data to string like JSON.stringify(data) and change the content-type to application/json, the file buffer will be uploaded successfully as string.
I hope explained it well to get a solution. Thanks.

It looks like you're attempting to upload a file to the pinJSONToIPFS endpoint, which is intended to purely be used for JSON that is passed in via a request body.
In your situation I would recommend using Pinata's pinFileToIPFS endpoint
Here's some example code based on their documentation that may be of help:
//imports needed for this function
const axios = require('axios');
const fs = require('fs');
const FormData = require('form-data');
export const pinFileToIPFS = (pinataApiKey, pinataSecretApiKey) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`;
//we gather a local file for this example, but any valid readStream source will work here.
let data = new FormData();
data.append('file', fs.createReadStream('./yourfile.png'));
return axios.post(url,
data,
{
maxContentLength: 'Infinity', //this is needed to prevent axios from erroring out with large files
headers: {
'Content-Type': `multipart/form-data; boundary=${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
//handle response here
}).catch(function (error) {
//handle error here
});
};

The proper code to pin any file to IPFS is as below.
Apparently, even Pinata support staff didn't know this.
You need to set an object with the property name filepath as your last parameter. The name doesn't matter, it can be a duplicate, it can be the same as others, or it can be unique.
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
const fileContents = Buffer.from(bytes);
const data = new FormData();
data.append("file", fileContents, {filepath: "anyname"});
const result = await axios
.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
"pinata_api_key": userApiKey,
"pinata_secret_api_key": userApiSecret,
"path": "somename"
}
});

Code to upload a file on IPFS using Pinata.
There are two methods available to upload files/images on Pinata. One is with Pinata SDK and the second is the pinFileToIPFS endpoint.
If you are uploading files from Next.js then you cannot convert your image into binary using fs.createReadStream or Buffer.from. These packages support the Node side. So if you want to upload the file with Next.js on Pinata then you can use this code.
// convert file into binary
const data = new FormData();
data.append("title", file.name);
data.append("file", file);
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
// pass binary data into post request
const result = await axios.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
pinata_api_key: "your_pinata_key",
pinata_secret_api_key:
"your_pinata_secret",
path: "somename",
},
});
console.log("RESULT", result);

this will upload a file to ipfs under the path ipfs://{cid}/images/{fileId}
const PINATA_BASE_URL = "https://api.pinata.cloud";
const PINATA_PIN_URI = "/pinning/pinFileToIPFS";
const fileExt = file.type.split("/")[1];
let nftId = 1
// creates a 64byte string '0000...0001' to follow ERC-1155 standard
const paddedId = createPaddedHex(nftId);
const ipfsFileId = `${paddedId}.${fileExt}`;
const ipfsImageFilePath = `/images/${ipfsFileId}`;
const fileUploadData = new FormData();
// this uploads the file and renames the uploaded file to the path created above
fileUploadData.append("file", file, ipfsImageFilePath);
fileUploadData.append(
"pinataOptions",
'{"cidVersion": 1, "wrapWithDirectory": true}'
);
fileUploadData.append(
"pinataMetadata",
`{"name": "${ipfsImageFilePath}", "keyvalues": {"company": "Pinata"}}`
);
const pinataUploadRes = await axios.post(
PINATA_BASE_URL + PINATA_PIN_URI,
fileUploadData,
{
headers: {
Authorization: `Bearer ${PINATA_JWT}`,
},
}
);
const ipfsCID = pinataUploadRes.data.IpfsHash;

Related

Axios : send image with FormData on Prestashop API

I'm trying to upload on image on the Prestashop API with form-data/axios.
For that, i just need to send a post request with the images joined in an "image" parameter.
I did this simple node.js script (mine is much more complicated but i simplified for here):
const FormData = require("form-data");
const fs = require("fs");
const axios = require("axios");
// Read the image file into a buffer
const imageBuffer = fs.readFileSync("image.jpg");
// Create a FormData object
const form = new FormData();
// Append the image buffer to the form data
form.append("image", imageBuffer);
// Make an HTTP POST request to the PrestaShop API
axios({
method: "POST",
url: "https://XXX/api/images/products/15924",
data: form, // set the request body using the data field
params: {
ws_key: "XXX",
},
headers: {
"Content-Type": "multipart/form-data; boundary=" + form.getBoundary(),
},
})
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error(error.response.data);
});
But i get this answer:
<message><![CDATA[Please set an "image" parameter with image data for value]]></message>
I tried a LOT of things. With fs.createReadStream instead of formData, with http instead of axios, with a buffer or a file, etc ... and i alway end with this error.
I would be glad if someone has an idea :-)
Thx !
I finally succeed!
Two changed were needed:
Adding the filename in the append (thanks Dmitriy Mozgovoy for pointing that out in the comment)
Adding the size in the headers with "Content-Length": formData.getLengthSync()
Without these 2 missing elements, it seems that PHP received an empty $_FILES variable.
Final version:
const FormData = require("form-data");
const fs = require("fs");
const axios = require("axios");
// Read the image file into a buffer
const imageBuffer = fs.readFileSync("image.jpg");
// Create a FormData object
const form = new FormData();
// Append the image buffer to the form data
form.append("image", imageBuffer, "image.jpg");
// Make an HTTP POST request to the PrestaShop API
axios
.post("https://example.com/api/images/products/15924", form, {
params: {
ws_key: "EXAMPLE",
},
headers: {
...form.getHeaders(),
"Content-Length": form.getLengthSync(),
},
})
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error(error.response.data);
});

Downloading and Reposting an Image in Node.js

Use case: Download a short-term expiring profile image from URL, and re-post it to my own server for long-term hosting.
First, I download the image data to a string:
const request = require('request-promise');
const img_data = await request(soon_to_expire_url);
I've got a string with binary data now. Now what? Convert it to base64? to a blob? to a stream of some sort?
Next, I need to post the downloaded image to an API for long-term hosting, using axios. The problem is I can only get it work with fs.createReadStream, which requires a file path, not a data string.
const axios = require('axios');
const FormData = require('form-data');
var data = new FormData();
data.append('user_id', user_id);
data.append('profile_file', fs.createReadStream('non-existing-file.jpg'));
var config = {
method: 'POST',
url: 'serverEndpoint',
headers: {
'Content-Type': 'application/json',
...data.getHeaders()
},
data : data
};
return axios(config)
.then(function (response) {
console.log(JSON.stringify(response.data));
})
So how can bridge the gap? How can I create something out of the downloaded image that then takes the form of whatever createReadStream produces?
I've tried reducing it to a base64 string starting with data:image/jpeg;base64,..., but that doesn't take.
Last resort would be saving the image to a temp file, then posting that file path, but I would rather avoid that.
Either way you have to store the file somewhere anyway.
You can try something like this
let file = null;
async function decodeBase64(base64String){
fetch(base64String)
.then(res => res.blob())
.then(blob => {
file = new File([blob], "temp",{ type: "image/png" })
})
}
await decodeBase64(base64String)
var data = new FormData();
data.append('user_id', user_id);
data.append('profile_file', file));

Convert buffer to readStream

I have endpoint which is receiving files and need to send these files to 3rd party. However i'm not converting these files properly cuz i receive Invalid request format but if i read the file from fs the file is successfully deployed. How to create readStream from the input file? I tried Readable.from(file.buffer) still error. If i pass createReadStream(join(process.cwd(), 'images...')) is working as expected, here is the script.
export const deployFile = async (
file: Express.Multer.File,
httpService: HttpService,
): Promise<string> => {
const formData = new FormData();
// Working
// formData.append('file', createReadStream(join(process.cwd(), '/images/1.jpeg')));
// Not working
formData.append('file', Readable.from(file.buffer));
try {
const observable = httpService
.post(process.env.PINATA_BASE_URL + '/pinning/pinFileToIPFS', formData, {
headers: {
'Content-Type': `multipart/form-data; boundary=${formData.getBoundary()}`,
pinata_api_key: process.env.PINATA_API_KEY,
pinata_secret_api_key: process.env.PINATA_API_SECRET_KEY,
},
})
.pipe(map((response) => response.data));
const response = await lastValueFrom(observable);
return response.IpfsHash;
} catch (error) {
logger.error(`Error deploying image reason: ${error.response.data.error}`);
}
};

I cannot send base 64 url encoded data to my nodejs backend

I am trying to send post request to backend containing base64url encoded value of my image. When I send request with any random string, the request is received at backend but when I try to do same thing using encoded value, it responds me back with
request failed with status code 500
My code for the request is:
const uploadFileHandler = async (e) => {
const file = e.target.files[0];
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onloadend = () => {
setPreviewSource(reader.result);
uploadtobackend(reader.result);
};
const uploadtobackend = async (filedata) => {
try {
const config = {
headers: {
'Content-Type': 'application/json',
},
};
console.log(config);
console.log(filedata);
const { data } = await axios.post(
`/api/uploads`,
{
data: filedata,
},
config,
);
setImages(data);
setUploading(false);
} catch (error) {
console.log(error);
setUploading(false);
}
};
};
Here filedata is the encoded value and it is displayed if I console it.
Instead of
data: filedata
If I send
data: "some random string"
then request reaches backend otherwise not.
You nedd to send your file by wrapping in FormData.
You can do something like
const fd = new FormData()
// data you want to send other than file
fd.append('payload',JSON.stringify(payload))
fd.append('file',file) // file you want to send to node backend
Now in your backed you can get this file using req.file
Probably you need to use JSON.stringify() here are some example of use
const { data } = await axios.post(
`/api/uploads`,
{
data: new Buffer(JSON.stringify(filedata)).toString("base64")
},
config
)

Corrupted image on uploading image to AWS-S3 via signed url

I'm trying to upload images to aws-s3 via a signed-url from NodeJS server (not from a browser). The image to upload has been generated by NodeJS. I'm getting the signed-url from aws and succeeding to upload it to s3.
But my image is corrupted. For some reason, S3 is adding some headers to my image (compare image attached).
What am I doing wrong?
getting the signed url:
try {
var params = {
Bucket: bucketName,
Key: 'FILE_NAME.png',
Expires: 60
};
const url = await s3.getSignedUrlPromise('putObject', params);
return url;
} catch (err) {
throw err;
}
uploading to s3
var stats = fs.statSync(filePath);
var fileSizeInBytes = stats["size"];
const imageBuffer = fs.readFileSync(filePath);
var formData = {
'file': {
value: imageBuffer,
options: {
filename: 'FILE_NAME.png'
}
}
};
request({
method: 'put',
url,
headers: {
'Content-Length': fileSizeInBytes,
'Content-MD': md5(imageBuffer)
},
formData
}, function (err, res, body) {
console.log('body',body);
});
Compare between the actual image and the uploaded image to s3. S3 added some headers:
I know this is old but I struggled with the same issue for a while. When uploading using a pre-sgined url, DO NOT use new FormData();
One thing I noticed that all of my files on s3 were exactly 2kb larger than the originals.
<input type="file" id="upload"/>
var upload = document.getElementById('upload');
var file = upload.files[0];
//COMMENTED OUT BECAUSE IT WAS CAUSING THE ISSUE
//const formData = new FormData();
//formData.append("file", file);
// Assuming axios
const config = {
onUploadProgress: function(progressEvent) {
var percentCompleted = Math.round(
(progressEvent.loaded * 100) / progressEvent.total
);
console.log(percentCompleted);
},
header: {
'Content-Type': file.type
}
};
axios.put(S3SignedPutURL, file, config)
.then(async res => {
callback({res, key})
})
.catch(err => {
console.log(err);
})
I followed the above solution for react js
What I was doing before uploading an image is passing through the createObject URL and then passing it to the API body.
if (e.target.files && e.target.files[0]) {
let img = e.target.files[0];
**setImage(URL.createObjectURL(img))**
Correct Way:
if (e.target.files && e.target.files[0]) {
let img = e.target.files[0];
**setImage(img)**
Work For me, Thanks Sam Munroe
Came here in 2023, was facing the same problem using formdata, but in postman, before handing it to the front end department.
To handle it in postman, use the type of request body as binary:
And don't forget to add the proper headers.
Try to specify the content type in the request as Content-Type multipart/form-data.

Resources