Downloading and Reposting an Image in Node.js - node.js

Use case: Download a short-term expiring profile image from URL, and re-post it to my own server for long-term hosting.
First, I download the image data to a string:
const request = require('request-promise');
const img_data = await request(soon_to_expire_url);
I've got a string with binary data now. Now what? Convert it to base64? to a blob? to a stream of some sort?
Next, I need to post the downloaded image to an API for long-term hosting, using axios. The problem is I can only get it work with fs.createReadStream, which requires a file path, not a data string.
const axios = require('axios');
const FormData = require('form-data');
var data = new FormData();
data.append('user_id', user_id);
data.append('profile_file', fs.createReadStream('non-existing-file.jpg'));
var config = {
method: 'POST',
url: 'serverEndpoint',
headers: {
'Content-Type': 'application/json',
...data.getHeaders()
},
data : data
};
return axios(config)
.then(function (response) {
console.log(JSON.stringify(response.data));
})
So how can bridge the gap? How can I create something out of the downloaded image that then takes the form of whatever createReadStream produces?
I've tried reducing it to a base64 string starting with data:image/jpeg;base64,..., but that doesn't take.
Last resort would be saving the image to a temp file, then posting that file path, but I would rather avoid that.

Either way you have to store the file somewhere anyway.
You can try something like this
let file = null;
async function decodeBase64(base64String){
fetch(base64String)
.then(res => res.blob())
.then(blob => {
file = new File([blob], "temp",{ type: "image/png" })
})
}
await decodeBase64(base64String)
var data = new FormData();
data.append('user_id', user_id);
data.append('profile_file', file));

Related

Axios : send image with FormData on Prestashop API

I'm trying to upload on image on the Prestashop API with form-data/axios.
For that, i just need to send a post request with the images joined in an "image" parameter.
I did this simple node.js script (mine is much more complicated but i simplified for here):
const FormData = require("form-data");
const fs = require("fs");
const axios = require("axios");
// Read the image file into a buffer
const imageBuffer = fs.readFileSync("image.jpg");
// Create a FormData object
const form = new FormData();
// Append the image buffer to the form data
form.append("image", imageBuffer);
// Make an HTTP POST request to the PrestaShop API
axios({
method: "POST",
url: "https://XXX/api/images/products/15924",
data: form, // set the request body using the data field
params: {
ws_key: "XXX",
},
headers: {
"Content-Type": "multipart/form-data; boundary=" + form.getBoundary(),
},
})
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error(error.response.data);
});
But i get this answer:
<message><![CDATA[Please set an "image" parameter with image data for value]]></message>
I tried a LOT of things. With fs.createReadStream instead of formData, with http instead of axios, with a buffer or a file, etc ... and i alway end with this error.
I would be glad if someone has an idea :-)
Thx !
I finally succeed!
Two changed were needed:
Adding the filename in the append (thanks Dmitriy Mozgovoy for pointing that out in the comment)
Adding the size in the headers with "Content-Length": formData.getLengthSync()
Without these 2 missing elements, it seems that PHP received an empty $_FILES variable.
Final version:
const FormData = require("form-data");
const fs = require("fs");
const axios = require("axios");
// Read the image file into a buffer
const imageBuffer = fs.readFileSync("image.jpg");
// Create a FormData object
const form = new FormData();
// Append the image buffer to the form data
form.append("image", imageBuffer, "image.jpg");
// Make an HTTP POST request to the PrestaShop API
axios
.post("https://example.com/api/images/products/15924", form, {
params: {
ws_key: "EXAMPLE",
},
headers: {
...form.getHeaders(),
"Content-Length": form.getLengthSync(),
},
})
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error(error.response.data);
});

Save images in S3 from React SPA with express backend

I have a React component where I ask the user to insert an image using react-dropze. On drop, I save the image into an image state.
Like this:
const handleOnDrop = (files) => {
setimage(files[0]);
}
Once I submit, I send a request to my back-end in order to get the URL with this function:
export const generateUploadURL = async () => {
const rawBytes = await randomBytes(16);
const imageName = rawBytes.toString('hex');
const params = ({
Bucket: process.env.S3_BUCKET_NAME,
Key: imageName,
ContentType: 'image/*',
Expires: 60
})
const uploadUrl = await s3.getSignedUrlPromise('putObject', params);
return uploadUrl;
}
I get the URL and finally execute a put into the s3 with the URL:
await axios.put(url, {
headers: {
"Content-Type": "multipart/form-data"
},
body: image
});
And then I save the data into my database but that's not important.
The thing is, after that, I can't render the image from the link I stored so I went into the link and encountered this:
{"headers":{"Content-Type":"multipart/form-data"},"body":{"path":"asdasdsadtest.jpg"}}
I tried setting the content-type to the imageType but that didn't work either. I have no clue on how I could make it work.
Why are you using the s3.getSignedUrlPromise('putObject ") api? Using s3.upload would allow you to send the file in one go and would make it much simpler in my opinion. See: https://stackabuse.com/uploading-files-to-aws-s3-with-node-js/ for an example of this solution
Solved it the following way:
url = await getS3Url(image);
await axios.put(url, image, {
headers: {
"Content-Type": image.type
}
});
Insted of placing it into the body, this worked.

Get image from Axios and send as Form Data to Wordpress API in a Cloud Function

What I'm trying to accomplish is using a Firebase Cloud Function (Node.js) to:
First download an image from an url (f.eg. from unsplash.com) using an axios.get() request
Secondly take that image and upload it to a Wordpress site using the Wordpress Rest API
The problem seems (to me) to be that the formData doesnt actually append any data, but the axios.get() request actually does indeed retrieve a buffered image it seems. Maybe its something wrong I'm doing with the Node.js library form-data or maybe I get the image in the wrong encoding? This is my best (but unsuccessfull) attempt:
async function uploadMediaToWordpress() {
var FormData = require("form-data");
var formData = new FormData();
var response = await axios.get(
"https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80"
{ responseType: "arraybuffer" }
);
formData.append("file", response.data);
try {
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
} catch (error) {
console.log(error);
throw new functions.https.HttpsError("failed-precondition", "WP media upload failed");
}
return uploadedMedia.data;
}
I have previously successfully uploaded an image to Wordpress with Javascript in a browser like this:
async function uploadMediaToWordpress() {
let formData = new FormData();
const response = await fetch("https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80");
const blob = await response.blob();
const file = new File([blob], "image.jpeg", { type: blob.type });
formData.append("file", file);
var uploadedMedia = await axios.post("https://wordpresssite.com/wp-json/wp/v2/media",
formData, {
headers: {
"Content-Disposition": 'form-data; filename="example.jpeg"',
"Content-Type": "image/jpeg",
Authorization: "Bearer <jwt_token>",
},
});
return uploadedMedia.data;
},
I have tried the last couple of days to get this to work but cannot for the life of me seem to get it right. Any pointer in the right direction would be greatly appreciated!
The "regular" JavaScript code (used in a browser) works because the image is sent as a file (see the new File in your code), but your Node.js code is not really doing that, e.g. the Content-Type value is wrong which should be multipart/form-data; boundary=----...... Nonetheless, instead of trying (hard) with the arraybuffer response, I suggest you to use stream just as in the axios documentation and form-data documentation.
So in your case, you'd want to:
Set stream as the responseType:
axios.get(
'https://images.unsplash.com/photo-1610303785445-41db41838e3e?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=634&q=80',
{ responseType: 'stream' }
)
Use formData.getHeaders() in the headers of your file upload request (to the /wp/v2/media endpoint):
axios.post( 'https://wordpresssite.com/wp-json/wp/v2/media', formData, {
headers: {
...formData.getHeaders(),
Authorization: 'Bearer ...'
},
} )
And because the remote image from Unsplash.com does not use a static name (e.g. image-name.jpg), then you'll need to set the name when you call formData.append():
formData.append( 'file', response.data, 'your-custom-image-name.jpeg' );
I hope that helps, which worked fine for me (using the node command for Node.js version 14.15.4, the latest release as of writing).

IPFS Pinata service not accepting file

I have a code as shown below that uploads files from the browser and saves in the server, once it has been saved to the server, I want the server to connect to the Pinata API so the file can also be saved to the IPFS node.
let data = new FormData();
const fileBuffer = Buffer.from(`./public/files/${fileName}`, 'utf-8');
data.append('file', fileBuffer, `${fileName}`);
axios.post('https://api.pinata.cloud/pinning/pinJSONToIPFS',
data,
{
headers: {
'Content-Type': `multipart/form-data; boundary= ${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
console.log("FILE UPLOADED TO IPFS NODE", fileName);
console.log(response);
}).catch(function (error) {
console.log("FILE WASNT UPLOADED TO IPFS NODE", fileName);
console.log(error);
});
The issue i'm having is that after creating a buffer of my file and wrapping it in a formdata, the pinata API returns an error :
data: {
error: 'This API endpoint requires valid JSON, and a JSON content-type'
}
If i convert the data to string like JSON.stringify(data) and change the content-type to application/json, the file buffer will be uploaded successfully as string.
I hope explained it well to get a solution. Thanks.
It looks like you're attempting to upload a file to the pinJSONToIPFS endpoint, which is intended to purely be used for JSON that is passed in via a request body.
In your situation I would recommend using Pinata's pinFileToIPFS endpoint
Here's some example code based on their documentation that may be of help:
//imports needed for this function
const axios = require('axios');
const fs = require('fs');
const FormData = require('form-data');
export const pinFileToIPFS = (pinataApiKey, pinataSecretApiKey) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`;
//we gather a local file for this example, but any valid readStream source will work here.
let data = new FormData();
data.append('file', fs.createReadStream('./yourfile.png'));
return axios.post(url,
data,
{
maxContentLength: 'Infinity', //this is needed to prevent axios from erroring out with large files
headers: {
'Content-Type': `multipart/form-data; boundary=${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
//handle response here
}).catch(function (error) {
//handle error here
});
};
The proper code to pin any file to IPFS is as below.
Apparently, even Pinata support staff didn't know this.
You need to set an object with the property name filepath as your last parameter. The name doesn't matter, it can be a duplicate, it can be the same as others, or it can be unique.
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
const fileContents = Buffer.from(bytes);
const data = new FormData();
data.append("file", fileContents, {filepath: "anyname"});
const result = await axios
.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
"pinata_api_key": userApiKey,
"pinata_secret_api_key": userApiSecret,
"path": "somename"
}
});
Code to upload a file on IPFS using Pinata.
There are two methods available to upload files/images on Pinata. One is with Pinata SDK and the second is the pinFileToIPFS endpoint.
If you are uploading files from Next.js then you cannot convert your image into binary using fs.createReadStream or Buffer.from. These packages support the Node side. So if you want to upload the file with Next.js on Pinata then you can use this code.
// convert file into binary
const data = new FormData();
data.append("title", file.name);
data.append("file", file);
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
// pass binary data into post request
const result = await axios.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
pinata_api_key: "your_pinata_key",
pinata_secret_api_key:
"your_pinata_secret",
path: "somename",
},
});
console.log("RESULT", result);
this will upload a file to ipfs under the path ipfs://{cid}/images/{fileId}
const PINATA_BASE_URL = "https://api.pinata.cloud";
const PINATA_PIN_URI = "/pinning/pinFileToIPFS";
const fileExt = file.type.split("/")[1];
let nftId = 1
// creates a 64byte string '0000...0001' to follow ERC-1155 standard
const paddedId = createPaddedHex(nftId);
const ipfsFileId = `${paddedId}.${fileExt}`;
const ipfsImageFilePath = `/images/${ipfsFileId}`;
const fileUploadData = new FormData();
// this uploads the file and renames the uploaded file to the path created above
fileUploadData.append("file", file, ipfsImageFilePath);
fileUploadData.append(
"pinataOptions",
'{"cidVersion": 1, "wrapWithDirectory": true}'
);
fileUploadData.append(
"pinataMetadata",
`{"name": "${ipfsImageFilePath}", "keyvalues": {"company": "Pinata"}}`
);
const pinataUploadRes = await axios.post(
PINATA_BASE_URL + PINATA_PIN_URI,
fileUploadData,
{
headers: {
Authorization: `Bearer ${PINATA_JWT}`,
},
}
);
const ipfsCID = pinataUploadRes.data.IpfsHash;

Issue uploading an image using axios

I'm trying to send an image to my server using axios with react-native.
For doing this, I'm passing the image data (the base 64 encoded image data) directly to an uploadPicture function which uses axios this way:
const uploadPicture = async (data): Promise<AxiosResponse<string>> => {
const response = publicApi.post(
`${API_URL}/upload`,
{
image: data,
},
{
headers: { 'Content-Type': 'multipart/form-data' },
transformRequest: [transformToFormData],
}
);
return response;
};
const transformToFormData: AxiosTransformer = data => {
const formData = new FormData();
for (const key in data) {
formData.append(key, data[key]);
}
return formData;
};
The issue I face :
I get an internal error, like if my image was not correctly transmitted through my request.
If I'm doing the exact same request using Postman, it works fine, setting the body like this :
Which make me think that the issue doesn't come from my server but from my axios request.
Any idea of what I could be doing wrong ? Am I missing any axios option somewhere ?
I managed to find a solution:
I used the fetch function from javascript instead of axios
I send a file object instead of the data directly
I had to disable the react-native network inspect, otherwise the upload won't work
My working solution below, image being the response of react native image picker:
const file = {
uri: image.uri,
name: image.fileName,
type: image.type,
size: image.fileSize,
slice: () => new Blob(),
};
const body = new FormData();
body.append('image', file);
const response = await fetch(`${API_URL}/upload`, {
method: 'POST',
body,
});

Resources