I have endpoint which is receiving files and need to send these files to 3rd party. However i'm not converting these files properly cuz i receive Invalid request format but if i read the file from fs the file is successfully deployed. How to create readStream from the input file? I tried Readable.from(file.buffer) still error. If i pass createReadStream(join(process.cwd(), 'images...')) is working as expected, here is the script.
export const deployFile = async (
file: Express.Multer.File,
httpService: HttpService,
): Promise<string> => {
const formData = new FormData();
// Working
// formData.append('file', createReadStream(join(process.cwd(), '/images/1.jpeg')));
// Not working
formData.append('file', Readable.from(file.buffer));
try {
const observable = httpService
.post(process.env.PINATA_BASE_URL + '/pinning/pinFileToIPFS', formData, {
headers: {
'Content-Type': `multipart/form-data; boundary=${formData.getBoundary()}`,
pinata_api_key: process.env.PINATA_API_KEY,
pinata_secret_api_key: process.env.PINATA_API_SECRET_KEY,
},
})
.pipe(map((response) => response.data));
const response = await lastValueFrom(observable);
return response.IpfsHash;
} catch (error) {
logger.error(`Error deploying image reason: ${error.response.data.error}`);
}
};
Related
I have a requirement of uploading a pdf file to the a third-party API to do that what we are using a controller and then sending its buffer to the endpoint in multipart/form-data but It's not working. What am i doing wrong??
I have my controller defined like
#Post('/upload')
#UseInterceptors(FileInterceptor('file'))
async uploadDocument(#UploadedFile() file: Express.Multer.File){
await this.httpRequestService.uploadDocument(file)
return [
Translation.translate('Create_Success_Message', {
Entity: 'File'
}),
{
file
}
]
}
This controller calls a service called uploadDocument which looks like:
async uploadDocument(file){
try{
const formData = new FormData()
formData.append('file', file)
const response = await this.httpService.post(`urlWhereItHits`,
formData,
{
headers:{
"X-AppSecretToken":"appsecrettoken",
"X-AgreementGrantToken":"agreementgranttoken",
'Content-Type' : 'multipart/form-data'
},
}).toPromise();
}
catch(error){
console.log(error)
}
}
Now i don't understand what's happening wrong here.
I get error saying
TypeError: source.on is not a function at Function.DelayedStream.create (C:\Users\Tushar\Squire-backend\node_modules\delayed-stream\lib\delayed_stream.js:33:10)
I had a similar task, receiving a .csv file and sending it to another server.
controller:
#UseInterceptors(FileInterceptor('file'))
#Post('upload')
async uploadFile(#UploadedFile() file: Express.Multer.File) {
await this.myService.uploadFile(file);
return {
message: 'File uploaded successfully.',
};
}
service:
import * as FormData from 'form-data';
import { Readable } from 'stream';
async uploadFile(file: Express.Multer.File) {
const form = new FormData();
form.append('files', Readable.from(file.buffer), {
filename: file.originalname,
});
try {
const result = await lastValueFrom(
this.httpService.post('The URL of your third-party service', form, {
headers: {
...form.getHeaders(),
},
}),
);
if (result.status === 200) return;
} catch {
throw new InternalServerErrorException(
'Error while uploading the .csv file.',
);
}
}
Some explanations here:
It requires importing Formdata from the package.
.append() requires a fieldName, and a readable Stream.
You can use the buffer of the file that's on memory (not recommended with large files) and use it to create a readable stream.
Also, you need to add the name of the file manually since Formdata only recognizes the filename if the file is saved on disk. (Which in our case is on memory as a Buffer).
You can create the stream in different ways, the example is not mandatory.
.getHeaders() will generate the default header 'Content-Type': 'multipart/form-data' and also it will generate the boundary of the file, like this: reference
{
'content-type': 'multipart/form-data; boundary=--------------------------286308325485491440714225'
}
I have a code as shown below that uploads files from the browser and saves in the server, once it has been saved to the server, I want the server to connect to the Pinata API so the file can also be saved to the IPFS node.
let data = new FormData();
const fileBuffer = Buffer.from(`./public/files/${fileName}`, 'utf-8');
data.append('file', fileBuffer, `${fileName}`);
axios.post('https://api.pinata.cloud/pinning/pinJSONToIPFS',
data,
{
headers: {
'Content-Type': `multipart/form-data; boundary= ${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
console.log("FILE UPLOADED TO IPFS NODE", fileName);
console.log(response);
}).catch(function (error) {
console.log("FILE WASNT UPLOADED TO IPFS NODE", fileName);
console.log(error);
});
The issue i'm having is that after creating a buffer of my file and wrapping it in a formdata, the pinata API returns an error :
data: {
error: 'This API endpoint requires valid JSON, and a JSON content-type'
}
If i convert the data to string like JSON.stringify(data) and change the content-type to application/json, the file buffer will be uploaded successfully as string.
I hope explained it well to get a solution. Thanks.
It looks like you're attempting to upload a file to the pinJSONToIPFS endpoint, which is intended to purely be used for JSON that is passed in via a request body.
In your situation I would recommend using Pinata's pinFileToIPFS endpoint
Here's some example code based on their documentation that may be of help:
//imports needed for this function
const axios = require('axios');
const fs = require('fs');
const FormData = require('form-data');
export const pinFileToIPFS = (pinataApiKey, pinataSecretApiKey) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`;
//we gather a local file for this example, but any valid readStream source will work here.
let data = new FormData();
data.append('file', fs.createReadStream('./yourfile.png'));
return axios.post(url,
data,
{
maxContentLength: 'Infinity', //this is needed to prevent axios from erroring out with large files
headers: {
'Content-Type': `multipart/form-data; boundary=${data._boundary}`,
'pinata_api_key': pinataApiKey,
'pinata_secret_api_key': pinataSecretApiKey
}
}
).then(function (response) {
//handle response here
}).catch(function (error) {
//handle error here
});
};
The proper code to pin any file to IPFS is as below.
Apparently, even Pinata support staff didn't know this.
You need to set an object with the property name filepath as your last parameter. The name doesn't matter, it can be a duplicate, it can be the same as others, or it can be unique.
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
const fileContents = Buffer.from(bytes);
const data = new FormData();
data.append("file", fileContents, {filepath: "anyname"});
const result = await axios
.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
"pinata_api_key": userApiKey,
"pinata_secret_api_key": userApiSecret,
"path": "somename"
}
});
Code to upload a file on IPFS using Pinata.
There are two methods available to upload files/images on Pinata. One is with Pinata SDK and the second is the pinFileToIPFS endpoint.
If you are uploading files from Next.js then you cannot convert your image into binary using fs.createReadStream or Buffer.from. These packages support the Node side. So if you want to upload the file with Next.js on Pinata then you can use this code.
// convert file into binary
const data = new FormData();
data.append("title", file.name);
data.append("file", file);
const url = "https://api.pinata.cloud/pinning/pinFileToIPFS";
// pass binary data into post request
const result = await axios.post(url, data, {
maxContentLength: -1,
headers: {
"Content-Type": `multipart/form-data; boundary=${data._boundary}`,
pinata_api_key: "your_pinata_key",
pinata_secret_api_key:
"your_pinata_secret",
path: "somename",
},
});
console.log("RESULT", result);
this will upload a file to ipfs under the path ipfs://{cid}/images/{fileId}
const PINATA_BASE_URL = "https://api.pinata.cloud";
const PINATA_PIN_URI = "/pinning/pinFileToIPFS";
const fileExt = file.type.split("/")[1];
let nftId = 1
// creates a 64byte string '0000...0001' to follow ERC-1155 standard
const paddedId = createPaddedHex(nftId);
const ipfsFileId = `${paddedId}.${fileExt}`;
const ipfsImageFilePath = `/images/${ipfsFileId}`;
const fileUploadData = new FormData();
// this uploads the file and renames the uploaded file to the path created above
fileUploadData.append("file", file, ipfsImageFilePath);
fileUploadData.append(
"pinataOptions",
'{"cidVersion": 1, "wrapWithDirectory": true}'
);
fileUploadData.append(
"pinataMetadata",
`{"name": "${ipfsImageFilePath}", "keyvalues": {"company": "Pinata"}}`
);
const pinataUploadRes = await axios.post(
PINATA_BASE_URL + PINATA_PIN_URI,
fileUploadData,
{
headers: {
Authorization: `Bearer ${PINATA_JWT}`,
},
}
);
const ipfsCID = pinataUploadRes.data.IpfsHash;
I am trying to send post request to backend containing base64url encoded value of my image. When I send request with any random string, the request is received at backend but when I try to do same thing using encoded value, it responds me back with
request failed with status code 500
My code for the request is:
const uploadFileHandler = async (e) => {
const file = e.target.files[0];
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onloadend = () => {
setPreviewSource(reader.result);
uploadtobackend(reader.result);
};
const uploadtobackend = async (filedata) => {
try {
const config = {
headers: {
'Content-Type': 'application/json',
},
};
console.log(config);
console.log(filedata);
const { data } = await axios.post(
`/api/uploads`,
{
data: filedata,
},
config,
);
setImages(data);
setUploading(false);
} catch (error) {
console.log(error);
setUploading(false);
}
};
};
Here filedata is the encoded value and it is displayed if I console it.
Instead of
data: filedata
If I send
data: "some random string"
then request reaches backend otherwise not.
You nedd to send your file by wrapping in FormData.
You can do something like
const fd = new FormData()
// data you want to send other than file
fd.append('payload',JSON.stringify(payload))
fd.append('file',file) // file you want to send to node backend
Now in your backed you can get this file using req.file
Probably you need to use JSON.stringify() here are some example of use
const { data } = await axios.post(
`/api/uploads`,
{
data: new Buffer(JSON.stringify(filedata)).toString("base64")
},
config
)
I am uploading a video file to my s3 bucket using a presigned URL using the following code
function postsignedURL(req) {
return new Promise(function (resolve, reject) {
const params = {
Bucket: 'bucket1',
Expires: 60 * 60, // in seconds,
Fields: {
key: req,
'Content-Type': 'video/'
},
Conditions: [
['content-length-range', 300, 4000e+7],
{ 'Content-Type': 'video/' },
["starts-with", "$Content-Type", "video/"]
]
}
s3.createPresignedPost(params, (err, data) => {
if (err) {
reject(err)
} else {
resolve(data);
}
})
})
}
I am able to upload the file correctly and I can even download it using the following code in
const video = await readObjectSignedUrl(key)
console.log(video, " video ");
This one returns a presigned url. I can download the video using fetch method below
const objectResult = await fetchS3Object(video);
const bufferType = Buffer.from(objectResult);
const fileTypeResult = await fileType.fromBuffer(bufferTy[e);
console.log(fileTypeResult )
The result of fileTypeResult is the following which is correct
{ ext: 'mov', mime: 'video/quicktime' }
The code for the fetchS3Object is the following
async function fetchS3Object(key) {
console.log(key," KEY ")
const result = await fetch(key, {
method: 'GET',
mode: "cors",
headers: {
'Access-Control-Allow-Origin': '*',
}
})
const response = await result.text()
return response
}
When i print the objectResult variable it is some gibberish.
I am trying to write the objectResult to file using writeFileSync method.
fs.writeFileSync('/tmp/file', objectResult);
The file is written correctly in the specified location, but the video file is corrupted. When I try to play it, it is not working.
Is there is a reason why the written file is corrupted
I have Managed to solve the problem. Here is the issue. The problem arise from the fact that the original uploaded file which is a video had an encoding of
windows-1252,
However the fetch method return a binary which is encoded with
UTF-8.
As a result when you try to use writeFileSync the output is a corrupted file.
To solve this problem this what has to be done. The fetchS3Object function has to be changed to the following
async function fetchS3Object(key) {
console.log(key," KEY ")
const result = await fetch(key, {
method: 'GET',
mode: "cors",
headers: {
'Access-Control-Allow-Origin': '*',
}
})
const response = await result.arrayBuffer();
return response
}
then
const objectResult = await fetchS3Object(video);
const bufferType = Buffer.from(objectResult);
fs.writeFileSync('/tmp/file', bufferType);
I am trying to send a FormData containing a pdf file from a node.js script to a server. The request fail with a status code 400 in response. In the server side, it seems that he do not consider the data as a pdf file, or maybe that that the whole data is undefined.
This is what my node.js script does : (I've tried other ways to send the request but without success)
import axios from "axios"
import fs from "fs-extra"
import FormData from 'form-data'
const formData = new FormData()
const diplomaBuffer = await fs.readFile(`../data/fake/foo`+i+`.pdf`)
formData.append("diploma", diplomaBuffer)
formData.append("diplomaId", 1)
axios({
method: 'post',
processData: false,
contentType: 'multipart/form-data',
cache: false,
url: 'http://localhost:8080/certificates',
data: formData,
config: { headers: formData.getHeaders() }
})
// const response = await axios
// .post(`http://localhost:8080/certificates`, {
// body: formData
// })
// const response = await axios
// .post(`http://localhost:8080/certificates`
// , formData
// , {headers: formData.getHeaders()})
This is the function called in the server side :
app.post('/certificates', async (req, res) => {
const files = req.files
if(!files || !files.diploma || !files.diplomaId) {
res.status(400).send(new Error("No file provided"))
}
if(files.diploma.mimetype !== "application/pdf") {
res.status(400).send(new Error("diploma is not a pdf"))
}
// operations made with the received pdf (not relevant for the question)
const certificateId = uniqid()
const outputDiploma = `data/diplomas/${certificateId}.pdf`
await Promise.all([
fs.writeFile(outputDiploma, files.diploma.data),
])
await pdf.sign(outputDiploma, `${sha("sha256").update(certificateId + diplomaSecret).digest("hex")} : ${date()}`)
const diplomaBuffer = await fs.readFile(outputDiploma)
const certificate_hash = verify_pdf.hashBuffer(diplomaBuffer)
const certificateRegistry = await app.bizNetworkConnection.getAssetRegistry("consortium.supchain.assets.Certificate")
const certificate = app.factory.newResource("consortium.supchain.assets", "Certificate", certificateId)
certificate.diploma = app.factory.newRelationship("consortium.supchain.assets", "Diploma", req.body.diplomaId)
certificate.hashInfo = certificate_hash
await app.registry.certificate.add(certificate)
res.status(200).send("ok")
})