I'm successfully converting and sending an image to base64 in my React frontend.
This is received in 'req.body.imageString'. When I receive it, I'm then removing the unnecessary text at the front of the base64 string, using the .pop() method.
However, when I'm then attempting to write the file to my filesystem to be saved and queried later, it isn't saving anything despite there being no error, and it successfully console logging.
Here's my serverless function I'm using:
module.exports = async (req, res) => {
let base64Image = req.body.imageString
let finalImageString = base64Image.split(';base64,').pop()
fs.writeFile('assets/profilePictures/', finalImageString, { encoding: 'base64'}, function(err) {
console.log("File successfully written.")
})
res.send(200)
}
What I'd like to do is save the file into my 'assets' folder in my React app. So this would be the basic high level structure:
/root
// api
/// myServerlessFunction.js
// public
/// assets
Any tips would be appreciated!
Related
I have a Node.js (16.13.1) REST API using Express and one of my endpoints receives one or more uploaded files. The client (web app) uses FormData into which the files are appended. Once they're submitted to my API, the code there uses multer to grab the files from the request object.
Now I'm having trouble trying to send those same files to another API. multer attaches the files to req.files and each file object in that array has several properties one of which is buffer. I tried using the stream package's Duplex object to convert this buffer to a stream so that I could append the file to another FormData object, but when the server the second API is running on receives the request, I get an error from the web server saying that "a potentially dangerous request.form value was detected from the client.".
Any suggestions?
I am working on a nest project I was also facing this issue did some research and found that we need to create a Readable from the Buffer of that file and it's working for me.
// Controller
#UseInterceptors(FileInterceptor('file'))
async uploadFile(#UploadedFile() file: Express.Multer.File) {
return this.apiservice.upload(file);
}
// Service
uploadFile(file: Express.Multer.File) {
const readstream = Readable.from(file.buffer)
console.log(readstream)
const form = new FormData();
form.append('file', file, { filename: extra.filename });
const url = `api_endpoint`;
const config: AxiosRequestConfig = {
headers: {
'Content-Type': 'multipart/form-data'
},
};
return axios.post(url, form, config);
}
Im receiving filedata from an API call from an external service.
I want to save this filedata to my MongoDB. I was met with the error that the files are too large.
I went to research GridFS as an extra collection in my MongoDB.
I really cant find anything that solves my issuse. Ive tried to use multer to upload the data like this:
async function addFileDataToDB(fileData) {
const storage = new GridFsStorage({
url: mongoose.connection,
file: (req, file) => {
console.log(file.mimetype)
if (file.mimetype === 'application/pdf') {
return {
bucketName: 'fileBucket'
};
} else {
return null;
}
}
});
const upload = multer({ storage });
upload(fileData)
console.log('YAY! : - )')
}
Doesnt seem like something i can use. If i understand it correctly i cant use multer to transfer the data received by the endpoint to MongoDB. Multer seems more like something you would use to upload files from a form etc.
Im looking for any kind of help to point me in the right dirrection to upload this file data from the endpoint to a collection in mongoDB.
To clearify the file data is in the format of a buffer containing bytes, and im trying to do this in nodejs/express
Im new to GridFS, keep that in mind.
I have a function within my app that I have working that lets me submit a form with images that then stores the image in a file using Multer and uploads a URL to my Postgres database. When I return the file to the front end I am just left with filename that relates to the images but I don't know how to get the file path added so that the image displaying.
Should I be adding the prefix file path when inserting it into the database? Or is there a security issue if I display the full file path on the front-end. Obviously I know my front and backends should be decoupled and operating independently. I could also have a separate file for images outside of the backend but I am not sure if this is a recommended process. If this was in a professional environment would It just be handled by the likes of Google Cloud, AWS etc so I'm not sure if following the decoupling process is completely possible in this case.
I have seen process-cwd in some similar cases but I'm not sure if this is what I need to do. Could I hard code the file path in the front-end React component and then use the redux data that has the filename at the end?
I have the photo filename stored in my Redux store however I don't know to go from there. The image is in my backend/assets file at the minute.
API inserting the image into Postgres.
exports.createDiveSpot = async (req, res) => {
const fileNameWithExtension = `${req.file.filename}-${req.file.originalname}`
const newPath = `./assets/diveSpot/${fileNameWithExtension}`
fs.rename(req.file.path, newPath, function (err) {
if (err) {
console.log(err)
res.send(500)
}
console.log(req.body)
diveSpot.create({
diveLocation: req.body.diveLocation,
diveRegionID: req.body.diveRegionID,
diveSpotTypeID: req.body.diveSpotTypeID,
diveSpotDescription: req.body.diveSpotDescription,
photos: fileNameWithExtension,
}).then((data) => {
res.send(data)
})
Update
In the below method. I notice that I can enter a file name easily in the "const fileNameWithExtension" line. Can I enter a url for the public file on my front-end or should I make a file in my backend public with express. Should I be putting a file from my backend into my front-end again as it is technically already being submitted from the front-end then multer would move it to the front?
I have included my latest error message below.
exports.createArticle = async (req, res) => {
console.log(req.body)
const fileNameWithExtension = "../home/backend/assets/article/"`${req.file.filename}-${req.file.originalname}`
const newPath = `./assets/article/${fileNameWithExtension}`
console.log(req.body)
fs.rename(req.file.path, newPath, function (err) {
if (err) {
console.log(err)
res.send(500)
}
article.create({
articleTitle: req.body.articleTitle,
articleContent: req.body.articleContent,
userID: req.body.userID,
articleTypeID: req.body.articleTypeID,
photos: fileNameWithExtension,
}).then((data) => {
res.send(data)
})
.catch((err) => {
res.status(500).send({
message: err.message || 'Some error occurred while creating the post.',
})
})
}
)}
error message
[Error: ENOENT: no such file or directory, rename 'C:\Users\James Greene\WebstormProjects\softwaredevproject\SustainableScuba\backend\assets\article\678c78
193bb73ab287bbb6b644a0c86e' -> 'C:\Users\James Greene\WebstormProjects\softwaredevproject\WebApp\sustainable-scuba-web-app\public\article\678c78193bb73ab28
7bbb6b644a0c86e-sharkfeat.jpg'] {
errno: -4058,
code: 'ENOENT',
syscall: 'rename',
path: '...\\SustainableScuba\\backend\\assets\\article\\678c78193bb73ab287bbb6b644a0c86e',
dest: '...\\sustainable-scuba-web-app\\public\\article\\678c78193bb73ab287bbb6b644a0c86
e-sharkfeat.jpg'
}
express deprecated res.send(status): Use res.sendStatus(status) instead controllers\article.controller.js:15:21
Executing (default): INSERT INTO "articles" ("articleID","articleTitle","articleContent","photos","userID","articleTypeID") VALUES (DEFAULT,$1,$2,$3,$4,$5)
RETURNING *;
Unhandled rejection Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:485:11)
at ServerResponse.header (...\backend\node_modules\express\lib\response.js:771:1
0)
It is being rendered on the front-end in Redux and is in the store. I am trying to display the image on the card component below.
<IconButton aria-label="settings">
<MoreVertIcon />
</IconButton>
}
title={articleList.articleTitle}
subheader={userID}/>
<CardMedia
id="Photos"
className={classes.media}
image={articleList.photos}
title="Dive Photos"/>
<CardContent>
<Typography className="Type" id="Type" variant="body2" color="textSecondary" component="p">
{articleType}
It is bad security practice to include full paths of your backend because it allows an attacker to traverse your installation and find exploits. It is also better to split up a complex application into different simpler responsibilities so that they are easier to solve.
This solution should 'separate the concerns' of the backend and frontend, as well as the two use cases to upload and download the file:
Use Case 1: Upload an image file to the server.
User selects image file from Client system
Client reads and uploads the file to the server
Server saves the file and associates it to content. (this is the sample code)
Use Case 2: Download and display an image file for a matching article on the UI.
Client requests an article
Server sends article content and ID for image files
Client requests image files and assigns the response to image components.
Server retrieves file by ID and returns it to the client.
Client image components render image files on the client system.
So, your backend needs to save the file in some way that it can publish it for later use. The best-practice as mentioned in comments is to put the file in a public folder and use the name as the ID, then let the webserver treat it as a static file. This allows the full benefit of web and cloud technologies to help you speed up and maintain your solution.
When the client uploads the image, the server should save it to a public folder and return its name/ID. The client path to the file is not enough in the upload, because the server can't access the client's system. Instead, the client needs to send a stream of bytes with metadata in the HTTP request. If you are only setting the filename in the upload, you need to read and send the bytes to the server like so:
JSON Example:
// read in the file bytes when opened:
onFileOpened(e) {
let data={};
let reader = new FileReader();
reader.onloadend = () => {
data.image= reader.result ;
};
reader.readAsDataURL(event.target.files[0]);
data.id=generateId();
}
// send the bytes on submit:
onSubmit(data){
POST(data);
}
Multipart Example:
With multer, the client is expected to send a Multipart/Form-Data request. This is falling back to older HTML tech and encoding the bytes similarly as above. However, it is still bytes in a wrapper.
<form action="/profile" method="post" enctype="multipart/form-data">
<input type="file" name="avatar" />
</form>
Then when the server receives the stream and invokes the createArticle method it should read the bytes and save them to a shared path, DB, or Content Delivery Network (CDN). The server should host or enable an endpoint to GET the same files from the recorded ID. Then the content can reference the file by appending the file's ID/name to the image endpoint.
Instead of renaming the file from the client path to the destination, the bytes need to be written to disk:
fs.createReadStream(req.file).pipe(fs.createWriteStream(newPath ));
This allows for the images to be hosted on multiple providers, allows many instances of your webserver to run in parallel, and allows you to size uploads and download services independently.
Context
I am working on a Proof of Concept for an accounting bot. Part of the solution is the processing of receipts. User makes picture of receipt, bot asks some questions about it and stores it in the accounting solution.
Approach
I am using the BotFramework nodejs example 15.handling attachments that loads the attachment into an arraybuffer and stores it on the local filesystem. Ready to be picked up and send to the accounting software's api.
async function handleReceipts(attachments) {
const attachment = attachments[0];
const url = attachment.contentUrl;
const localFileName = path.join(__dirname, attachment.name);
try {
const response = await axios.get(url, { responseType: 'arraybuffer' });
if (response.headers['content-type'] === 'application/json') {
response.data = JSON.parse(response.data, (key, value) => {
return value && value.type === 'Buffer' ? Buffer.from(value.data) : value;
});
}
fs.writeFile(localFileName, response.data, (fsError) => {
if (fsError) {
throw fsError;
}
});
} catch (error) {
console.error(error);
return undefined;
}
return (`success`);
}
Running locally it all works like a charm (also thanks to mdrichardson - MSFT). Stored on Azure, I get
There was an error sending this message to your bot: HTTP status code InternalServerError
I narrowed the problem down to the second part of the code. The part that write to the local filesystem (fs.writefile). Small files and big files result in the same error on Azure.fs.writefile seams unable to find the file
What is happpening according to stream logs:
Attachment uploaded by user is saved on Azure
{ contentType: 'image/png',contentUrl:
'https://webchat.botframework.com/attachments//0000004/0/25753007.png?t=< a very long string>',name: 'fromClient::25753007.png' }
localFilename (the destination of the attachment) resolves into
localFileName: D:\home\site\wwwroot\dialogs\fromClient::25753007.png
Axios loads the attachment into an arraybuffer. Its response:
response.headers.content-type: image/png
This is interesting because locally it is 'application/octet-stream'
fs throws an error:
fsError: Error: ENOENT: no such file or directory, open 'D:\home\site\wwwroot\dialogs\fromClient::25753007.png
Some assistance really appreciated.
Removing ::fromClient prefix from attachment.name solved it. As #Sandeep mentioned in the comments, the special characters where probably the issue. Not sure what its purpose is. Will mention it in the Botframework sample library github repository.
[update] team will fix this. Was caused by directline service.
I am using the react-native-fs and I am trying to save a base64 of a pdf file to my android emulators file system.
I receive base64 encoded pdf from the server.
I then decode the base64 string with the line:
var pdfBase64 = 'data:application/pdf;base64,'+base64Str;
saveFile() function
saveFile(filename, pdfBase64){
// create a path you want to write to
var path = RNFS.DocumentDirectoryPath + '/' + filename;
// write the file
RNFS.writeFile(path, base64Image, 'base64').then((success) => {
console.log('FILE WRITTEN!');
})
.catch((err) => {
console.log("SaveFile()", err.message);
});
}
Error
When I try saving the pdfBase64 the saveFile() function catches the following error:
bad base-64
Question
Can anyone tell where or what I am doing wrong?
Thanks.
For anyone having the same problem, here is the solution.
Solution
react-nativive-pdf-view must take the file path to the pdf_base64.
Firstly, I used the react-native-fetch-blob to request the pdf base64 from the server.(Because RN fetch API does not yet support BLOBs).
Also I discovered that react-native-fetch-blob also has a FileSystem API which is way better documented and easier to understand than the 'react-native-fs' library. (Check out its FileSystem API documentation)
Receiving base64 pdf and saving it to a file path:
var RNFetchBlob = require('react-native-fetch-blob').default;
const DocumentDir = RNFetchBlob.fs.dirs.DocumentDir;
getPdfFromServer: function(uri_attachment, filename_attachment) {
return new Promise((RESOLVE, REJECT) => {
// Fetch attachment
RNFetchBlob.fetch('GET', config.apiRoot+'/app/'+uri_attachment)
.then((res) => {
let base64Str = res.data;
let pdfLocation = DocumentDir + '/' + filename_attachment;
RNFetchBlob.fs.writeFile(pdfLocation, pdf_base64Str, 'base64');
RESOLVE(pdfLocation);
})
}).catch((error) => {
// error handling
console.log("Error", error)
});
}
What I was doing wrong was instead of saving the pdf_base64Str to the file location like I did in the example above. I was saving it like this:
var pdf_base64= 'data:application/pdf;base64,'+pdf_base64Str;
which was wrong.
Populate PDF view with file path:
<PDFView
ref={(pdf)=>{this.pdfView = pdf;}}
src={pdfLocation}
style={styles.pdf}
/>
There is a new package to handle the fetching (based on react-native-fetch-blob) and displaying of the PDF via URL: react-native-pdf.
Remove application type in base64 string and it's working for me
var pdfBase64 = 'data:application/pdf;base64,'+base64Str;
To
var pdfBase64 = base64Str;