I am trying to send a picture from my mobile hybrid app (Ionic 3) to my Heroku backend (Node.js) and have the backend upload the picture to Firebase Storage and return the newly uploaded fil download url to the mobile app.
Keep in mind that I am using the Firebase Admin SDK for Node.js.
So I send the base64 encoded image to Heroku (I check the encoded string with an online base64 decoder and it is alright) which is handle by the following function:
const uploadPicture = function(base64, postId, uid) {
return new Promise((resolve, reject) => {
if (!base64 || !postId) {
reject("news.provider#uploadPicture - Could not upload picture because at least one param is missing.");
}
let bufferStream = new stream.PassThrough();
bufferStream.end(new Buffer.from(base64, 'base64'));
// Retrieve default storage bucket
let bucket = firebase.storage().bucket();
// Create a reference to the new image file
let file = bucket.file(`/news/${uid}_${postId}.jpg`);
bufferStream.pipe(file.createWriteStream({
metadata: {
contentType: 'image/jpeg'
}
}))
.on('error', error => {
reject(`news.provider#uploadPicture - Error while uploading picture ${JSON.stringify(error)}`);
})
.on('finish', (file) => {
// The file upload is complete.
console.log("news.provider#uploadPicture - Image successfully uploaded: ", JSON.stringify(file));
});
})
};
I have 2 major issues:
Upload succeeds but I when I go to Firebase Storage console, there is an error when I try to display the preview of the picture and I cannot open it from my computer when I download it. I guess it is an encoding thing....?
How can I retrieve the newly uploaded file download url ? I was expecting an object to be returned in the .on('finish), like in the upload() function, but none is returned (file is undefined). How could I retrieve this url to send it back in the server response?
I want to avoid using the upload() function because I don't want to host files on the backend as it is not a dedicated server.
My problem was that I add data:image/jpeg;base64,at the beginning of the base64 object string ; I just had to remove it.
For the download url, I did the following:
const config = {
action: 'read',
expires: '03-01-2500'
};
let downloadUrl = file.getSignedUrl(config, (error, url) => {
if (error) {
reject(error);
}
console.log('download url ', url);
resolve(url);
});
Related
I am uploading images to firebase storage using the Admin SDK from NodeJS. When I try to preview the file it doesn't load because it is broken.
Its size is correct, but the preview in the dashboard just throws an error, and the image url returns a white/black small square (depends on the browser).
This is my code from NodeJS:
const bufferStream = new stream.PassThrough();
await bufferStream.end(Buffer.from(user.photoURL, 'base64'));
const mimeType = user.photoURL.match(/[^:]\w+\/[\w-+\d.]+(?=;|,)/)[0];
const fileExtension = mimeType.split('/').pop();
const file = storageBucket.file(`avatars/${user.username}.${fileExtension}`);
const uid = v4();
console.log(uid);
bufferStream.pipe(file.createWriteStream({
metadata: {
contentType: mimeType,
metadata: {
firebaseStorageDownloadTokens: uid,
},
},
}))
.on('error', (error) => {
console.log('error', error);
})
.on('finish', () => {
// The file upload is complete.
console.log('COMPLETED, WORKED');
});
I managed to solve it myself in the end, just for future reference:
Given the exact same code I posted in the original question, I just had to remove the text preceding the actual base64.
Original base64: data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQA=
Needed base64: /9j/4AAQSkZJRgABAQAAAQA=
Just remove the first part of the string so it looks like this:
bufferStream.end(Buffer.from(user.photoURL.split(';base64,')[1], 'base64'));
I have a react web app that allows users to record mp3 files in the browser. These mp3 files are saved in an AWS S3 bucket and can be retrieved and loaded back into the react app during the user's next session.
Saving the file works just fine, but when I try to retrieve the file with getObject() and try to create an mp3 blob on the client-side, I get a small, unusable blob:
Here's the journey the recorded mp3 file goes on:
1) Saving to S3
In my Express/Node server, I receive the uploaded mp3 file and save to the S3 bucket:
//SAVE THE COMPLETED AUDIO TO S3
router.post("/", [auth, upload.array('audio', 12)], async (req, res) => {
try {
//get file
const audioFile = req.files[0];
//create object key
const userId = req.user;
const projectId = req.cookies.currentProject;
const { sectionId } = req.body;
const key = `${userId}/${projectId}/${sectionId}.mp3`;
const fileStream = fs.createReadStream(audioFile.path)
const uploadParams = {
Bucket: bucketName,
Body: fileStream,
Key: key,
ContentType: "audio/mp3"
}
const result = await s3.upload(uploadParams).promise();
res.send(result.key);
} catch (error) {
console.error(error);
res.status(500).send();
}
});
As far as I know, there are no problems at this stage. The file ends up in my S3 bucket with "type: mp3" and "Content-Type: audio/mp3".
2) Loading file from S3 Bucket
When the react app is loaded up, an HTTP GET Request is made in my Express/Node server to retrieve the mp3 file from the S3 Bucket
//LOAD A FILE FROM S3
router.get("/:sectionId", auth, async(req, res) => {
try {
//create key from user/project/section IDs
const sectionId = req.params.sectionId;
const userId = req.user;
const projectId = req.cookies.currentProject;
const key = `${userId}/${projectId}/${sectionId}.mp3`;
const downloadParams = {
Key: key,
Bucket: bucketName
}
s3.getObject(downloadParams, function (error, data) {
if (error) {
console.error(error);
res.status(500).send();
}
res.send(data);
});
} catch (error) {
console.error(error);
res.status(500).send();
}
});
The "data" returned here is as such:
3) Making a Blob URL on the client
Finally, in the React client, I try to create an 'audio/mp3' blob from the returned array buffer
const loadAudio = async () => {
const res = await api.loadAudio(activeSection.sectionId);
const blob = new Blob([res.data.Body], {type: 'audio/mp3' });
const url = URL.createObjectURL(blob);
globalDispatch({ type: "setFullAudioURL", payload: url });
}
The created blob is severely undersized and appears to be completely unusable. Downloading the file results in a 'Failed - No file' error.
I've been stuck on this for a couple of days now with no luck. I would seriously appreciate any advice you can give!
Thanks
EDIT 1
Just some additional info here: in the upload parameters, I set the Content-Type as audio/mp3 explicitly. This is because when not set, the Content-Type defaults to 'application/octet-stream'. Either way, I encounter the same issue with the same result.
EDIT 2
At the request of a commenter, here is the res.data available on the client-side after the call is complete:
Based on the output of res.data on the client, there are a couple of things that you'd need to do:
Replace uses of res.data.Body with res.data.Body.data (as the actual data array is in the data attribute of res.data.Body)
Pass a Uint8Array to the Blob constructor, as the existing array is of a larger type, which will create an invalid blob
Putting that together, you would end up replacing:
const blob = new Blob([res.data.Body], {type: 'audio/mp3' });
with:
const blob = new Blob([new Uint8Array(res.data.Body.data)], {type: 'audio/mp3' });
Having said all that, the underlying issue is that the NodeJS server is sending the content over as a JSON encoded serialisation of the response from S3, which is likely overkill for what you are doing. Instead, you can send the Buffer across directly, which would involve, on the server side, replacing:
res.send(data);
with:
res.set('Content-Type', 'audio/mp3');
res.send(data.Body);
and on the client side (likely in the loadAudio method) processing the response as a blob instead of JSON. If using the Fetch API then it could be as simple as:
const blob = await fetch(<URL>).then(x => x.blob());
Your server side code seems alright to me. I'm not super clear about the client-side approach. Do you load this into the blob into the HTML5 Audio player.
I have a few approaches, assuming you're trying to load this into an audio tag in the UI.
<audio controls src="data:audio/mpeg;base64,blahblahblah or html src" />
Assuming that the file you had uploaded to S3 is valid here are two approaches:
Return the data as a base64 string instead of as a buffer directly from S3. You can do this in your server side by returning as
const base64MP3 = data.Body.toString('base64');
You can then pass this in to the MP3 player in the src property and it will play the audio. Prefix it with data:audio/mpeg;base64
Instead of returning the entire MP3 file, have your sectionID method return a presigned S3 URL. Essentially, this is a direct link to the object in S3 that is authorized for say 5 minutes.
Then you should be able to use this URL directly to stream the audio
and set it as the src. Keep in mind that it will expire.
const url = s3.getSignedUrl('getObject', {
Bucket: myBucket,
Key: myKey,
Expires: signedUrlExpireSeconds
});
You stated: "The created blob is severely undersized and appears to be completely unusable"
This appears to me that you have an encoding issue. Once you read the MP3 from the Amazon S3 bucket, you need to encode it properly so it functions in a web page.
I did a similar multimedia use case that involved MP4 and a Java app. That is, i wanted a MP4 obtained from a bucket to play in the web page - as shown in this example web app.
Once I read the byte stream from the S3 bucket, I had to encode it so it would play in a HTML Video tag. Here is a good reference to properly encode a MP3 file.
I am working nodejs and angular and I am trying to download an image that I uploaded using nodejs (created a upload folder all the uploaded images are in server side upload folder), I already written an rest api but its not working.
let me explain the exact situation, there is a form for teacher for uploading the assignment he/she upload the assignment in the form of image now students can see the assignment uploaded by the teacher I also want to give the download option so the image can get download to the local machine.
I am using mime-types for this
download api
app.get("/download/:filename", (req, res) => {
console.log(req.body);
var imagename = req.params.filename;
var file = __dirname + "/uploads/" + imagename;
var filename = path.basename(file);
var mimetype = mime.lookup(file);
res.setHeader('Content-disposition', 'attachment; filename=' + filename);
res.setHeader('Content-type', mimetype);
res.download(file)
var filestream = fs.createReadStream(file);
filestream.pipe(res);
})
from the angular side I am passing the object that contain name of the image that present in server side, folder name is upload
service.ts
(Right now I am passing the static name)
private downloadImageUrl = "http://localhost:3000/download/1592945165166-java-1.png";
downloadImage(assignment: Assignment) {
return this.http.get(this.downloadImageUrl, {
observe: "body",
responseType: 'blob'
})
}
component.ts
onDownload(index: number) {
console.log(this.assignment[index])
this.teacherServ.downloadImage()
.subscribe(
res => console.log(res),
err => console.log(err)
)
}
I want to image get downloaded in the local system of download folder
No error message is there but still image is not downloaded to my local storage of download folder.
UPDATE:
I successfully able to save the image using the file-saver in angular-7, I will share the code here but still there is a one problem, problem is jpg format images are getting saved as a file format or an unknown format (basically I am unable to open it.), there is no problem with png format. So I want to how how can I save the jpg as well as png format of image using file-saver
service.ts
private downloadImageUrl = "http://localhost:3000/download";
downloadImage(file: any){
return this.http.post(this.downloadImageUrl, file, {
responseType : 'blob',
headers : new HttpHeaders().append('content-type','application/json')
})
}
component.html
<td>
<img
height="92px"
style="cursor: pointer"
src="http://localhost:3000/{{receivedAssignment.assignment}}"
alt="not found"
(click)="onDownload(i)">
component.ts
onDownload(index: number) {
console.log(this.assignment[index].assignment)
var filename = this.assignment[index].filename
this.teacherServ.downloadImage(this.assignment[index]) //sending the particular assignment that I want to download.
.subscribe(
(data) => {
if(data && data != undefined && data != null) {
saveAs(data, filename)
}
}
)
}
app.js //nodejs api
app.post('/download', function(req,res,next){
console.log(req.body.assignment);
console.log(path.join(__dirname,'uploads','/'+req.body.assignment))
filepath = path.join(__dirname,'uploads','/'+req.body.assignment);
res.sendFile(filepath);
});
So, this method is working for png images but not for jpg image, please let me know why? and how can I solve this?
Thanks.
I created cloud function that generates an xlsx file, I need the user to download that file after it's generated.
Method 1: Upload to Bucket, then redirect
So far i've tried uploading the file to a bucket using this API, and then redirect him to the bucket file url, I also double checked the bucket name using this API, but I get the same error every time:
{"error":{"code":500,"status":"INTERNAL","message":"function crashed","errors":["socket hang up"]}}
Portion of the code that contains uploading to bucket:
const { Storage } = require('#google-cloud/storage');
const storage = new Storage();
await storage.bucket('bucket-name').upload('myfile.xlsx', {
gzip: false,
});
Portion of the code that proves file exists:
fs.access('myfile.xlsx', fs.constants.F_OK, (err) => {
console.log(`${file} ${err ? 'does not exist' : 'exists'}`);
});
I also checked if the library "#google-cloud/storage" reads the file, and it reads it correctly and gets the file size right.
Method 2: Direct Download
Download the file directly, the problem is that every doc online for nodejs to download a local file to the user is setting up a custom server to download the file, but i'm using firebase, so it's not in control of that server.
Just wanted to add more detail to the answer, since there's no need to write into a file and read from it to download it's data, simply take the data and send it, using the few lines below.
res.setHeader('Content-Type', 'application/vnd.openxmlformats');
res.setHeader("Content-Disposition", "attachment; filename=" + fileName);
res.end(fileData, 'binary');
If your excel file is created and should be returned to the client as a response to an HTTP request (calling to an API endpoint) then this is how you can do it.
export const getExcelFile = functions.https.onRequest(async (request, response) => {
// ...
// Create your file and such
// ..
await storage.bucket('bucket-name').upload('myfile.xlsx', {
gzip: false,
});
response.setHeader('Content-Type', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet');
response.send(fs.readFileSync('myfile.xlsx'));
return null;
});
Otherwise, if the excel file is created as a response to an event, and you want the user to download the file at another time, then you create a download link and serve it to the user in any way you want.
// ...
// Create your file and such
// ..
const [file] = await storage.bucket('bucket-name').upload('myfile.xlsx', {
gzip: false,
});
const [downloadUrl] = await file.getSignedUrl({
action: 'read',
expires: '20-03-2019' // Link expiry date: DD-MM-YYYY
});
console.log(downloadUrl);
I am implementing a web app using MEAN Stack and Angular 6. There I want to submit a form with file upload. '.png' files should be uploaded.
I want to save the file in a different file server and send the url to the image.Currently I upload files into a folder in my project and save the image in db (I used ng2fileupload and multer for that.). Then it saves like this.
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAV4AAAFUCAYAAABssFR8AAAK..."
But I want to save the image url and the image should be retrived by the url. Does anyone can explain a proper method for that?
I faced the same problem a month ago and find out a solution to this problem. Though I haven't used multer in the app.
From my frontend, I will be sending an object to Node API endpoint /event which will look like:-
let img = {
content: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUg...",
filename: 'yourfile.png'
}
At the backend, I'm using Cloudinary to store my images (Its free plan allows 10GB storage) and returns secure https URLs. So install it using npm i cloudinary and require in your api.js file.
And add the below configuration
cloudinary.config({
cloud_name: 'yourapp',
api_key: 'YOUR_KEY',
api_secret: 'YOUR_SECRET_KEY'
});
Last Step:- (Not so optimized code)
Let say I have an event Schema which has images array, where I'll be storing the URLs returned by cloudinary.
app.post('/event', (req, res) => {
try {
if (req.body.images.length > 0) {
// Creating new Event instance
const event = new Event({
images: [],
});
// Looping over every image coming in the request object from frontend
req.body.images.forEach((img) => {
const base64Data = img.content.split(',')[1];
// Writing the images in upload folder for time being
fs.writeFileSync(`./uploads/${img.filename}`, base64Data, 'base64', (err) => {
if (err) {
throw err;
}
});
/* Now that image is saved in upload folder, Cloudnary picks
the image from upload folder and store it at their cloud space.*/
cloudinary.uploader.upload(`./uploads/${img.filename}`, async (result) => {
// Cloudnary returns id & URL of the image which is pushed into the event.images array.
event.images.push({
id: result.public_id,
url: result.secure_url
});
// Once image is pushed into the array, I'm removing it from my server's upload folder using unlinkSync function
fs.unlinkSync(`./uploads/${img.filename}`);
// When all the images are uploaded then I'm sending back the response
if (req.body.images.length === event.images.length) {
await event.save();
res.send({
event,
msg: 'Event created successfully'
});
}
});
});
}
} catch (e) {
res.status(400).send(e);
}
});
P.S. Go ahead and suggest some optimization solution for this code here