Puppeteer, exchange buffers with node.js - node.js

The page running inside puppeteer download images. When calling page.evaluate, some of the images must be written to disk before doing other operations on them.
What's the best way to do this? Be able to write those images from the browser running in puppeteer? Send buffers from puppeteer to node.js?
NOTE: some of these images might be result canvas operations, so they're not necessary the result of a request.

There is a few possible solutions:
1. If the page allows cors, you can send those images to your backend server. Use something like fetch
2. Another option is to get all images from page and pass them back to server. Here is also a few possible implementations:
2.1. You can get all image urls with something like this return JSON.stringify(Array.from(document.querySelectorAll('img'), i => i.src)) and then download them directly to your server.
2.2. Put image to canvas, and then use toDataUrl() to get base64 encoded data, which you could then pass and process on backend. More info here and here is about saving base64 data on disk
P.S. Just remember to JSON.stringify() your data when returning from puppeteer to backend, because puppeteer could not process the data.

Related

NodeJS, how to handle image uploading with MongoDB?

I would like to know what is the best way to handle image uploading and saving the reference to the database. What I'm mostly interested is what order do you do the process in?
Should you upload the images first in the front-end (say Cloudinary), and then call the API with result links to the images and save it to the database?
Or should you upload the images to the server first, and upload them from the back-end and save the reference afterwards?
OR, should you do the image uploading after you save the record in the database and then update it once the images were uploaded?
It really depends on the resources, timeline, and number of images you need to upload daily.
So basically if you have very few images to upload then you can upload that image to your server then upload it to any cloud storage(s3, Cloudinary,..) you are using. As this will be very easy to implement(you can find code snippet over the internet) and you can securely maintain your secret keys/credential to your cloud platform on the server side.
But, according to me best way of doing this will be something like this. I am taking user registration as an example
Make server call to get a temporary credential to upload files on the cloud(Generally, all the providers give this functionality i.e. STS/Signed URL in AWS).
The user will fill up the form and select the image on the client side. When the user clicks the submit button make one call to save the user in the database and start upload with credentials. If possible keep a predictable path for upload. Like for user upload /users/:userId or something like that. this highly depends on your use case.
Now when upload finishes make a server call for acknowledgment and store some flag in the database.
Now advantages of this approach are:
You are completely offloading your server from handling file operations which are pretty heavy and I/O blocking and you are distributing that load to all clients.
If you want to post process the files after upload you can easily integrate this with serverless platforms and do that on there and again offload that.
You can easily provide retry mechanism to your users in case of file upload fails but they won't need to refill the data, just upload the image/file again
You don't need to expose the URL directly to the client for file upload as you are using temporary Creds.
If the significance of the images in your app is high then ideally, you should not complete the transaction until the image is saved. The approach should be to create an object in your code which you will eventually insert into mongodb, start upload of image to cloud and then add the link to this object. Finally then insert this object into mongodb in one go. Do not make repeated calls. Anything before that, raise an error and catch the exception
You can have many answers,
if you are working with big files greater than 16mb please go with gridfs and multer,
( changing the images to a different format and save them to mongoDB)
If your files are actually less than 16 mb, please try using this Converter that changes the image of format jpeg / png to a format of saving to mongodb, and you can see this as an easy alternative for gridfs ,
please check this github repo for more details..

How manage time consuming API Calls in NodeJS

I am using an API from TinyPng.com which helps in compressing images.
When a user uploads images, I make an API call to TinyPng.com with the URL of the image that I want to compress, it provides a response with a link of the compressed file. Based on image size it could take about a few seconds to get a response.
How can I handle such time consuming processes in NodeJS ?
What you should be doing is handling that in the background on the server side. Once the compressed image is ready, download and replace the original one

Create thumbnails using AWS S3 & Lambda and then load them

I'm building a react webapp which allows users to upload a picture and then render the thumbnail, just like facebook chat. But I'm not sure what's the best practice to solve the uploading-resizing-generatingThumbnail time gap before I can render them.
The workflow is like:
1) User uploads a picture to S3, stored in bucket1
2) Lambda function invoked, getting the newly uploaded picture and do the resizing work, then store thumbnail in bucket2
3) Thumbnail rendered in browser(client-side). Here it's a bit tricky - I just hardcode the img url because it's predictable, however, it takes a while before available(generating process).
But I don't know how to let lambda notifies the browser when a thumbnail successfully generated and ready to render. In production it might be lambda tells node server first, and then node server tells the client; but in developing mode, it seems impossible as I'm running an express server on my own laptop. Should lambda do the notification in a proper way or there's other better solution?
AWS Lambda cannot "notify the browser" because it is a process that was independently triggered and has no connection with the web page request.
One option would be to code the web page to keep trying to download the image. You'd need some fancy JavaScript/node code that can check whether the image was successfully downloaded and then retry if necessary.
By the way, there are also services that can resize images on-the-fly so you don't have to create your own thumbnails:
Cloudinary
Imgix

How to send or stream base64 images with AJAX to Node.js Express server?

I am new to working with images in web development. We have a Node.js Express server that will run on Heroku and uses Cloudinary to store images.
Ideally we could save images directly to Cloudinary, but I am not sure if that's possible and we are afraid of putting our Cloudinary credentials on the client.
Assuming we must send images data to our server first instead of sending them directly to Cloudinary - if the images are encoded as base64 on the client, is it possible to stream the images from the client to the server - or must we send all the data at once? Either way, what headers do we use to send binary / base64 data?
is it possible to send or even stream binary data from the client to the server?
since it is a Node.js server, it would be ideal to use streams and to stream the file from our server to Cloudinary.
hope this makes sense, and info would be very helpful.
Why not using direct uploads from the client side to Cloudinary using the jQuery plugin?
This method supports both signed or unsigned uploads, when the signature can (and should) be generated on your server before rendering the page, for privacy reasons. Uploading from Base64 URI is also possible with this mechanism.
Note that Cloudinary's client-libraries also wrap this plugin and provide you with "off the shelf" solutions for embedding the upload-fields in your web app, with the signature already inside.
Let us know if you need any further guidance.

Uploading user images to s3 and generating thumbnails from node

I'm currently considering developing a Meteor node.js app, but am struggling with how best to handle uploading of user images. In particular, I want to create a photography website that will allow the photographer to upload images in an 'admin' section, and these images will then be displayed on the website. I need to create a thumbnail of these images, and save the respective URLs to the database. I'm struggling with how to best accomplish this in meteor.
Is my best bet to use something like s3 combined with an AWS process for generating thumbnails?
Or should I save and host the images directly in the Meteor/node session?
Or should I scrap Meteor and use something like Express.js for this project?
Why don't you just use something like Filepicker.io to handle uploading and hosting images and simply store the image unique url (given to you by filepicker in the callback)?
Thumbnails can also be dynamically generated by Filepicker (using simple url modifications).
Cloudinary is a nicer alternative to filepicker when it comes to images, but integration process will be messier.
I would store the images on the filesystem, not in a database. If you have a unique id, you can use that as part of the url, for example an id of the item the image belongs to. Might look like this:
./uploads/img-<id>-<size>.jpg
You can write to disk and resize if necessary with node-imagemagick and your cdn should just poll these images from time to time. Not exactly sure how that part would work in terms of including the url to the image in the html.

Resources