The further I go with my blog the more problems I hit :) Can anyone tell me what is the best way to save images. I store the data in mongodb, should I save images in there as well or should I use local file system? Thank you
use file system to store images, free up db resources to serve data. for bigger site images should really use CDN.
Store them in MongoDB using GridFS. That way you're not limited by file size, the images are easily shared between multiple app servers, and the images are naturally backed up with the rest of your MongoDB data.
Related
I want to create a website to provide PDF notes to students.
I am planning to upload pdf on my googledrive and want to serve those pdf using my website.
I am using MongoDb As Database and my plan is to read link of pdf from my google drive and want to save it on collection of mongodb so that it will render on my website dynamically.
Is there any convenient way to do this automatically rather than copy pasting each link.(because it will be a time draining stuff and we have to upload 1000 of pdfs).
If this method is not possible Please suggest me some better way to do this work using nodejs and mongodb.
If using Google Drive is not a must, I recommend you to use MongoDB's GridFS. GridFS allows you to store documents that are higher than 16MB. In this way, you do not need to manage/update document links, because MongoDB will be already in charge of metadata and the document itself.
If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed.
When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory.
When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities.
This tutorial might be the starting point.
I need to add images to my mongoDB using Node and Express.
I am able to normal data in it by running the mongo shell. But I cannot find any method to add images to it.
Can anybody help?
Please don't do this. Databases are not particularly well suited to storing large bits of data like images, files, etc.
Instead: you should store your images in a dedicated static file store like Amazon S3, then store a LINK to that image in your MongoDB record.
This is a lot better in terms of general performance and function because:
It will reduce your database hosting costs (it is cheaper to store large files in S3 or other file services than in a database).
It will improve database query performance: DBs are fast at querying small pieces of data, but bad at returning large volumes of data (like files).
It will make your site or application much faster: instead of needing to query the DB for your image when you need it, you can simply output the image link and it will be rendered immediately.
Overall: it is a much better / safer / faster strategy.
Simply, I need to build an app to store images for users. So each user can upload images and view them on the app.
I am using NodeJS and Mongo/Mongoose.
Is this a good approach to handle this case:
When the user uploads the image file, I will store it locally.
I will use Multer to store the file.
Each user will have a separate folder created by his username.
In the user schema, I will define a string array that records the file path.
When user needs to retrieve the file, I will check the file path, retrieve it from the local disk.
Now my questions are:
Is this a good approach (storing in local file system and storing path in schema?
Is there any reason to use GridFS, if the file sizes are small (<1MB)?
If I am planning to use S3 to store files later, is this a good strategy?
This is my first time with a DB application like this so very much appreciate some guidance.
Thank you.
1) Yes, storing the location within your database for use within your application and the physical file elsewhere is an appropriate solution. Depending on the data store and number of files it can be detrimental to store within a database as it can impede processes like backup and replication if there are many large files
2) I admit that I don't know GridFS but the documentation says it is for files larger than 16MB so it sounds like you don't need it yet
3) S3 is a fantastic product and enables edge caching and backup through services and many others too. I think your choice needs to look at what AWS provides and if you need it e.g. global caching or replication to different countries and data centres. Different features cause different price points but personally I find the S3 platform excellent and have around 500G loaded there for different purposes.
I am working on application in which users will upload huge number of images and i have to show those image webpage
What is the best way to store and retrieve images.
1) Database
2) FileSystem
3) CDN
4) JCR
or something else
What i know is
Database: saving and retrieving image from database will lead to lot of queries to database and will convert blob to file everytime. I think it will degrade the website performance
FileSystem: If i keep image information in database and image file in filesystem there will be sync issues. Like if i took a backup of the database we do have take the backup of images folder. ANd if there are millions of files it will consume lot of server resources
i read it here
http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/
Another options are CDNs and JCR
Please suggest the best option
Regards
Using the File System is only really an option if you only plan to deploy to one server (i.e. not several behind a load balancer), OR if all of your servers will have access to a shared File System. It may also be inefficient, unless you cache frequently-accessed files in the application server.
You're right that storing the actual binary data in a Database is perhaps overkill, and not what databases do best.
I'd suggest a combination:
A CDN (such as AWS CloudFront), backed by a publicly-accessible (but crucially publicly read-only) storage such as Amazon S3 would mean that your images are efficiently served, wherever the browsing user is located and cached appropriately in their browser (thus minimising bandwidth). S3 (or similar) means you have an API to upload and manage them from your application servers, without worrying about how all servers (and the outside world) will access them.
I'd suggest perhaps holding meta data about each image in a Database however. This means that you could assign each image a unique key (generated by your database), add extra info (file format, size, tags, author, etc), and also store the path to S3 (or similar) via the CDN as the publicly-accessible path to the image.
This combination of Database and shared publicly-accessible storage is probably a good mix, giving you the best of both worlds. The Database also means that if you need to move / change or bulk delete images in future (perhaps deleting all images uploaded by an author who is deleting their account), you can perform an efficient Database query to gather metadata, followed by updating / changing the stored images at the S3 locations the Database says they exist.
You say you want to display the images on a web page. This combination means that the application server can query the database efficiently for the image selection you want to show (including restricting by author, pagination, etc), then generate a view containing images referring to the correct CDN path. It means viewing the images is also quite efficient as you combine dynamic content (the page upon which the images are shown) with static content (the image themselves via the CDN).
CDNs may be a good option for you.
You can store the link to the images along with the image information in your database.
I use Node.js for my project.
Should I store an image in local file system, or should I store it in MongoDB?
Which way is more scalable?
The most scalable solution is to use a shared storage service such as Amazon's S3 (or craft your own).
This allows you to scale horizontally a lot easier when you decide to add machines to your application layer, as you won't have to worry about any migration nightmares.
The basic idea behind this is to keep the storage layer decoupled from the application layer. So using this idea you could create a node.js process on a separate machine that accepts file uploads then writes them to disk.
If you're designing a performance sensitive system, use file system to store your images no doubt.
You can find the performance compare from this blog:
http://blog.thisisfeifan.com/2013/12/mongodb-gridfs-performance-test.html
Actually, you can find the recommended MongoDB GridFS use cases here:
https://docs.mongodb.com/manual/core/gridfs/#when-to-use-gridfs
I would use GridFS to take advantage of sharding but for best performance I would use filesystem with nginx.