I need to add images to my mongoDB using Node and Express.
I am able to normal data in it by running the mongo shell. But I cannot find any method to add images to it.
Can anybody help?
Please don't do this. Databases are not particularly well suited to storing large bits of data like images, files, etc.
Instead: you should store your images in a dedicated static file store like Amazon S3, then store a LINK to that image in your MongoDB record.
This is a lot better in terms of general performance and function because:
It will reduce your database hosting costs (it is cheaper to store large files in S3 or other file services than in a database).
It will improve database query performance: DBs are fast at querying small pieces of data, but bad at returning large volumes of data (like files).
It will make your site or application much faster: instead of needing to query the DB for your image when you need it, you can simply output the image link and it will be rendered immediately.
Overall: it is a much better / safer / faster strategy.
Related
Simply, I need to build an app to store images for users. So each user can upload images and view them on the app.
I am using NodeJS and Mongo/Mongoose.
Is this a good approach to handle this case:
When the user uploads the image file, I will store it locally.
I will use Multer to store the file.
Each user will have a separate folder created by his username.
In the user schema, I will define a string array that records the file path.
When user needs to retrieve the file, I will check the file path, retrieve it from the local disk.
Now my questions are:
Is this a good approach (storing in local file system and storing path in schema?
Is there any reason to use GridFS, if the file sizes are small (<1MB)?
If I am planning to use S3 to store files later, is this a good strategy?
This is my first time with a DB application like this so very much appreciate some guidance.
Thank you.
1) Yes, storing the location within your database for use within your application and the physical file elsewhere is an appropriate solution. Depending on the data store and number of files it can be detrimental to store within a database as it can impede processes like backup and replication if there are many large files
2) I admit that I don't know GridFS but the documentation says it is for files larger than 16MB so it sounds like you don't need it yet
3) S3 is a fantastic product and enables edge caching and backup through services and many others too. I think your choice needs to look at what AWS provides and if you need it e.g. global caching or replication to different countries and data centres. Different features cause different price points but personally I find the S3 platform excellent and have around 500G loaded there for different purposes.
I am working on application in which users will upload huge number of images and i have to show those image webpage
What is the best way to store and retrieve images.
1) Database
2) FileSystem
3) CDN
4) JCR
or something else
What i know is
Database: saving and retrieving image from database will lead to lot of queries to database and will convert blob to file everytime. I think it will degrade the website performance
FileSystem: If i keep image information in database and image file in filesystem there will be sync issues. Like if i took a backup of the database we do have take the backup of images folder. ANd if there are millions of files it will consume lot of server resources
i read it here
http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/
Another options are CDNs and JCR
Please suggest the best option
Regards
Using the File System is only really an option if you only plan to deploy to one server (i.e. not several behind a load balancer), OR if all of your servers will have access to a shared File System. It may also be inefficient, unless you cache frequently-accessed files in the application server.
You're right that storing the actual binary data in a Database is perhaps overkill, and not what databases do best.
I'd suggest a combination:
A CDN (such as AWS CloudFront), backed by a publicly-accessible (but crucially publicly read-only) storage such as Amazon S3 would mean that your images are efficiently served, wherever the browsing user is located and cached appropriately in their browser (thus minimising bandwidth). S3 (or similar) means you have an API to upload and manage them from your application servers, without worrying about how all servers (and the outside world) will access them.
I'd suggest perhaps holding meta data about each image in a Database however. This means that you could assign each image a unique key (generated by your database), add extra info (file format, size, tags, author, etc), and also store the path to S3 (or similar) via the CDN as the publicly-accessible path to the image.
This combination of Database and shared publicly-accessible storage is probably a good mix, giving you the best of both worlds. The Database also means that if you need to move / change or bulk delete images in future (perhaps deleting all images uploaded by an author who is deleting their account), you can perform an efficient Database query to gather metadata, followed by updating / changing the stored images at the S3 locations the Database says they exist.
You say you want to display the images on a web page. This combination means that the application server can query the database efficiently for the image selection you want to show (including restricting by author, pagination, etc), then generate a view containing images referring to the correct CDN path. It means viewing the images is also quite efficient as you combine dynamic content (the page upon which the images are shown) with static content (the image themselves via the CDN).
CDNs may be a good option for you.
You can store the link to the images along with the image information in your database.
I'd like to just use .json files to store data, rather than using a database. Many simple sites have little data, and reading/writing to a file (that can be added to version control) seems adequate, and eliminates the need for database versioning / deployment logistics.
npm: node-store
Here's one way to do it, yet I'd need to implement all kinds of query functionality.
I'm really unfamiliar with CouchDB. From the little I've read, it looks like it might use files to store the JSON data, but it might use some kind of disk storage. Can someone shed some light on this?
Does CouchDB store its JSON in text-based files that can be added to version control (git)?
Does anyone know of another text-based storage system with some query functionality?
CouchDB is a full fledged database. The value that gives you above simply using file based storage is additional indexing. Ie., if you do file based then you can either only do key based look ups (the file name) or build your own secondary indexing methodology (symlinks or whatever). Now you're in the database building business instead of the app building business, which is silly because your entire premise seems to be simplicity and focusing on your app.
Also, keep in mind that when you have many (even just 2) people causing writes to your file(s), then you're going to run into either file system locking problems or users overwriting one another.
You're correct though, if you only have a few pieces of information then a single JSON file - basically a config file - is far easier than a database. Especially if people are only reading from the file.
Also, keep in mind that there are Database-as-a-Service solutions that remove the need for DIY install/configure/maintenance/administration. One of them is Cloudant which is based on CouchDB, is API compatible, contributes back, etc. (I work at Cloudant).
Does anyone know of another text-based storage system with some query functionality?
You can use ueberDB module with Dirty file storage.
As far as I remember, this storage just appends your data to the same text file over and over again, so if you really have small dataset, it'll work just fine.
If you data will grow too much, you can always change storage while using the same module.
The further I go with my blog the more problems I hit :) Can anyone tell me what is the best way to save images. I store the data in mongodb, should I save images in there as well or should I use local file system? Thank you
use file system to store images, free up db resources to serve data. for bigger site images should really use CDN.
Store them in MongoDB using GridFS. That way you're not limited by file size, the images are easily shared between multiple app servers, and the images are naturally backed up with the rest of your MongoDB data.
I use Node.js for my project.
Should I store an image in local file system, or should I store it in MongoDB?
Which way is more scalable?
The most scalable solution is to use a shared storage service such as Amazon's S3 (or craft your own).
This allows you to scale horizontally a lot easier when you decide to add machines to your application layer, as you won't have to worry about any migration nightmares.
The basic idea behind this is to keep the storage layer decoupled from the application layer. So using this idea you could create a node.js process on a separate machine that accepts file uploads then writes them to disk.
If you're designing a performance sensitive system, use file system to store your images no doubt.
You can find the performance compare from this blog:
http://blog.thisisfeifan.com/2013/12/mongodb-gridfs-performance-test.html
Actually, you can find the recommended MongoDB GridFS use cases here:
https://docs.mongodb.com/manual/core/gridfs/#when-to-use-gridfs
I would use GridFS to take advantage of sharding but for best performance I would use filesystem with nginx.