I want to create a website to provide PDF notes to students.
I am planning to upload pdf on my googledrive and want to serve those pdf using my website.
I am using MongoDb As Database and my plan is to read link of pdf from my google drive and want to save it on collection of mongodb so that it will render on my website dynamically.
Is there any convenient way to do this automatically rather than copy pasting each link.(because it will be a time draining stuff and we have to upload 1000 of pdfs).
If this method is not possible Please suggest me some better way to do this work using nodejs and mongodb.
If using Google Drive is not a must, I recommend you to use MongoDB's GridFS. GridFS allows you to store documents that are higher than 16MB. In this way, you do not need to manage/update document links, because MongoDB will be already in charge of metadata and the document itself.
If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed.
When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory.
When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities.
This tutorial might be the starting point.
Related
I want to store as a file in server. It may be in the disk and store the path of that file in Postgres table. I know storing the file in the disk in no more Sync up in the Postgres. It may happen that someone deletes that file on the disk and my column still referencing it. So, I go deep down and find that Postgres has some feature which allows users to save the file in the disk and create a symbolic link and stored in the database itself which allows to sync up. It also works as a backup. I also want to control the file from accessing it based on some condition. How can we do it in Postgres? Any help is appreciated.
You can either store files in the filesystem and put their file names or paths into the database (but that way you will have to remember about those files when you do backups or migrate data) or you can let Postgres store the files in the database using BLOB, bytea or text data types.
See the documentation:
https://www.postgresql.org/docs/current/static/largeobjects.html
https://www.postgresql.org/docs/8.4/static/datatype-binary.html
https://www.postgresql.org/docs/8.4/static/datatype-character.html
If you store files in the database then they will be backed up automatically, you will have ACID and other database goodies.
If you store files in the file system then you are pretty much on your own, but the performance characteristics may have some advantages for certain use cases.
There is also a third option of storing files in services like S3, Cloudinary, Uploadcare etc. and storing their IDs (usually UUIDs) in the database - which is a pretty common use case for certain data, especially user-uploaded photos etc.
See this for more info:
https://wiki.postgresql.org/wiki/BinaryFilesInDB
I need to add images to my mongoDB using Node and Express.
I am able to normal data in it by running the mongo shell. But I cannot find any method to add images to it.
Can anybody help?
Please don't do this. Databases are not particularly well suited to storing large bits of data like images, files, etc.
Instead: you should store your images in a dedicated static file store like Amazon S3, then store a LINK to that image in your MongoDB record.
This is a lot better in terms of general performance and function because:
It will reduce your database hosting costs (it is cheaper to store large files in S3 or other file services than in a database).
It will improve database query performance: DBs are fast at querying small pieces of data, but bad at returning large volumes of data (like files).
It will make your site or application much faster: instead of needing to query the DB for your image when you need it, you can simply output the image link and it will be rendered immediately.
Overall: it is a much better / safer / faster strategy.
Simply, I need to build an app to store images for users. So each user can upload images and view them on the app.
I am using NodeJS and Mongo/Mongoose.
Is this a good approach to handle this case:
When the user uploads the image file, I will store it locally.
I will use Multer to store the file.
Each user will have a separate folder created by his username.
In the user schema, I will define a string array that records the file path.
When user needs to retrieve the file, I will check the file path, retrieve it from the local disk.
Now my questions are:
Is this a good approach (storing in local file system and storing path in schema?
Is there any reason to use GridFS, if the file sizes are small (<1MB)?
If I am planning to use S3 to store files later, is this a good strategy?
This is my first time with a DB application like this so very much appreciate some guidance.
Thank you.
1) Yes, storing the location within your database for use within your application and the physical file elsewhere is an appropriate solution. Depending on the data store and number of files it can be detrimental to store within a database as it can impede processes like backup and replication if there are many large files
2) I admit that I don't know GridFS but the documentation says it is for files larger than 16MB so it sounds like you don't need it yet
3) S3 is a fantastic product and enables edge caching and backup through services and many others too. I think your choice needs to look at what AWS provides and if you need it e.g. global caching or replication to different countries and data centres. Different features cause different price points but personally I find the S3 platform excellent and have around 500G loaded there for different purposes.
I am working on application in which users will upload huge number of images and i have to show those image webpage
What is the best way to store and retrieve images.
1) Database
2) FileSystem
3) CDN
4) JCR
or something else
What i know is
Database: saving and retrieving image from database will lead to lot of queries to database and will convert blob to file everytime. I think it will degrade the website performance
FileSystem: If i keep image information in database and image file in filesystem there will be sync issues. Like if i took a backup of the database we do have take the backup of images folder. ANd if there are millions of files it will consume lot of server resources
i read it here
http://akashkava.com/blog/127/huge-file-storage-in-database-instead-of-file-system/
Another options are CDNs and JCR
Please suggest the best option
Regards
Using the File System is only really an option if you only plan to deploy to one server (i.e. not several behind a load balancer), OR if all of your servers will have access to a shared File System. It may also be inefficient, unless you cache frequently-accessed files in the application server.
You're right that storing the actual binary data in a Database is perhaps overkill, and not what databases do best.
I'd suggest a combination:
A CDN (such as AWS CloudFront), backed by a publicly-accessible (but crucially publicly read-only) storage such as Amazon S3 would mean that your images are efficiently served, wherever the browsing user is located and cached appropriately in their browser (thus minimising bandwidth). S3 (or similar) means you have an API to upload and manage them from your application servers, without worrying about how all servers (and the outside world) will access them.
I'd suggest perhaps holding meta data about each image in a Database however. This means that you could assign each image a unique key (generated by your database), add extra info (file format, size, tags, author, etc), and also store the path to S3 (or similar) via the CDN as the publicly-accessible path to the image.
This combination of Database and shared publicly-accessible storage is probably a good mix, giving you the best of both worlds. The Database also means that if you need to move / change or bulk delete images in future (perhaps deleting all images uploaded by an author who is deleting their account), you can perform an efficient Database query to gather metadata, followed by updating / changing the stored images at the S3 locations the Database says they exist.
You say you want to display the images on a web page. This combination means that the application server can query the database efficiently for the image selection you want to show (including restricting by author, pagination, etc), then generate a view containing images referring to the correct CDN path. It means viewing the images is also quite efficient as you combine dynamic content (the page upon which the images are shown) with static content (the image themselves via the CDN).
CDNs may be a good option for you.
You can store the link to the images along with the image information in your database.
The further I go with my blog the more problems I hit :) Can anyone tell me what is the best way to save images. I store the data in mongodb, should I save images in there as well or should I use local file system? Thank you
use file system to store images, free up db resources to serve data. for bigger site images should really use CDN.
Store them in MongoDB using GridFS. That way you're not limited by file size, the images are easily shared between multiple app servers, and the images are naturally backed up with the rest of your MongoDB data.