Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
What would be the best practice to upload files in MongoDB. I recently came accross Multer. It's easy to use, but it uploads files in the file system instead of uploading directly into the MongoDB.
Multer is a high-level wrapper for Busboy and unfortunately doesn't callback with a Stream unless you're writing a StorageEngine for it. The reason you want a stream so badly is because otherwise you would have to buffer the whole file in-memory of your Node process before being able to do anything with it. Streams are much more efficient and allows you to stream the data somewhere meanwhile you're receiving it, only buffering in-memory in case you cannot pipe the data somewhere at equal rate as you're receiving it.
Combining a custom StorageEngine with gridfs-stream would allow you to write the data to GridFS in real-time as you receive the data (e.g the user is still uploading).
I've found two GridFS Storage Engines for Multer:
https://github.com/ISMAELMARTINEZ/gridfs-storage-engine
https://github.com/arjandepooter/multer-gridfs
The latter doesn't seem to have any docs but it's still easy to use just looking through the source.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 days ago.
Improve this question
I'm creating a website using MERN Stack the website will help people to create videos that create a voice over and subtitle on video.
The user would have to upload a video and write some text that would be sent to server and return for them the video they uploaded with subtitle and voice over of the content they have wrote.
so I have created a website that produce this functionality but what I got stuck with is how should I handle the files (videos uploaded, audio files, subtitles and final videos generated)
my current implementation handle everything on the production server as the client prefer that everything stay on the server (and he doesn't mind to through extra bucks for that)
the website is expected to handle between 1000-3000 request per day but I'm not sure with this load could a single server handle all of that? or should I get a server for handling the files and the other for my functionality?
I'm currently finishing my code to test this but I thought maybe someone here could help me to think about it differently or maybe I'm doing something naive and I should insist on using a third party service to handle the database and the files.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
TL;DR Is is possible to proxy torrent larger than available local disk while piping it to outbound stream ?
According to BitTorrent spec , all torrents are stored as pieces of equal length , I want to write a node app could pipe the torrent pieces to a http upload stream , does any library provide such functionality ?
All the implementation I have found downloads the whole file to local storage then propagate it further which can cause problems when running on small disk and large files .
Bittorrent is designed for random access to keep data available via the rarest-first strategy. See Section 2.4.2 of the bittorrent econ paper. While it is possible to operate it in a streaming manner anyway this generally isn't recommended and certainly shouldn't be the default, otherwise performance could severely degrade for all swarm members or content could even become unavailable.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I am building some kind of video streaming web app using node.js/express and MongoDB. But I am facing an issue related to where to store the mp4 files that my clients will upload to my back-end. I am not sure if MongoDB is capable of storing large files(in the GB order) and my current idea is to keep the files on a directory and then keep track of each file path on MongoDB. Is this a good idea or is there a better method to do so?
My advice, use
s3.amazonaws.com
Yes, it's way better to store only a path inside a MongoDB instead of storing directly the video file inside the DB. Because your DB will grow up so fast if you did that. The disk space taken by both solutions is the same, but overloading your DB with these files will just result in a slower DB result
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am working on a project, where each user can post many images. We use Node.js with Express and MongoDB as the database. I was wondering which way would be better in terms of speed and scalability:
Storing images in Mongo GridFS
Or
Storing images on Amazon S3, and paths of images on MongoDB. Then retrieve images using paths.
Any thoughts are appreciated !
Thank you,
This is like comparing Go vs Node.js. There's no better general solution.
Each might have their own advantages and solutions. MongoDB is more like the DIY solution, and Amazon S3 is the managed solution. With MongoDB you have to scale it yourself. I can say S3 will be faster initially and it's already scaled by Amazon, and probably cheaper(S3 is cheaper than EBS). You can get a lot of servers with huge amounts of RAM and MongoDB will definitely be faster. Also if the MongoDB instance is in the same instance as your App, you will have less latency.
Also check this question: MongoDB as static files provider?
And this: What are the advantage of using MongoDB GridFS vs Amazon S3 to store assets for a product with MongoDB as the database backend?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am developing a website which uses a lot of images.
The images get manipulated very often (every few seconds, by the clients). All images are on a linux server. It is also possible that two clients try to change an image at the same time.
So my question is: should I put the images into a database or just leave them in a folder (how does the OS handle the write-write-collisions?)?
I use node.js and mongoDB on the server.
You usually store the reference to the file location inside of the database. As far as write-write collisions In most whoever has the file open first gets it however it mostly depends on the OS that you are working with. You will want to look into file locking. This wikipedia article gives a good overview.
http://en.wikipedia.org/wiki/File_locking
It is also considered good practice in your code to check and notify the user if the file is in use if write collisions are likely to occur.
I suggest you store your images within the MongoDB using the GridFS file system. This allows you to keep images and their metadata together, it has an atomic update and two more advantages, if you are using replica sets for your database:
Your images have the same high availability as the rest of your data and get backed-up together
You can, eventually, direct queries to secondary members of the set
For more information, see
http://docs.mongodb.org/manual/applications/gridfs
http://docs.mongodb.org/manual/replication/
Does this help?
Cheers
Ronald