Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
We have a web platform that allows users to upload and download their data stored in a storage connected to Linux via NFS.
currently about 350GB data is stored.
I want to show each user their current storage usage on the web in real time like Google drive or Dropbox when you update a file, the remaining disk space is shown right away.
My idea was to run du command on Linux and send an update query to DB but du takes so long.
Is there any better idea that I can do this on real time?
Thanks
Given that users are only allowed to upload or download data, the sizes of files stored will not change while on your storage server since "Edits" are not possible.
If you store the name and location of the file upon insert into a database, it would be wise to include the size of the file uploaded too.
In php, you can do this with filesize().
This way, if you need to fetch the size of all the users files, all you need to do is grab all records owned by them, and add the file size column. This would give you a precise size of the users files, even if the files are located across different servers.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I was wondering that how do storage solutions like S3 or Google Drive check whether their storage platform is being abused for the storage of malicious content?
e.g. if someone uploads a password protected zip file to their servers, I don't see a way on how they can verify it.
For unencrypted files, I can understand some sort of file parser could work. But if someone uploads a password protected file, the only way to see/verify the contents is try to brute force your way into it (ignoring the moral obligations for the organisation to not do that).
So, how do these companies/solutions verify the kind of data that is being uploaded on their platforms?
There isn't technical solution, but on legal solution. They say: "We are only a service provider, not a content provider. We aren't responsible of the illegal use of our services".
This stand has been the same with Youtube, where you was able to upload content with copyright without issue with Google (but with the owner of the copyright). Now, it has changed and Youtube performed check, but it was the same legal principle.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I am building some kind of video streaming web app using node.js/express and MongoDB. But I am facing an issue related to where to store the mp4 files that my clients will upload to my back-end. I am not sure if MongoDB is capable of storing large files(in the GB order) and my current idea is to keep the files on a directory and then keep track of each file path on MongoDB. Is this a good idea or is there a better method to do so?
My advice, use
s3.amazonaws.com
Yes, it's way better to store only a path inside a MongoDB instead of storing directly the video file inside the DB. Because your DB will grow up so fast if you did that. The disk space taken by both solutions is the same, but overloading your DB with these files will just result in a slower DB result
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed last year.
Improve this question
I have requirement to unzip a file automatically whenever it is pasted in specified folder in onedrive (ex: zipfolder). So i basically want to know that, is it possible to run any program in onedrive that can fulfill my requirement.
If you were really dedicated to the task, what you could do is set up a web service with a webhook attached to the folder that you are wanting to do this in. When the webhook fires, your service can connect to OneDrive, download the file, unzip, and re-upload the contents.
It may not be possible to run a program from onedrive...One drive is a cloud based server or rather storage,You may be forced to download it and install it from your hard disk
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
There are lot of photo sharing applications out there, some are making money and some don't. Photo sharing takes lot of space, so I doubt where they host these! Rich services probably using Amazon or their own server, but the rest? Do they have access to any kind of free service? Or they have purchased terabytes from their web host?
AWS S3 is what you are generally referring to. The cost is mainly due to the reliability they give to the data they store. For photo-sharing, generally this much reliability is not required (compared with say a financial statement).
They also have other services like S3 RRS (Reduced redundancy), and Glacier. They are lot cheaper. Say those photos not accessed for a long time may be kept on Glacier (it will take time to retrieve, but cheap). RRS can be used for any transformed images (which can be re-constructed even if lost) - like thumbnails. So these good photo-sharing services, will do a lot of such complicated decisions on storage to manage cost.
You can read more on these types here : http://aws.amazon.com/s3/faqs/
There is also a casestudy of SmugMug on AWS. I also listened to him once, where he was telling about using his own hard-disks initially to store, but later S3 costs came down and he moved on to AWS. Read the details here:
AWS Case Study: SmugMug's Cloud Migration : http://aws.amazon.com/solutions/case-studies/smugmug/
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I am developing a website which uses a lot of images.
The images get manipulated very often (every few seconds, by the clients). All images are on a linux server. It is also possible that two clients try to change an image at the same time.
So my question is: should I put the images into a database or just leave them in a folder (how does the OS handle the write-write-collisions?)?
I use node.js and mongoDB on the server.
You usually store the reference to the file location inside of the database. As far as write-write collisions In most whoever has the file open first gets it however it mostly depends on the OS that you are working with. You will want to look into file locking. This wikipedia article gives a good overview.
http://en.wikipedia.org/wiki/File_locking
It is also considered good practice in your code to check and notify the user if the file is in use if write collisions are likely to occur.
I suggest you store your images within the MongoDB using the GridFS file system. This allows you to keep images and their metadata together, it has an atomic update and two more advantages, if you are using replica sets for your database:
Your images have the same high availability as the rest of your data and get backed-up together
You can, eventually, direct queries to secondary members of the set
For more information, see
http://docs.mongodb.org/manual/applications/gridfs
http://docs.mongodb.org/manual/replication/
Does this help?
Cheers
Ronald