Im having photo albums on my site which will be a vital part of the website and most users will upload images, lets say i get 10.000 visitors with an avarge upload of 10 images =100.000 images, should each visitor get his own folder MKDIR() or should i just store all the images in the same folder? there is no risk for images having the same name or anything
but i just wonder what is best practice
By putting too many files in one folder, you're risking hitting limits of the filesystem of use. You didn't mention what filesystem is going to store all those images, but indeed all of them have some sort of limits (or it may slow things down). You'd better put each users images in their own folders.
Related
Is anyone using a cloud to provide access to their RETS MLS images so that the pubic can access at will w/o password? I currently have the images storing on my server for direct public access through my website, however need lots more space and just not sure about how to get the images to the web and make them available to view through my website. I download them each 15 min from the RETS server and store them on my server. So, what I need is the procedure to convert this storage to a cloud, knowing there may be some password requirements to access the cloud.
We are using Azure Storage (https://learn.microsoft.com/en-us/azure/storage/) to cache all of the images for one of our products. We sort of "lazy load" the images in there. If a request is made for an image, we pull it from our cloud storage (where the image is made public) and stream it from there much faster than over RETS media request. If we ever have to make an on demand request from a RETS server for the image we immediately cache it. We'll also pre-fetch images for slower MLS's. Images have been a severe performance impact on working with RETS systems for us.
This also lets us do some cool stuff like resizing the image on the fly pretty quickly too for our customers so they don't have to waste bandwidth downloading full size images if that's not what they want.
Here is a link to our open source SDK on how our customers end up using the image service.
https://github.com/timitek/getrets-php-sdk#imageurl
Amazon S3 Buckets is a better solution. We have around 18 TB of data in it. And also like you said " each image needs it own directory", we have millions of directories for 400+ MLSs.
Everything works fine and there is no delay as well. Its scalable too.
Note: We are getting binary raw data for MLS images and that we are directly writing to S3 and making the image urls.
I have developed a large web application with NodeJS. I allow my users to upload multiple images to my Google Cloud Storage bucket.
Currently, I am storing all images under the same directory of /uploads/images/.
I'm beginning to think that this is not the safest way, and could effect performance later down the track when the directory has thousands of images. It also opens up a threat since some images are meant to be private, and it could allow users to search for images by guessing a unique ID, such as uploads/images/29rnw92nr89fdhw.png.
Would I be best changing my structure to something like /uploads/{user-id}/images/ instead? That way each directory only has a couple dozen images. Although, can a directory handle thousands of other subdirectories without suffering performance issues? Does Google Cloud Storage happen to accomodate for issues like this?
GCS does not actually have "directories." They're an illusion that the UI and command-line tools provide as a nicety. As such, you can put billions of objects inside the same "directory" without running into any problems.
One addendum there: if you are inserting more than a thousand objects per second, there are some additional caveats worth being aware of. In such a case, you would see a performance benefit to avoiding sequential object names. In other words, uploading /uploads/user-id/images/000000.jpg through /uploads/user-id/images/999999.jpg, in order, in rapid succession, would likely be slower than if you used random object names. GCS has documentation with more on this, but this should not be a concern unless you are uploading in excess of 1000 objects per second.
A nice, long GUID should be effectively unguessable (or at least no more guessable than a password or an access token), but they do have the downside of being non-revocable without renaming the image. Once someone knows it, they know it forever and can leak it to others. If you need firm control of your objects, you could keep them all private and visible only to your project and allow users to access them only via signed URLs. This offers you the most flexibility and control, but it's also harder to implement.
I am working on an App that will have a bunch of pictures and videos the user will need to have. The app's mantra is having these videos and pictures available on demand and on the go. However, I am concerned about the app being too fat (big), in essence turning off some customers who prioritize size over substance. The preference has always been to have the customer be able to access these videos from anywhere regardless of 3g/wifi (i.e. local on the device). What are the really good options? Assuming I choose to elect to go the non-local route (i.e. using wifi/3g), what are my options there as well? Please advice. The size of the videos/pics could be up to 1.5GB.
The usual approach seems to be to download and store the videos on demand. A lot of the apps I've seen allow the user to manage the stored videos themselves so that they can remove videos if they aren't going to use them again.
Some background info:
I am developing a website, on which users will have profiles and will be able to upload profile pictures. I am not very experienced, and do not have lots of time available on my hands (as I do it in my free time). Yet I am aware that uploads can leave a huge security gap for any website if implemented incorrectly.
My actual question:
Is it safe to limit images to, say .gif, .jpg and .png extensions, knowing the server can only parse php files (don't know if I'm using the terminology correctly)? Or is there some other security risk in doing this? Note, I also store the files in a private directory after renaming them with random numbers, and pass them through a php file whenever it is necessary.
Additional safeguards you could use:
limit the size of the uploaded file
What are the major security issues to consider when accepting image uploads, beyond the normal stuff for all HTTP uploads?
I'm accepting image uploads, and then showing those images to other users.
How should I verify, for example, that the uploaded image is actually a valid image file? Are there any known vulnerabilities in viewers that are exploitable by malformed image files for which I should be concerned about accidentally passing along exploits? (Quickly googling seems to show that there once was in IE5/6.)
Should I strip all image metadata to help users prevent unintentional information disclosures? Or are there some things that are safe and necessary or useful to allow?
Are there any arcane features of common image formats that could be security vulnerabilities?
Are there any libraries that deal with these issues? (And/or with other issues like converting progressive JPEGs to normal JPEGs, downsampling to standardize sizes, optimizing PNGs, etc.)
Some things I learned recently from a web security video:
The nuclear option is to serve all uploaded content from a separate domain which only serves static content - all features are disabled and nothing important is stored there.
Considering processing images through imagemagick etc. to strip out funny business.
For an example of what you are up against, look up GIFAR, a technique that puts a GIF and Java JAR in the same file.
The risk of propogation of bugs inside image formatters isn't "exactly" your problem, but you can help anyway, by following the general practice of mapping ".jpg" to your executable language, and processing each image manually (in this way you can do refer checks as well).
You need to be careful of:
People uploading code as images (.jpg with actual c# code inside)
any invalid extensions (you check for this)
People trying to do path-related attacks on you
The last one is what you'll need to be wary of, if you're dynamically reading in images (as you will be, if you follow my first bit of advice).
So ensure you only open code in the relevant folder, and, probably more importantly, lock down the user that does this work. I mean the webserver user. Make sure it only has permissions to read from the folder you are working in, and other such logical things.
Stripping metadata? Sure why not, it's quite polite of you, but I wouldn't be nuts about it.
Your biggest risk is that an attacker tries to upload some type of executable code to your server. If the uploaded file is then browsable on the web, the attacker may be able to cause the code to run on your server. Your best protection is to first save the uploaded file to a non-publicly browsable location, try to load it as an image in your programming language and allow it if it can be successfully parsed as an image. A lot of the time people will want to resize the image anyway so really doing this is no extra work. Once the image is validated, you can move it into the publicly browsable area for your web server.
Also, make sure you have a limit on file upload size. Most platforms will have some kind of limit in place by default. You don't want a malicious user filling up your disk with an endless file upload.
One of the vulnerabilities I know of is a "WMF backdoor". WMF is "Windows Metafile"--a graphical format rendered by Windows GDI library. Here's wikipedia article.
The attacker is capable to execute arbitrary code on user's machine. This can happen when user just views the file via the browser, including, but not limited to Internet Explorer. The issue is said to be fixed in 2006.