I am creating a web app for some co-working. I have text with assets (most of them pictures in print quality, say 5MB, all in all about 5GB per month).
I am going to host this on amazon's cloud using EC2 instances for a node.js server and a mongodb with attached block storage.
The assets are private to an object so not everyone will have access to it.
How should i handle the assets? Save them as binarys in the database or load them up to S3? (or any other amazon service)
Does somebody have experience on this? Or maybe some helpful links. Thanks in advance
I would probably store the assets on S3 without public access, then you can grant access to authorized users by generating temporal signed urls from your webservers when needed.
This way you can leverage your servers complexity by handing over the storage dirty work to S3, and you can still have your files accessed only by who has access to them.
If you need to do access control there are lots of things you could do but the most obvious would be to server the asserts through your web server and have it implement the access control logic desired. Your app could proxy through the source object from S3 or MongoDB GridFS. If you are already using MongoDB, in this particular case I would use GridFS unless you want some of the cost saving features of S3 such as reduced redundancy storage.
Related
I'm trying to make an architecture for a data lake, I already generated my CSV, txt, and Avro files they are in an On-Premise machine and I want to upload them to Google Cloud Storage, but I see that I have to go through the public internet and I don't want to that.
What options do I have to make as safe as possible?
I was trying to make a Compute Engine environment to upload the files here through SFTP and then moved them to Cloud Storage, but that will make my cost to go up.
I think this the use case of Google VPC. The key benefit provided on the product description is:
For on-premises, you can share a connection between VPC and on-premises resources with all regions in a single VPC
If you look deeper into documentation you may find document Configuring Private Google Access for on-premises hosts. Some information maybe be found here as well. This might be a good option for you.
I'm looking for the best way (or a way) to upload large files (1GB+) from the client side of my app to my Google Cloud Storage bucket.
The bucket is private and I'm currently trying to send the file to my node/express server and stream it to my GCS bucket. I think I'm running into issues due to file size constraints on the server though. Works fine for smaller files, but the 1GB file I'm using to test it isn't getting through.
These are the alternate methods I'm exploring:
file splitting client side and then reassembling server-side prior to sending to GCS
create a temporary service account through the Google IAM API (GCS write access), send this to the client to be able to upload the file, delete account after upload confirmed (not ideal since it exposes the service account)
???
Just looking to get pointed in the right direction at this point.
The most direct transfer would be from the client directly to GCS. You'd probably want to send the client a signed URL that they would use to start and then carry out the upload. You'd likely want to use a resumable upload unless you expect your customers to have fast enough Internet not to need to bother.
It would probably be a good idea to have clients upload to some staging GCS bucket and then notify your app that the upload is complete, at which point your app would then copy it to the final bucket (which is an instant operation if the objects are in the same region and storage class), although that's not necessarily required.
I would probably try to avoid streaming data through your app unless you wanted to transform it in some way or want to use multiple cloud providers transparently.
Creating one-off service accounts is probably not a great idea. I'm not sure what the limit on those is off-hand, but you may run into issues scaling that up.
I am building an asp.net Web App hosted with azure. I have decided to use file system to store images instead of in the database, with the image path's stored in the database. The images are added by users, and they can add a lot at once, up to 100.
My question is, what are the limits on file system storage of images? Currently, they are all just being stored in a single directory. Eventually there will be thousands (possibly hundreds of thousands) images in a single directory. Will this have a drastic effect on performance when these images are embedded in a page? None of the images are bigger than 1 mb. Would it be better practice to make sub-directories?
I strongly recommand to use Azure Blob storage to store file which is the best way to store static content in Azure.
And, if you have to scale your application in the future (web farm, azure web app or whatever that has more than one VM instance), storing files on the server is not a best practice.
You can find some "best practices" about static content like images on this page: https://msdn.microsoft.com/en-us/library/dn589776.aspx
And the Azure Storage documentation that will help you to get started: https://azure.microsoft.com/en-us/documentation/services/storage/
Hope this helps,
Julien
we have 3 servers which are in load balancer.Whenever I'm saving a image it saves only in one server I want to access that image from whenever I call the load balancer url?
Can any one please advise me how to achieve this?
I would recommend against saving images on your actual web servers. Instead, find a centralized solution, possibly using something like AWS or Azure storage, which would give you the ability to safely store your uploads somewhere without depending on your local filesystem. You can configure these storage accounts so that the files are publicly accessible or not, depending on what you're doing with them.
If you have to use a local filesystem, it should be something shared, like a file server on your network that is designed to be accessed from multiple servers. But AWS and Azure storage is ridiculously cheap (a few cents per GB per month).
I'm in the process of deciding which technology to use for a project. The project will store large amounts of documents (Word, PDF etc) and I'm trying to figure out the best way of storing them. So far, I've come up with:
Standard hosting, and using the file system
Standard hosting, use Full Text Search and store documents in SQL Server
Use Azure Blobs.
Under no circumstances can the documents be visible to anyone. Only certain, authorised people should be able to view documents. Can anyone point me in the direction of how you can secure the documents so that you can't just point a browser to it and it will be visible?
Windows Azure blobs are a great place to store a lot of documents. By default, blobs can only be retrieved by someone who has the access key for the account. (The headers on API calls have to be signed with that key, so there's a cryptographic guarantee that unauthorized third-parties can't access them.)
I assume you'll host a front-end of some sort that gives access to authorized users, and this front-end will act as a proxy to the files themselves.
Don't store your documents in a web server directory. It's as simple as that. Why go through the all the efforts of configuring a web server when you don't want the files on the web in the first place?