Is it a good idea for the client to communicate directly to Azure Blob Storage? If we do it this way, how will we perform server side validation?
For example. Say I want to use blob storage to manage my uploaded images. But I want to prevent users from uploading certain image types and also files that are larger than 10 mb. How can I implement server side validation for this?
You have to handle the files in your back-end, you cannot allow them to upload the files directly in this case.
So you take the file in, validate it, and then upload it to Storage.
And don't give the keys to the client.
Related
I am working on a project where we have to store some audio/video files on Azure Blob Storage and after the file is uploaded we need to calculate some price on the basis of the length of the file in minutes. We have an Angular frontend and the idea was to upload the file directly from the frontend, get the response from Azure with the file stats , then call a backend API to put that data in the database.
What I am wondering is what are the chances of manipulation of data in between getting the file data back from Azure and calling our backend API. Is there any chance the length could be modified before sending it to our API?
One possible solution would be to make use of Azure Event Grid with Blob integration. Whenever a blob is uploaded, an event will be raised automatically that you can consume in an Azure Function and save the data in your database.
There's a possibility that a user might re-upload same file with different size. If that happens, you will get another event (apart from the original event when the blob was first created). How you handle with updates would be entirely up to you.
I'm looking for the best way (or a way) to upload large files (1GB+) from the client side of my app to my Google Cloud Storage bucket.
The bucket is private and I'm currently trying to send the file to my node/express server and stream it to my GCS bucket. I think I'm running into issues due to file size constraints on the server though. Works fine for smaller files, but the 1GB file I'm using to test it isn't getting through.
These are the alternate methods I'm exploring:
file splitting client side and then reassembling server-side prior to sending to GCS
create a temporary service account through the Google IAM API (GCS write access), send this to the client to be able to upload the file, delete account after upload confirmed (not ideal since it exposes the service account)
???
Just looking to get pointed in the right direction at this point.
The most direct transfer would be from the client directly to GCS. You'd probably want to send the client a signed URL that they would use to start and then carry out the upload. You'd likely want to use a resumable upload unless you expect your customers to have fast enough Internet not to need to bother.
It would probably be a good idea to have clients upload to some staging GCS bucket and then notify your app that the upload is complete, at which point your app would then copy it to the final bucket (which is an instant operation if the objects are in the same region and storage class), although that's not necessarily required.
I would probably try to avoid streaming data through your app unless you wanted to transform it in some way or want to use multiple cloud providers transparently.
Creating one-off service accounts is probably not a great idea. I'm not sure what the limit on those is off-hand, but you may run into issues scaling that up.
I have an MVC application which allows users to sell items and upload images associated with each item. I am using Azure Blob storage to store the images. Once the image is stored in Azure, the MVC backend might retrieve it and resize it (usually shrink it) before sending it to the client, depending on requirements. Is there anyway I can request a file from Azure and have Azure resize it for me? This would save on the expense of sending a large image file to my MVC controller, which I then resize and stream to the client. Of course, I know I could save multiple copies of each image in Azure, each with the size I might need.
No, Azure Blob storage is not designed process the content of your files in any way.
Simple binary upload and download.
Can anyone please advise me if it is possible to directly upload a file to a blob container without routing it through my web server? I'm thinking some sort of client-side JS/jQuery script or a 3rd party upload module that streams the file directly to the blob container.
With Amazon S3 I used a component called Flajaxian Direct Uploader to achieve this.
I have the need to upload zip files to an Azure blob container that are 50 mb - 200 mb in size and routing via the web server is slower and consumes additional bandwidth.
Yes, it is possible. This can be achieved by having the client contact your web server and ask for a Shared Access Signature with (w)rite only access and limited expiry. Your client can then use the simple REST API to upload the blob. The trick here is that if your blob is bigger than 64mb, you must use use the PUT block and PUT block list option. The latter is not as straightforward for a Javascript client. If your client can use curl, it works well.
Reference:
PUT Blob
PUT Block
PUT Block list
I am in the middle of developing a cloud server and I need to store HDF files ( http://www.hdfgroup.org/HDF5/ ) using blob storage.
Functions related to creating, reading writing and modifying data elements within the file come from HDF APIs.
I need to get the file path to create the file or read or write it.
Can anyone please tell me how to create a custom file on Azure Blob ?
I need to be able to use the API like shown below, but passing the Azure storage path to the file.
http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c
These files i am trying to create can get really huge ~10-20GB, So downloading them locally and modifying them is not an option for me.
Thanks
Shashi
One possible approach, admittedly fraught with challenges, would be to create the file in a temporary location using the code you included, and then use the Azure API to upload the file to Azure as a file input stream. I am in the process of researching how size restrictions are handled in Azure storage, so I can't say whether an entire 10-20GB file could be moved in a single upload operation, but since the Azure API reads from an input stream, you should be able to create a combination of operations that would result in the information you need residing in Azure storage.
Can anyone please tell me how to create a custom file on Azure Blob ?
I need to be able to use the API like shown below, but passing the
Azure storage path to the file.
http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c
Windows Azure Blob storage is a service for storing large amounts of unstructured data that can be accessed via HTTP or HTTPS. So from application point of view Azure Blob does not work as regular disk.
Microsoft provides quite good API (c#, Java) to work with the blob storage. They also provide Blob Service REST API to access blobs from any other language (where specific blob storage API is not provided like C++).
A single block blob can be up to 200GB so it should easily store files of ~10-20GB size.
I am afraid that the provided example will not work with Windows Azure Blob. However, I do not know HDF file storage; maybe they provide some Azure Blob storage support.