We are having a VM hosting our web application where its users upload big files to their profiles -mainly they are 3d models- via the Web API of the web app.
What I'm looking to do is to find a way to handle these long running uploading jobs/process efficiently to some place rather than the VM where I'm hosting the application.
I know there are some Azure methods but I want to make it correct. Which is the most efficient way to do that ? Worker Role, a web page running a scripts to upload it to storage or Web Jobs over a queue maybe ?
The function which uploads the file also is having some other processes like generating thumbnails, storing the data to sql and etc.
Related
I heard a lot that Azure Web Applications uploaded content such as images or any files should be stored in Azure Storage service, not in the app File System.
But I would like to keep the solution simple and store those files into local file system.
Does storing images or any files on the application's local file system hampers somehow the application deployment with more than one instance?
After researching a lot, I understand that, unlike Amazon Beanstalk, all instances of a Web App share the same file storage. despite of ecah apps runs into a different VM, they file system is the same.
The best way to think about it is that all the instances of your app map to the same network drive share that have your files.
when you spin up 2 or more instances the files in the file are using that same storage based file system even if you only have one instance.
You can see that easily by dropping a file (e.g. via FTP), and seeing it reflected instantly in all instances.
Sources: Microsoft, This question and this question
By storing files on the Web application, you're limiting your ability to scale.
The web server/app should do one thing: process your request and output the HTML. Everything else should be handled elsewhere if you truly want to take advantage of cloud computing. So for this instance you should store any files off the web app.
Now if you want to keep this as simple as possible and you're not overly concerned with achieving scale, there's really nothing wrong with your approach. It's my understanding that Web Apps, don't actually spin up new virtual machines for you, or if they do, they replicate exactly what is on the VM. For instance think about this -- if you had all your files stored on one VM and you spun up two new ones, you'd have to copy all those files over to the next two, and now you'd have to create a way to sync all the uploaded files among all your VMs.
I don't actually think you'd run into this problem with Azure Web Apps, but it i a problem that can arise if you're handling the VMs yourself or through an auto-scaling policy. You'll definitely run into the issue if you decide to spin up new web apps in different regions (Say the Ireland region to your EU customers get better performance - you'd now have two different locations where files could be uploaded to and you'd need to sync them as opposed to uploading them to Azure storage all along and keepin them centrally located)
I'm thinking about setting up 2 web VMs with a load balancer and availability set, and another VM for SQL server (not sure if I can set an availability set for a SQL Server as well - SQL Server Express / Standard?)
My main problem is how to keep both web servers in sync (prefer not to use the DFS) or having the files in more than one location...
Another issue - is user uploaded content that I want to be available in both web servers (I wonder if I can also direct cache objects to be saved on a specific storage disk)
So, I was thinking to setup a storage account and attach it to both web VMs for user uploaded content and images while each server still serve it's own separate web application with same shared access to content files...
Is that a good idea? I understand that Azure storage is a virtual disk that is supposed to be highly available and fast - is it true??
Do I get a major performance hit if using the same storage disk from 3 different VMs (is that even possible?)
UPDATE:
I found out that because I'm using the BizSpark program I can't really connect more than one server - and share resources between them (unless I pay extra for it). so this became irrelevant for now
Also, I'm talking about ASP.NET but this shouldn't matter
Azure Files enables you to run multiple IIS instances against a single file share and thus not have to worry about replicating files across the multiple shares - so this is definitely an option. See Getting Started with File Storage for more information.
I present my problem if anyone can help will be appreciated:
I have a web site (implemented in WebForms in Azure) where the user can make configurations that will result into a XML with all the data of the configuration.
In the other hand, I have a Worker Role which is working and need this XML configuration.
The question is: how can I send this XML file generated by the user to the Worker Role?
I have looked for something similar to a REST API as interol communication seems not to be the correct path to follow.
Save the XML to the blob storage
Use Azure Queue - web app will add the "job" message pointing to XML (XML url + maybe some other data) and worker roles periodically check for "jobs" in this queue and processing them
It is recommended practice in Azure, and it works very well (I am using it on some projects with very high load - hundreds of thousands "jobs" per day. Azure queues are very reliable and fast, those performance issues you have read about were certainly in some "user code".
I´m planning a Azure Web Api and one of the requests is to let users upload Images from their mobile devices to our Blob Storage. The Clients are Windows Phones, iPhones and Android Phones.
Trying to plan the solution, I found a lot of tutorials doing this with Azure Mobile Services. But with scaling requirements, this solution should not be used. Instead we want the Clients to upload Images directly into our Blob Storage, secured by SAS (generated by our Api).
Does anyone knows how to handle the upload of all kinds of actual mobile OS without Mobile Services? Does anyone have some links to tutorials, how-to´s, etc.?
Edit:
I´m not shure, how all different devices handle their photo Uploads?
Posting form data? Mulitpart?
Uploading via REST?
Uploading via FTP?
Because of this question I search for some hints to plan the project forward to accept the simplest solution for all mobile OS.
Any application capable of sending web requests could do the trick, you'll have to be more specific about your needs.
Using the blob REST API ("put" method) as described here is doable from any application. It's only a matter of sending a Http request containing the right headers and the blob as request body. You'll probably have to come up with a way for distributing the shared acces keys if the containers aren't public.
I've been using the Azure Mobile Service for the last 4 months and I'm very happy with the results.I can create and maintain a backend service with no issues.
The only thing I did not find how to implement is Image Manipulation and Hosting.
My iOS app needs to upload files to the server and the server will resize/crop the images to generate some thumbnails.
Right now Im hosting the images in Amazon s3 and the resizing is done in the client. The problem with is that I need to upload 2 images instead of just one.
What are the options to solve my situation using only the Azure Mobile Services?
For storing images, you could possibly use Azure Blob Storage instead of Amazon S3. One benefit you would get out of it is lower latency between your mobile service and storage.
Coming to your 2nd question, you would need some kind of server side processing to work with the images once they're received at the mobile service end. I've not worked extensively with node.js, but a quick search led me to this thread on SO: Which library should I use for server-side image manipulation on Node.JS?. You could possibly use one of the libraries mentioned there to process the image in mobile service itself and then push both images into storage.