I want to build a image sharing platform for customers to use. This platform will take an image provided by a user, create copies of it at multiple resolutions, and store them ready to be shared or downloaded. How to achieve this using azure in a cost effective ways
Im thinking to use Azure functions(for the api calls) storage blobs , event grid and cosmos db for the same.
To keep the costs low keep it simple:
Store data in Blob storage. Price varies based on redundancy, speed of access and location
Azure functions for processing images, consumption plan gives 1M free requests per month
Azure app service to host web site for uploading images, there is a free tier
Related
SUMMARY: Can I use an Azure Storage Plan as additional storage that is available to an Azure Web App?
DETAILS: We have a very simple Web App using a low amount of CPU and large amount of storage (all it does is list and allow you to download files). I need at least 50GB of storage for video and audio files for this app and to get that amount of storage on a plan pushes me up to unreasonable CPU and other resources that I don't need with the consequences of a much higher price.
My plan (hope) is that I could create a Web App and remap the root directory of the Web App to a Storage Plan that is 50GB. Two challenges that I have spent the last day researching but at this point, I have not been able to find an answer.
I have created the Web App. I see that /wwwroot is on the D: drive of the Web App. I also created a storage plan and have created a sharable area of 50GB on the storage plan.
So... Can anyone give me some insight into the following:
How do I tell the Web App to use the 50GB of storage which is now available to it?
How can I map a drive letter to the area in the storage plan?
How do I tell the Web App that it should use the 50GB area on the storage plan as the root drive instead of using D:/Webroot
Many Thanks!
Refer this feedback link on a similar request: https://feedback.azure.com/forums/169385-web-apps/suggestions/13536996-the-ability-to-store-iis-logs-in-azure-file-storag - it has been denied.
By default, on Azure WebApps, all files are stored in the file system with the application, including the media files. You may wish to know about the main types of files (https://github.com/projectkudu/kudu/wiki/Azure-runtime-environment) that are dealt on Azure WebApp (Persisted files, Temporary files and Machine level read-only files)
Refer to the article (https://github.com/projectkudu/kudu/wiki/File-structure-on-azure) File structure on azure to know the sets of files & dirs on Azure WebApp, and check the directories which include the possible increasing files, such as LogFiles, site/repository, site/deployments(for deployment slots) and your directory for uploading files.
To verify you can go to your scm site's debug console
(https://{sitename}.scm.azurewebsites.net/DebugConsole) and query for the free space on d:\local. The “Disk Space” depends on the App Service plan you're using. It's 1 GB for Free, 10 GB for Basic, 50 GB for Standard and 250 GB for Premium, refer the document for more details on these limits: https://azure.microsoft.com/en-us/pricing/details/app-service/.
If your requirement fits you may use ASE - Azure App Service Environment is an Azure App Service feature that provides a fully isolated and dedicated environment for securely running App Service apps at high scale.
https://learn.microsoft.com/en-us/azure/app-service/environment/app-service-web-how-to-create-a-web-app-in-an-ase
According to Microsoft Azure Support:
"... since the Product Group confirmed that it is not possible to mount
additional storage to the web app, you can integrate Azure storage
with the Azure SDK or rest API. But you can't mount the drive and use
it as storage.
Another option that you have would be to replicate the scenario on a
Virtual Machine where you can choose its capabilities (Number of
cores, RAM, and Storage Memory)."
So there you have it. It appears that WebApps are pretty fixed configurations which means that when you scale up a Web App, you get more CPU resources AND more Disk storage. It's a packaged deal most likely designed for ease of deployment. Nothing, it appears, you can do about that.
The best alternative, it seems, is to spin up a VM with your chosen OS and then add additional disk storage as needed. It's a "do-it-yourself" approach but the best solution that seems to be available.
We are developing a "multi tenant application" (MTA) on AZURE. In addition we develop "single tenant applications" (STA) for customers that utilise MTA data via a REST API end point i.e so the STA can be hosted anywhere.
A specific STA uploads and stores video files. Security for these video files is important and 1xVideo 1xConcurrentUser most likely consumption use case. It not clear at this stage the user will consume the content by streaming or download.
QUESTIONS
Using Azure MEDIA SERVICES account/keys its easy to upload , store and download media content. What are the benefits of using MEDIA SERVICES over a standard Azure STORAGE ACCOUNT ? ? I understand MEDIA SERVICES use a STORAGE ACCOUNT.
Does isolating a STA into a new Azure subscription makes sense to isolate video related costs categorically ? the itemised bill contains 6000+ rows. Difficult to extract the relevant data for an STA each month. In theory a STA customer could in future take control of this account management and costs.
Is there a max number of CONTAINERS that can be added to a STORAGE ACCOUNT ?
Should the CONTAINER be of type PRIVATE to secure the content but still allow access for the STA?
Thank you
Scott,
Media Services is good if you're looking to accept incoming video and process it to serve in other formats or to leverage streaming media playback. Serving video directly out of an Azure Blob Storage Account is possible but it will not provide smooth streaming or transcoding (no streaming playback may mean stop / start of video for users with high latency connections).
I would advise against putting each STA into their own subscription. While it will give you a degree of control over the management of charging back usage to the STA user it will be a big overhead to manage. Your best bet would be to use an appropriate storage account / container setup to allow you track calls some other way and provide estimated costs. Don't forget that Azure is always changing and it may be that future features give you the ability to tag and track costs inside a subscription more effectively.
There is no limit on number of containers in a storage account. The limits are 50 storage accounts per subscription and a maximum of 500TB of storage per account. Storage and Subscription Limits are documented here: http://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits/#storagelimits
You can use Shared Access Signatures to control access to Blobs in Azure Blob Storage. See here for how to create and use them: http://msdn.microsoft.com/en-us/library/azure/jj721951.aspx and here for guidance on setting permissions on Blob Storage containers: http://msdn.microsoft.com/en-us/library/azure/ee393343.aspx
HTH
Simon.
I will try to answer the first question:
Using Azure MEDIA SERVICES account/keys its easy to upload , store and download media content. What are the benefits of using MEDIA SERVICES over a standard Azure STORAGE ACCOUNT ? ? I understand MEDIA SERVICES use a STORAGE ACCOUNT.
Answer: Azure Media Services origin server is the IIS media service in the cloud. All video contents are stored in Azure Blob storage and there is a mapping between the media service and storage. There are many advantages of using media server rather than directly downloading from storage: (1) Media server has the intelligent to forward the right data fragment(right bitrate, time stamp) to your client efficiently. (2) our origin server dynamically package multiple bitrate MP4 from storage account into multiple streaming format (HLS, Smooth streaming and MPEGDASH), which get to played on various devices and platform. Hence, you save on the cost for encoding your video into multiple formats. (3) Our origin server supports live streaming.
I think this question goes into why we invent media server. I have a blog explains how video streaming works for your reference: http://mingfeiy.com/adaptive-streaming-video-streaming.
I've recently begun using the Web Deployment Accelerator for my Windows Azure account. It is providing an immediate return in time saved and is an excellent offering.
However since "everything" is now stored to Azure Storage rather to the regular E:Drive I am immediately seeing a cost consequence for using the tool.
In one day I have racked up a mighty 4 cent NZD charge. In order to do that I had to burn through about 80,000 storage transactions and frankly i cant figure where they all went.
I uploaded 6 sites that are very small wouldn't have more than 300 files each. So I'm wondering:
a. is there is a profiling tool for the Web Deployment Accelerator that will allow me to see where and how 80,000 storage transactions were used for such a small offering. Is it storage transaction intensive tool? Has any cost analysis been carried out in terms of how this tool operates? Has it been optimised with cost in mind?
b. If I'm using this tool do i pay for 2 storage transactions per http request to a site? As since the tool now writes the web server logs to table storage, that would be one storage request to pull the http request resource (img, script, etc) and a storage request to write the log entry as well would it not?
I'm not concerned about current charges I 'm concerned about the future if i start rolling all my hosted business into the cloud. I mean Im now being charged even just to "look" at my data right? If i list the contents of a storage folder using a tool like Azure Storage Explorer that's x number of storage transactions where x = number of files in the folder?
Not sure of a 3rd-party profiler tool, but Windows Azure Storage logging and metrics will give you very detailed info regarding both individual accesses and hourly rollups. It's pretty straightforward to enable, and the November 2011 SDK includes support for the API calls required for enabling. See here for an overview of what's offered for metrics and logging.
My team worked with Fullscale180 to build a storage library, Azure Store XRay, to demonstrate how to enable and query storage metrics and logging. Note: This was published before the SDK had logging and metrics support, so it uses the REST API calls instead. But that won't impact you if you try to use the library.
You can also look at another code demo, Cloud Ninja, which calls the XRay library for its metrics display (see here for running demo).
Regarding querying storage for objects in blob containers: that's not a 1:1 transaction:file scenario. You can specify the maximum number of blobs to return when listing items in a container. It's possible that all blobs are returned in one transaction. Of course, if you then grab each blob, each of these will be at least one transaction (depending on blob size). See here for details about listing blobs.
I am quite sure I know the answer, just want to make sure I got this right.
From Azure In Action :
If I use the CloudBlobClient from a WCF service that sits in my WebRole, to access blobs (read/write/update) , so :
1) Does read/write/update charge as transaction or are they free ?
2) Does the speed of accessing those blobs is fast as mentioned in the note ?
If I use the CloudBlobClient from a WCF service that sits in my
WebRole, to access blobs (read/write/update) , so :
1) Does read/write/update charge as transaction or are they free ?
Transaction metering is independent of where the requests are made from. Storage read/write/update is done via REST API calls (or through an SDK call that wraps the REST API calls). Each successful REST API call will effectively count as a transaction. Specific details of what constitutes a transaction (as well as what's NOT counted as a transaction) may be found here.
By accessing blob storage from your Worker / Web role, you'll avoid Internet-based speed issues, and you won't pay for any data egress. (Note: Data ingress to the data center is free).
2) Does the speed of accessing those blobs is fast as mentioned in the note ?
Speed between your role instance and storage is governed by two things:
Network bandwidth. The DS and GS series have documented network bandwidth. The other sizes only advertise IOPS rates for attached disks.
Transaction rate. On a given storage account, there are very specific documented performance targets. This article breaks down the numbers in detail for a storage account itself, as well as targets for blobs, tables and queues.
I want to pull large amount of data, frequently from different third party API web services and store it in a staging area (this is what I want to decide right now) from where it will be then moved one by one as required into my application's database.
I wanted to know that can I use Azure platform to achieve the above? How good is it to use Azure platform for this task?
What if the data to be pulled is of large amount and the frequency of the pull is high i.e. may be half-hourly or hourly for 2,000 different users?
I assume that if at all this is possible, then the bandwidth, data storage and server capability etc. will not be a thing to worry for me but for ©Microsoft. And obviously, I should be able to access the data back whenever I need it.
If I would have to implement it on Windows Servers, then I know that I would use a windows service to do this. But I don't know how it can be done for Windows Azure Platform if at all it is possible?
As Rinat stated, you can use Lokad's solution. If you choose to do it yourself, you can run a timed task in your worker role - maybe spawn a thread that sleeps, waking every 30 minutes to perform its task. It can then reach out to the Web Services in question (or maybe one thread per Web Service?) and fetch data. You can store it temporarily in Azure Table Storage, which is a fraction of the cost of SQL Azure (0.15 per GB), and then easily read it out of Table Storage on-demand and transfer to SQL Azure.
Assuming you host your services, storage and SQL Azure are in the same data center (by setting the affinity appropriately), you'd only pay for bandwidth when pulling data from the web service. There'd be no bandwidth charges to retrieve from Table Storage or insert into SQL Azure.
In Windows Azure that's usually Worker Role used to host the cloud processing. In order to accomplish your tasks you'll either need to implement this messaging/scheduling infrastructure yourself or use something like Lokad.Cloud or Lokad.CQRS open source projects for Azure.
We use Lokad.Cloud for distributed BI processing of hundreds of thousands of series and Lokad.CQRS allows to reliably retrieve and synchronize millions of products on schedule.
There are samples, docs and community in both projects to get you started.