I have been using Azure Media Services to upload files, encode them to multi-bitrate mp4 and then expose them as smooth streaming units using locators. My problem is that the encoding process is taking a lot of time, say 25-30 minutes for a 30MB file. The mp4 files that I will actually use will be much bigger and I suppose that would take time in hours.
Is there a way to speed up this process, using some other encoders or other means ?
What speed of Encoding units are you using? Are you using just the free shared pool with no reserved units?
Take a look at the different sizes available (S1, S2, S3) here: https://learn.microsoft.com/en-us/azure/media-services/media-services-scale-media-processing-overview#choosing-between-different-reserved-unit-types
S3 size units are the fastest.
Related
Download ~60MB with python api can take more than 10-15 minutes oven more.
Is there a way/example to imporove the performance?
Thanks for your help!
I found that the container was cold blob storage.
Afer upgrade it to the hot container, the performance increase dramatically.
It could be related to a lot of factor such as slow internet speed, different Geo Location.
If client downloads a file through your Python API, you can definitely improve the overall performance by just passing valet token to client.
Then let the client downloads straight from Azure Blob Storage.
According to your description to compute your actual network bandwidth, it seems to be less than 1Mbps, which 544~816Kbps or 68.3~102.4KB/s for costing 10~15 min to download. Based on this bandwidth if it's truth, I think your case is normal.
Per my experience, for downloading a large blob, if the blob includes multi-blocks，a way to improve the download performance is first to Get Block List of the blob, then to download them concurrently via Get Blob with multithread or multiprocess in Python to write a file with file.seek(offset) method which offset parameter is the size value of block ordered by block id. However, your blob size is just ~60MB which less than 64MB, normally it's a single block. So I think the way is not suitable for this case, besides to improve network bandwidth.
I'm trying to create an app which uses Windows Media Services REST API to upload video files and transcode them. Everything works great but i have encountered a situation, in which i'm not able to upload a 160 MB video file without knowing why. It's extremely painfull to debug the upload process in a regular way cause uploading 160 MB file takes ages so i decided to ask my question here:
I know about Azure Storage limitations of single file size (which is up to 64 MB). Is there such limitation for uploading files to Windows Media Services as well? Do i need to send that file in 4 MB chunks?
If so, how can actually do that using REST API ? I can send chunked file to a regular storage account but, when it comes to WMS, things are a bit different. Basically, when dealing with WMS, i need to upload my file (or file blocks) to a specific temporary url and i'm not sure how to combine that with chunks and setting a block id etc. I also can't find any info about that on the internet.
Thanks in advance for any advice!
You didn't say which platform are you using to build your application (I'm guessing it's .net?)
According to MSDN, single file (blob) is not limited to 64 MB:
Each block can be a different size, up to a maximum of 4 MB. The maximum size for a block blob is 200 GB, and a block blob can include no more than 50,000 blocks. If you are writing a block blob that is no more than 64 MB in size, you can upload it in its entirety with a single write operation.
This means you can upload files (blobs) which are up to 200 GB. If file is smaller than 64 MB, you can upload it as one big chunk (block). If it's bigger than 64 MB, you will have to split it into smaller blocks (up to 4 MB each) and upload it that way.
Full disclosure: I wrote this blog post which explains how to build an async upload page which can upload files in chunks. It uses the Azure REST API to upload all the file blocks and the Windows Azure Media Services SDK to communicate with the Media Service and create locators (temporary URL's you mentioned) which are used to upload files.
There is quite a bit of code involved in making this work so I created a simple demo app (written in JS and .net) to go together with the post. If you're not doing this in .net, you will still be able to use the JS portion of the code, it's just that you'll need to obtain the upload locators through Azure REST API as well.
Very simplified, the upload/transcoding workflow goes something like this:
Obtain a temporary upload URL (locator) through the back-end of your application (to secure your Azure credentials)
Split a file into smaller chunks, upload them all to Azure storage on behalf of WAMS (Windows Azure Media Services) through the REST API while keeping track of all block ID's
Submit an XML containing all block ID's to the REST API
If needed, transcode videos (I used the WAMS SDK in the back-end of the app to create the encoding jobs) - each video will be a new separate Asset
Publish assets - get locators (URL's) for accessing the original and/or transcoded videos
I also recommend you read Gaurav Mantri's post about uploading large files to Azure storage. It explains a lot and is a very good read about this topic.
I am facing a problem of slow execution of exe in Azure platform
Following are the Steps:
Read data from SQL Azure Server& CSV files & display in on HTML5 pages.
Write data on CSV files.
Executing a external Fortron exe, which reads data from csv files generated in step 23.
Fortron exe after calculations write data on .txt file.
Read text file data generated in step 5 & display it on HTML5 pages.
Issue:
In point # 3, when we are invoking fortron exe using process start method, then –
On local machines in usually take 17~18 secs
On cloud server this is taking 34~35 secs.
Rest all other activities are taking same time on local as well as cloud server.
Regarding step 3: What size local machine are you using (e.g. number of cores), since you're running an exe that may be doing some number-crunching. Now compare that the the machine size allocated in Windows Azure? Are you using an Extra Small (shared core) or Small (single core)? Plus what size cpu does your local machine have? If you're not comparing like-kind configurations, you'll certainly have performance differences. Same goes for RAM (an Extra Small offers 768MB, with Small through XL offering 1.75GB per core) and bandwidth (XS has 5Mbps, Small through XL have 100Mbps per core).
The Azure systems have slower IO process than a local server this will be the reason you see the performance impact you are also on a shared system so your IO also may vary depending on your neighbours also and server load. If you are task is IO intensive the best bet is to run a VM and you need to persist the data is to attach multiple disks to the VM and then use stripping on the disk.
http://www.windowsazure.com/en-us/manage/windows/how-to-guides/attach-a-disk/
Striped IO Disks performance stats.
http://blinditandnetworkadmin.blogspot.co.uk/2012/08/vm-io-performance-on-windows-azure.html
You will need to have a warm set of disks to get true performance set.
Also I found the Temp storage on VM Normally the D drive to have very good IO so maybe worth if you are going to use a VM to try there first.
I know there are two methods available to upload files in AWS S3 (i.e. PutObject and TransferUtility.Upload). Can someone please explain which one to use?
FYI, I have files ranging from 1kb to 250MB.
Thanks in advance.
Amazon deprecated the S3 Transfer Manager and migrated to the new
Transfer Utility.The Transfer Utility is a simple interface for handling the most common uses of S3.It has a single constructor, which
requires an instance of AmazonS3Client. Working with it is so easy
and let the develpers perform all operations with less code.
Following are key features of using Transfer Utility over Transfer Manager
When uploading large files, TransferUtility uses multiple threads to
upload multiple parts of a single upload at once.When dealing with
large content sizes and high bandwidth, this can increase throughput
significantly.TransferUtility detects if a file is large and switches into
multipart upload mode. The multipart upload gives the benefit of
better performance as the parts can be uploaded simultaneously as
well, and if there is an error, only the individual part has to be
retried.
Mostly we people try to upload large files on S3 that take too much
time to upload,at those situations we required progress information
such as the total number of bytes transferred and remaining amount of
data to transfer.To track current progress of transfer with the
Transfer Manager, developers pass an S3ProgressListener callback to
upload or download, which periodically fires the method below.
Pausing transfers using the Transfer Manager is not possible with
stream based uploads or downloads.But Transfer Utility provide us
pause and resume option, it also has one single-file-based method for
uploads, and downloads.
transferUtility.upload(MY_BUCKET,OBJECT_KEY,FILE_TO_UPLOAD)
transferUtility.download(MY_BUCKET,OBJECT_KEY,FILE_TO_UPLOAD)
The Transfer Manager only requires the INTERNET permission. However,
since the Transfer Utility automatically detects network state and
pauses/resumes transfers based on the network state
pause functionality to the Transfer Utility is easy, since all transfers can be paused and resumed.A transfer is paused because of a loss of network connectivity, it will automatically be resumed and there is no action you need to take.Transfers that are automatically paused and waiting for network connectivity will have the state.Additionally, the Transfer Utility stores all of the metadata about transfers to the local SQLite database, so developers do not need to persist anything.
Note :
Every thing else is good.But Transfer Utility does not support a copy() API.To accomplish it use AmazonS3Client class copyObject() method.
Based in Amazon docs, I would stick with TransferUtility.Upload:
Provides a high level utility for managing transfers to and from Amazon S3.
TransferUtility provides a simple API for uploading content to and downloading content from Amazon S3. It makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance, and reliability.
When uploading large files by specifying file paths instead of a stream, TransferUtility uses multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can increase throughput significantly.
But please be aware of possible concurrency issues and the recommendation about using BeginUpload (the asynchronous version), like in this related post
Speed and cost in mind.
Say I have a few JS and images files shared for multiple websites. that is not huge images files, this is only few static files like PNG sprites and common JS files.
I'm kind of lost on the choice :
- Should i keep it in my webpackage to release in Azure ?
- Or should i put these in blobs ?
The things I don't know is if i have a lot of hits on the blob solution, it might cost more than the hits on the IIS level of the package ?
Right, wrong ?
Edit : I realize storing JS files on the blob won't deliver it gziped ?
No need for the blobs that I can see. The database round trip isn't adding value. I'd just put the static content on the web server and let it serve it up. Let the web server handle compressing the bytes on the wire for those cases where the client indicates that they can handle GZIP compression.
Will your JS and image files be modified often? If so, putting them into the service package would mean that every time you want to update those files, you will have to recompile the service package and redeploy your instance. If you find yourself needing to update often, this will become cumbersome. From a speed perspective, you're not going to see too much of a difference between service them files up from the blogs or serving them up from the web role (assuming the files are in fact not huge). Last but not least, from a cost perspective, if you look at the cost of blob storage ($0.15 per GB stored per month, $0.01 per 10,000 storage transactions), its really not much. Your site would have to have a lot of traffic for the cost to become significant.