There is a Javascript library where you can get the file upload progress. This however only shows the progress of the upload to the api - when the progress reaches 100%, the API method is entered. I am using Azure Storage to store the files uploaded, it takes about the same time for the method to upload the stream to the storage container. Is there a good way to relay progress back? I had a look at SignalR and I don't think this is what im looking for, anyone have any ideas of how to relay this information?
I do use background jobs a lot too and was considering turning the stream to a byte array then sending as a job, this worries me for large files tho.
Any pointers would be greatly appreciated
EDIT: Front end is separate to API
Related
I have created a meeting app with angular as front-end and ASP.NET Core as back-end. The application is working fine, but sometimes when I am trying to download the recorded file with servercallid, I am getting the error like Service Unavailable. But most of the time the file is generating and storing to blob storage. But I do not know, why is getting error sometimes. We did not get any proper exception or reason as a response.
Do we have any specific size or timeframe constrain for generating the recorded file?
As per the documentation recordings are only available to download for a limit amount of time:
Azure Communication Services provides short term media storage for recordings. Recordings will be available to download for 48 hours. After 48 hours, recordings will no longer be available.
I am wanting to create a custom Azure Logic App which does some heavy processing. I am reading as much as I can about this. I want to describe what I wish to do, as I understand it currently, then I am hoping someone can point out where I am incorrect in my understanding, or point out a more ideal way to do this.
What I want to do is take an application that runs a heavy computational process on a 3D mesh and turn it into a node to use in Azure Logic App flows.
What I am thinking so far, in a basic form, is this:
HTTP Trigger App: This logic app receives a reference to a 3D mesh to be processed, it then saves this mesh to Azure Store and passes that that reference to the next logic app.
Mesh Computation Process App: This receives the Azure Storage reference to the 3D mesh. It then launches a high performance server with many CPUs and GPU's, the high performance server downloads the mesh, processes the mesh, then uploads the mesh back to Azure Storage. This app then passes the reference to the processed mesh to the next logic app. Finally this shuts down the high performance server so it doesn't consume resource unnecessarily.
Email notification App: This receives the Azure Storage reference to the processed mesh, then fires off an email with the download link to the user.
Is this possible? So far what I've read this appears possible. I am just wanting someone to verify this in case I've severely misunderstood something.
Also I am hoping a to get a little bit of guidance on the mechanism to launch and shut down a high performance server within the 'Mesh Computation Process App'. The only place the Azure documentation mentions asynchronous, long-term, task processing in Logic Apps is on this page:
https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-create-api-app
It talks about it requiring you to launch an API App or a Web App to receive the Azure Logic App request, then ping back status to Azure Logic Apps. I was wondering, is it possible to do this in a serverless manner? So the 'Mesh Computation Process App' would fire off an Azure Function which spins up the higher performance server, then another Azure Function periodically pings that server to report back status until complete, at which point an Azure Function then triggers the higher performance server to shut down, then signals to the 'Mesh Computation Process App' that it is complete and it continues onto the next logic app. Is it possible to do it in that manner?
Any comments or guidance on how to better approach or think about this would be appreciated. This is the first time I've dug into Azure, so I am simultaneously trying to orient myself on proper understanding in Azure and make a system like this.
Should be possible. At the moment I'm not exactly sure if Logic Apps themselves can create all of those things for you, but it definitely can be done with Azure Functions, in a serverless manner.
For your second step, if I understand correctly, you need it to run for long, just so that it can pass something further once the VM is done? You don't really need that. When in serverless, try to not think of long running tasks, and remember that everything is an event.
Putting stuff into Azure Blob storage is an event you can react to, this removes your need for linking.
Your first step, saves stuff to Azure Store, and that's it, doesn't need to do anything else.
Your second app, triggers on inserted stuff to initiate processing.
The VM processes your stuff, and puts it in the store.
The email app triggers on stuff being put into "processed" folder.
Another App triggers on the same file to shut down the VM.
This way you remove the long running state management and chaining the apps directly, and instead each of them does only what it needs to do, and then apps can trigger automatically to the results of the previous flows.
If you do need some kind of state management/orchestration in all of your steps, and you want to still be in serverless, look into durable azure functions. They are serverless, but the actions they take and results they get are stored in table storage, so it can be recreated and restored to a state it was in before. Of course everything is done for you automatically, it just changes a bit on what exactly you can do inside of it to still be durable.
The actual state management you might want to do is maybe something to keep track of all the VM's and try to reuse stuff, instead of spending time spinning them up and killing them. But don't complicate it too much for now.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview
Of course you still need to think about error handling, like what happens if your VM just dies without uploading anything, you don't want to just miss on stuff. So you can trigger special flows to handle repeats/errors, maybe send different emails, etc.
I've been looking for a decent example to answer my question but, not sure if its possible at this point.
I'm curious if its possible to upload a image or any file and stream it so separate server? In my case I would like to stream it to imgur.
I ask this because I don't want to reduce the bandwidth hit for all the files to come to the actual nodejs server and upload it from there. Again, I'm not sure if this is possible or if I'm reaching but, some insight or an example would help a lot.
Took a look at Binary.js which may do what I'm looking for but, its IE10+ so no dice with that...
EDIT: based on comment on being vague
When the file is uploaded to the node server, it takes a bandwidth hit. When node takes the file and upload it to the remote server, it takes another hit. (if I'm not mistaken) I want to know it its possible to pipe it to the remote service(imgur in this case) and just used the node server as a liaison. Again, I'm not sure if this is possible which is why I'm attempting to articulate the question. I'm attempting to reduce the amount of bandwidth and storage space used.
I'm trying to create an app which uses Windows Media Services REST API to upload video files and transcode them. Everything works great but i have encountered a situation, in which i'm not able to upload a 160 MB video file without knowing why. It's extremely painfull to debug the upload process in a regular way cause uploading 160 MB file takes ages so i decided to ask my question here:
I know about Azure Storage limitations of single file size (which is up to 64 MB). Is there such limitation for uploading files to Windows Media Services as well? Do i need to send that file in 4 MB chunks?
If so, how can actually do that using REST API ? I can send chunked file to a regular storage account but, when it comes to WMS, things are a bit different. Basically, when dealing with WMS, i need to upload my file (or file blocks) to a specific temporary url and i'm not sure how to combine that with chunks and setting a block id etc. I also can't find any info about that on the internet.
Thanks in advance for any advice!
You didn't say which platform are you using to build your application (I'm guessing it's .net?)
According to MSDN, single file (blob) is not limited to 64 MB:
Each block can be a different size, up to a maximum of 4 MB. The maximum size for a block blob is 200 GB, and a block blob can include no more than 50,000 blocks. If you are writing a block blob that is no more than 64 MB in size, you can upload it in its entirety with a single write operation.
This means you can upload files (blobs) which are up to 200 GB. If file is smaller than 64 MB, you can upload it as one big chunk (block). If it's bigger than 64 MB, you will have to split it into smaller blocks (up to 4 MB each) and upload it that way.
Full disclosure: I wrote this blog post which explains how to build an async upload page which can upload files in chunks. It uses the Azure REST API to upload all the file blocks and the Windows Azure Media Services SDK to communicate with the Media Service and create locators (temporary URL's you mentioned) which are used to upload files.
There is quite a bit of code involved in making this work so I created a simple demo app (written in JS and .net) to go together with the post. If you're not doing this in .net, you will still be able to use the JS portion of the code, it's just that you'll need to obtain the upload locators through Azure REST API as well.
Very simplified, the upload/transcoding workflow goes something like this:
Obtain a temporary upload URL (locator) through the back-end of your application (to secure your Azure credentials)
Split a file into smaller chunks, upload them all to Azure storage on behalf of WAMS (Windows Azure Media Services) through the REST API while keeping track of all block ID's
Submit an XML containing all block ID's to the REST API
If needed, transcode videos (I used the WAMS SDK in the back-end of the app to create the encoding jobs) - each video will be a new separate Asset
Publish assets - get locators (URL's) for accessing the original and/or transcoded videos
I also recommend you read Gaurav Mantri's post about uploading large files to Azure storage. It explains a lot and is a very good read about this topic.
I know there are two methods available to upload files in AWS S3 (i.e. PutObject and TransferUtility.Upload). Can someone please explain which one to use?
FYI, I have files ranging from 1kb to 250MB.
Thanks in advance.
Amazon deprecated the S3 Transfer Manager and migrated to the new
Transfer Utility.The Transfer Utility is a simple interface for handling the most common uses of S3.It has a single constructor, which
requires an instance of AmazonS3Client. Working with it is so easy
and let the develpers perform all operations with less code.
Following are key features of using Transfer Utility over Transfer Manager
When uploading large files, TransferUtility uses multiple threads to
upload multiple parts of a single upload at once.When dealing with
large content sizes and high bandwidth, this can increase throughput
significantly.TransferUtility detects if a file is large and switches into
multipart upload mode. The multipart upload gives the benefit of
better performance as the parts can be uploaded simultaneously as
well, and if there is an error, only the individual part has to be
retried.
Mostly we people try to upload large files on S3 that take too much
time to upload,at those situations we required progress information
such as the total number of bytes transferred and remaining amount of
data to transfer.To track current progress of transfer with the
Transfer Manager, developers pass an S3ProgressListener callback to
upload or download, which periodically fires the method below.
Pausing transfers using the Transfer Manager is not possible with
stream based uploads or downloads.But Transfer Utility provide us
pause and resume option, it also has one single-file-based method for
uploads, and downloads.
transferUtility.upload(MY_BUCKET,OBJECT_KEY,FILE_TO_UPLOAD)
transferUtility.download(MY_BUCKET,OBJECT_KEY,FILE_TO_UPLOAD)
The Transfer Manager only requires the INTERNET permission. However,
since the Transfer Utility automatically detects network state and
pauses/resumes transfers based on the network state
pause functionality to the Transfer Utility is easy, since all transfers can be paused and resumed.A transfer is paused because of a loss of network connectivity, it will automatically be resumed and there is no action you need to take.Transfers that are automatically paused and waiting for network connectivity will have the state.Additionally, the Transfer Utility stores all of the metadata about transfers to the local SQLite database, so developers do not need to persist anything.
Note :
Every thing else is good.But Transfer Utility does not support a copy() API.To accomplish it use AmazonS3Client class copyObject() method.
Based in Amazon docs, I would stick with TransferUtility.Upload:
Provides a high level utility for managing transfers to and from Amazon S3.
TransferUtility provides a simple API for uploading content to and downloading content from Amazon S3. It makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance, and reliability.
When uploading large files by specifying file paths instead of a stream, TransferUtility uses multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can increase throughput significantly.
But please be aware of possible concurrency issues and the recommendation about using BeginUpload (the asynchronous version), like in this related post