I am trying to upload mlflow artifacts to an azure blob storage instance.
The backend server is able to connect to the blob storage and is working fine.
The client is also working fine.
The only issue on the client-side is a time-out while uploading artifacts to the blob storage.
The following exception is thrown:
/venv/lib/python3.9/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 361, in send
raise error
azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))
Usually, I would just increase the timeout, but I don't know how to do that for mlflow. I searched for a possible solution here as well as checking their GitHub for possible open and/or closed issues regarding the same problem, but I have yet to find a solution. Is it possible to adjust the timeout for the artifact logging in mlflow?
Related
I have a file service (WCF) writing file info to an on-prem DB and saving the file to the Azure blob storage. We are thinking of updating this process to publish an event to Azure EventHubs. Some team members are saying that everything should be reversed if the event can't be posted but I think we should just re-try in case something happens while publishing the event, so the user doesn't have to re upload the file.
What would you do?
I want to perform retries in case an upload fails to azure blob storage. Does the sdk offer such a mechanism? I was able to find this stuct azblob.RetryReaderOptions but as I understand it can only be used while downloading files.
Do I just have to have to just run the upload through a loop and breakaway in case of success?
This is the code I have right now.
azup.uploadResult, azup.err = azblob.UploadFileToBlockBlob(ctx, f, *azup.blockBlobURL, azblob.UploadToBlockBlobOptions{})
I want to perform retries in case an upload fails to azure blob
storage. Does the sdk offer such a mechanism?
The blob storage API doc about go is here:
https://godoc.org/github.com/Azure/azure-storage-blob-go/azblob
You can find sdk didn't offer such a mechanism.
Do I just have to run the upload through a loop and
breakaway in case of success?
I think this idea is fine. SDK does not provide an upload retry mechanism, so you have to implement it through code.
I have a requirement that should allow user to allow download file through Azure Blob Storage. I am not supposed to expose the blob storage or generate SAS for a file and expose it to end user. So for this purpose i have used API Management and in the inbound policy i am generating SAS and forming the complete URL for blob download and setting it as Backend service.
Eg: After the backend service is formed it will look like this
https://myblobstorage.blob.core.windows.net/container/file.zip?sv=2018-03-28&sr=b&sig=fceSGjsjsuswsZk1yv0Db7EYo%3D&st=2020-02-14T12%3A36%3A13Z&se=2020-03-15T12%3A41%3A13Z&sp=r
I am able to download files with size of 14 GB through API Management with a through put of 10MBPS. But I also want to download a file that is of size 200 GB. When i try to download this file, the download is initiated and i am able to download some content but after a while it fails with below error. And during the download the max throughput achieved is 10 MBPS.
After I check App Insight log for this failure, i see following error - BackendConnectionFailure: at transfer-response, Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. It seems that this error means there was a problem at blob storage but does not exactly state what it could be.
If i use the actual SAS url that is generated out of API Management and download file. The downloads completes with a much higher throughput of 90 MBPS.
I have not set any bandwidth limit or download limit using policy in APIM.
I am trying to check if there is any default setting that is preventing this file to be downloaded either on Blob or on APIM? And also trying to figure out why the throughput is so low when i download the file using APIM.
Note: I am using an Azure VM with good configuration and using curl to test my API.
I am trying to upload an application to newly created azure batch account from the portal I followed below steps-
1) Created a azure batch account.
2) Create .zip of the application exe in local desktop
3) Go to the application options
4) Click on add applications
5) Give applicationid, version and application package path by selecting the .zip from local machine.
6) Click on submit
Got different Errors-
ajaxExtended call failed
Upload Error for ffmpeg.zip
Upload block blob to blob store failed. Details: StatusCode = 201, StatusText = Created.
This happened to me as well, and like Phil G, I noticed that a message was showing up in F12 developer tools that 'the auto storage account keys are invalid.'. However, they were valid.
The problem was that I had turned off 'allow access from all networks' under firewalls and network configuration. Changing this back to 'allow access from all networks' worked, at the tradeoff of security.
If relevant, I'm using a cluster with public access disabled, and user subscription pool allocation mode.
Actually when we upload a .zip file in this case it's failing, its better to use Azure Batch Explorer, its a desktop application.
https://azure.github.io/BatchExplorer/
Than you can easily add a package/ application to your batch account.
Also I was getting the same error when uploading a file to blob container from azure portal, so I used Microsoft Azure Explorer to upload and download the files.
I had a slightly different error and the message was very vague:
Upload Error for ffmpeg-3.4-win64-static.zip
File Upload encountered an unexpected error during upload.
Batch Explorer also failed to upload the file.
By looking at the network traffic in my browser I saw the POST request recieved a success 200 code, but looking inside the response JSON I saw the detailed error:
HHTP Status 409 - The auto storage account keys are invalid, please
sync auto storage keys.
I'd changed them a day ago, and had successfully used the new ones in a batch app, but in order for the batch account to automatically upload the application to the storage account they keys needed to be synchronized.
Quick fix was to sync the keys and all was good.
I'm trying to upload a file using webapi hosted as an azure website. I'm getting a 400 bad request error.
Failed request tracing tells me that the module ManagedPipelineHandler is giving the 400 status with a notification of 128.
Googling suggests this is down to file size limits.
The MultipartFormDataStreamProvider is successfully saving the file into a temp folder on azure, and I know the code "works on my machine" so I suspect it's a config issue (the files are under a meg at the moment)
I've tried changing the maxRequestLength to something quite high in the config but that hasn't resolved the issue, and I can't really see anything to change for webapi itself.
Any advice would be great!
Ta
Ross
Avoid uploading files to local storage of the Azure Website. Instead, upload the file to centralized Azure blob storage.