I have written a Cygwin app that uploads (using the REST API PUT operation) Block Blobs to my Azure storage account, and it works well for different size blobs when using HTTP. However, use of SSL (i.e. PUT using HTTPS) fails for Blobs greater than 5.5MB. Blobs less than 5.5MB upload correctly. Anything greater and I find that the TCP session (as seen by Wireshark) reports a dwindling window size that goes to 0 once the aforementioned number of bytes are transferred. The failure is repeatable and consistent. As a point of reference, PUT operations against my Google/AWS/HP cloud storage accounts work fine when using HTTPS for various object sizes, which suggests the problem is not my client but specific to the HTTPS implementation on the MSAZURE storage servers.
If I upload the 5.5MB blob as two separate uploads of 4MB and 1.5MB followed by a PUT Block List, the operation succeeds as long as the two uploads used separate HTTPS sessions. Notice the emphasis on separate. That same operation fails if I attempt to maintain an HTTPS session across both uploads.
Any ideas on why I might be seeing this odd behavior with MS Azure? Same PUT operation with HTTPS works ok with AWS/Google/HP cloud storage servers.
Thank you for reporting this and we apologize for the inconvenience. We have managed to recreate the issue and have filed a bug. Unfortunately we cannot share a timeline for the fix at this time, but we will respond to this forum when the fix has been deployed. In the meantime, a plausible workaround (and a recommended best practice) is to break large uploads into smaller chunks (using the Put Block and Put Block List APIs), thus enabling the client to parallelize the upload.
This bug has now been fixed and the operation should now complete as expected.
Related
We have an interesting behaviour that we are trying to understand/workaround.
We have an Azure Function running on a consumption host that has nothing but proxies on it.
One of these proxies points to an Azure CDN endpoint, that in turn points to an Azure Storage Account.
From time to time we update the file in the storage account and purge the CDN endpoint.
Requesting the file directly from the CDN returns the (correct) new file data.
However, the Function Proxy url continues to return the (incorrect) old file data.
Browser caches are disabled/cleared and all that normal stuff.
We can see the Last-Modified headers are different, so clearly the proxy is returning the old file.
Further, adding a querystring to the proxy URL - anything you like (we used ?v=1) would return the (correct) new file. Removing the querystring gets us back to the old file again.
Is this behaviour intentional? I have read UserVoice requests where people wanted caching added to Functions and it was explicitly declined due to the number of alternatives available. I see no other explanation for this behaviour, though.
Does anyone know how to disable/defeat proxy response caching?
We are using AzureFunction created with python, in which we are facing an issue "Request body too large." While calling an API with multipart file upload around 200MB.
While we gone through some support link given below, we noticed HTTP request length is limited to 100MB.
We also tried editing the web.config through ssh, but still we are facing the same issue, also after restart functionapp service web.config file gets resetted to 100MB restriction.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook-trigger?tabs=csharp
Kindly provide some workaround to resolve this "Request body too large." issue.
On this ticket that originally discussed increasing size to 100 MB, seems like this work around was suggested for sizes greater than the max:
paulbatum commented on Dec 14, 2018 #two2tee You're not totally off.
In the case of functions you can't configure this (thats what this
issue tracks). Our general recommendation is to switch to a flow where
large files are uploaded to blob storage which are then processed by
your functions.
There is also this open item on allowing content body over 100MB that you might want to follow.
I am using the Logic App to upload 1 Gb file as below-
Trigger - When files are added or modified (properties only)
Action1 - Get file content
Action 2- Create file(Azure fileshare)
Till 35 MB all triggers and actions works fine. After the file uploaded in SFTP crosses 40 MB, SFTP-SSH trigger and action all works fine. But while the workflow moves to the second action - 'Create File': it fails with the below error The specified resource may be in use by an SMB client'. When I see the Azure file share storage account, I see filename.partial.lock getting created. I modified the access policy as well, but the issue persists.
Logic apps are not designed to upload or download the large amount data from source/destination, its a workflow solution which you can design to provide solution your business need, however you can still use chunk upload functionality in logic-app to upload or download large file via logic app.
please refer
https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-handle-large-messages#set-up-chunking
To upload large file, make sure enable the Allow chunking.
From your description, suppose it should be SharingViolation, you could check the error codes here.
And in the official doc, there are two scenario to get the Sharing Violation error:
Sharing Violation Due to File Access
Client A opens the file with FileAccess.Write and FileShare.Read
(denies subsequent Write/Deletewhile open).
Client B then opens the file with FileAccess.Write with
FileShare.Write (denies subsequent Read/Delete while open).
Result: Client B encounters a sharing violation since it specified a
file access that is denied by the share mode specified previously by
Client A.
Sharing Violation Due to Share Mode
Client A opens the file with FileAccess.Write and FileShare.Write
(denies subsequent Read/Delete while open).
Client B then opens the file with FileAccess.Write with
FileShare.Read (denies subsequent Write/Delete while open).
Result: Client B encounters a sharing violation since it specified a
share mode that denies write access to a file that is still open for
write access.
These are scenario you need to consider, another choice you could try to use the REST API to upload the file and in the HTTP action set the Allow chunking.
I think one more way to resolve the issue is redesign which could be more scale-able solution:
Use logic app to get notification if file is added or modified on SFTP/FTP location.
once the file is added read the file path for that file.
Create Service bus message and send the file Path to Service Bus message as message content.
Create Service Bus queue message trigger Azure function which listen to those message (created in step 2)
Azure function will read the Chunk of files from SFTP using the file path.
this way you can read or write more then 30 GB file.
this solution will be more scale able solution as azure function and auto scale on demand.
Thank you all. The reason for the error 'The specified resource may be in use by an SMB client' is due to the mounting of the file share with two Linux Virtual machines. We unmounted the Linux VMs and did fresh single mounting. The error got fixed with that.
MSFT has confirmed in our discussion that "Create File share" has a limitation of 100 or 300 MB. It was just able to work with SFTP as data arrives in chunk. MSFT is working further to give proper error statement, when the file size is beyond 100 or 300 MB. Below is quoted from MSFT email-
"Thanks for the details , actually the Product team confirm to me that your flow is working by luck , it should not work with this sizes an they are working to implement the limits correctly to prevent files larger than maximum size which is maybe 300 or 100 “I am not yet sure ”
And this strange behavior is only happing when we are reading the content from SFTP chucked.
"
I'm trying to log the total size of a request sent out from an ASP.NET Core website hosted in Azure. The goal is to be able to attribute some sense of what the data out cost is for specific functionality within the application. (Specifically we are looking at some functionality that uses Azure Blob Storage and allows downloads of the blobs but I don't think that's relevant to the question.)
I think the solution is some simple middleware that logs out the sizes to a storage mechanism (less concerned with that part) but not sure what to put inside the middleware.
Does anyone know how you would "construct" the total size of the response being sent out of the web app.?
I would presume it will include HttpContext.Response.Body.Length but I'm pretty confident that doesn't include the headers. Also not sure if that's the compressed size of the response body or not.
Thanks
I have a Firebase app with a Cloud Function that generates some thumbnails when an image is uploaded to a particular bucket.
I keep getting these errors, pretty much nonstop:
My question is, and granted I am somewhat new to the Google Cloud Platform, how many times does DNS resolution happen? Does it happen on any upload and downloaded between Firebase and Google Cloud Storage?
All my operations are between Firebase and Google Cloud Storage (i.e. - download from bucket, resize in temp space, and upload back to bucket), and I have a check to make sure not to automatically return if an image begins with 'thumb_' to avoid infinite loop.
That being said, I believe I have this error because initially I accidentally did get myself into an infinite loop and blow out my quota.
Here is some more info about DNS resolutions, I'm not entirely sure how to interpret it but it appears 'DNS resolutions per 100 seconds' is exceeded, but 'DNS resolutions per day is not'
I think the quota limts are 40,000 per 100 seconds Either you are having so many calls, or there is some bug in your code which might be making too many calls unknowingly.