Azure websites store IIS logs in blobs. I am writing utilities to get blob data periodically and monitor the server.
Problem here is, how can I optimize my code not to read same data again and again? I don't think I can read a blob store from a certain point. can I?
Azure Blobs support range requests. So if you had read 254 bytes the previous time you could read from 255 to end of blob using the following syntax:
Range: bytes=255-
x-ms-range: bytes=255-
See for details: http://msdn.microsoft.com/library/azure/ee691967.aspx
Related
So I am incrementally loading (chunking) data to an Azure blob, due to low memory in the Azure Function. Everything is going fine, but I want to do gzip on that blob data, once all the data is in place (i.e. stored within the container of the storage account). Is this possible to send an HTTP request or something for Azure to handle that gzip operation INSTEAD of me downloading the full data again to do the gzip operation, as that would entirely defeat the purpose and cause me to run back into the low memory issue.
Thanks
Simple answer, no this is not possible. Azure blob storage does not provide any such compute operations for you. You need to download the files, run compute (such as zipping) on them and re-upload.
I have a requirement that should allow user to allow download file through Azure Blob Storage. I am not supposed to expose the blob storage or generate SAS for a file and expose it to end user. So for this purpose i have used API Management and in the inbound policy i am generating SAS and forming the complete URL for blob download and setting it as Backend service.
Eg: After the backend service is formed it will look like this
https://myblobstorage.blob.core.windows.net/container/file.zip?sv=2018-03-28&sr=b&sig=fceSGjsjsuswsZk1yv0Db7EYo%3D&st=2020-02-14T12%3A36%3A13Z&se=2020-03-15T12%3A41%3A13Z&sp=r
I am able to download files with size of 14 GB through API Management with a through put of 10MBPS. But I also want to download a file that is of size 200 GB. When i try to download this file, the download is initiated and i am able to download some content but after a while it fails with below error. And during the download the max throughput achieved is 10 MBPS.
After I check App Insight log for this failure, i see following error - BackendConnectionFailure: at transfer-response, Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. It seems that this error means there was a problem at blob storage but does not exactly state what it could be.
If i use the actual SAS url that is generated out of API Management and download file. The downloads completes with a much higher throughput of 90 MBPS.
I have not set any bandwidth limit or download limit using policy in APIM.
I am trying to check if there is any default setting that is preventing this file to be downloaded either on Blob or on APIM? And also trying to figure out why the throughput is so low when i download the file using APIM.
Note: I am using an Azure VM with good configuration and using curl to test my API.
We have websites and webjobs hosted in Azure app services that log custom application log data to Azure blob storage (using Monitoring > App service Logs > Application Logging (Blob) option in the app service). We would like to send these log files to Azure Monitor Log analytics workspace as and when they are inserted into the blob storage, so we can aggregate the logs, send alerts etc. Looks like it's easy to send custom log data from a Azure VM to a Log analytics workspace by installing a Microsoft Monitoring agent on the VM but looks like there is no direct support to send the log data from a blob storage. Does anybody have a solution for this?
I've explored using Logic apps for sending data from Blob storage to a Log analytics workspace but didn't have much luck.
AFAIK current best approach to accomplish your requirement is to make use of Azure Log Analytics HTTP Data Collector API which helps to send custom log data to Log Analytics workspace repository. For illustration, you can see sample code as well in the article.
Hope this helps!! Cheers!!
One thing to watch for are the data limits for the Azure Log Analytics HTTP Data Collector API, especially if you're logging potentially large blobs from blob storage.
Quoting from the Data limits section of the Send log data to Azure Monitor by using the HTTP Data Collector API (preview) document:
The data posted to the Azure Monitor Data collection API is subject to
certain constraints:
Maximum of 30 MB per post to Azure Monitor Data Collector API. This is a size limit for a single post. If the data from a single post
exceeds 30 MB, you should split the data into smaller sized chunks and
send them concurrently.
Maximum of 32 KB for field values. If the field value is greater than 32 KB, the data will be truncated.
Recommended maximum of 50 fields for a given type. This is a practical limit from a usability and search experience perspective.
Tables in Log Analytics workspaces support only up to 500 columns (referred to as fields in this article).
Maximum of 45 characters for column names.
I have signed up for Azure Storage the other day. I noticed today when I went into the Azure portal that there are about 500 requests per hour to the table storage. The strange thing is that I'm not using Table Storage and my site isn't live at the moment. So what could possibly be making all these requests? Any ideas?
Azure Storage has this feature called Storage Analytics which performs logging and provides metrics data for a storage account. This data gets stored in the same storage account under special tables (starting with $ e.g. $MetricsCapacityBlob). By default some analytics data is collected and this is why you're seeing these requests.
One way to check the transactions is by exploring contents of $logs blob container. It will tell you in details from where the requests to your storage accounts are being originated.
OK, mystery solved. It turns out it's the actual Azure Portal that is generating the traffic. I originally thought it was the SDK somehow making the calls, but then I had the website turned off, and the portal open, and it continued making requests. Close portal for a while, no requests.
I am in the middle of developing a cloud server and I need to store HDF files ( http://www.hdfgroup.org/HDF5/ ) using blob storage.
Functions related to creating, reading writing and modifying data elements within the file come from HDF APIs.
I need to get the file path to create the file or read or write it.
Can anyone please tell me how to create a custom file on Azure Blob ?
I need to be able to use the API like shown below, but passing the Azure storage path to the file.
http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c
These files i am trying to create can get really huge ~10-20GB, So downloading them locally and modifying them is not an option for me.
Thanks
Shashi
One possible approach, admittedly fraught with challenges, would be to create the file in a temporary location using the code you included, and then use the Azure API to upload the file to Azure as a file input stream. I am in the process of researching how size restrictions are handled in Azure storage, so I can't say whether an entire 10-20GB file could be moved in a single upload operation, but since the Azure API reads from an input stream, you should be able to create a combination of operations that would result in the information you need residing in Azure storage.
Can anyone please tell me how to create a custom file on Azure Blob ?
I need to be able to use the API like shown below, but passing the
Azure storage path to the file.
http://davis.lbl.gov/Manuals/HDF5-1.4.3/Tutor/examples/C/h5_crtfile.c
Windows Azure Blob storage is a service for storing large amounts of unstructured data that can be accessed via HTTP or HTTPS. So from application point of view Azure Blob does not work as regular disk.
Microsoft provides quite good API (c#, Java) to work with the blob storage. They also provide Blob Service REST API to access blobs from any other language (where specific blob storage API is not provided like C++).
A single block blob can be up to 200GB so it should easily store files of ~10-20GB size.
I am afraid that the provided example will not work with Windows Azure Blob. However, I do not know HDF file storage; maybe they provide some Azure Blob storage support.