I've just discovered that I have 100's of GB of log files/failed request logs on Azure Blob storage that have been accumulating over the years. Is there a tool or technique for managing them - the directory structure is convoluted so its not as easy as just sorting by date (I use Cloud Storage Studio as an Azure management tool)
[With apologies in advance if it feels like product plug] You could possibly look into Azure Diagnostics Manager (http://www.cerebrata.com/Products/AzureDiagnosticsManager). This tool is built specifically for viewing/managing Windows Azure Diagnostics. You could also look into Azure Management Studio (http://www.cerebrata.com) which combines Cloud Storage Studio and Azure Diagnostics Manager into one product and is currently in public beta.
Both tools allow you to purge old data, search for logs data based on date ranges.
(Disclosure: I'm part of Cerebrata team)
Related
I am looking to do the following in Azure however I should point out that on my local machine I have no visual studio, no admin rights, no IT support and no tools (except SSMS) but I have a VERY strong drive to complete this work if its possible.
I have created an Azure blob which receives a file each day (zipped) from a 3rd party. I am looking to do the following:
1)Unzip the data in an automated fashion
2)Get the data into an Azure SQL database (already created) in an automated fashion
What I want to know is if this is possible to do using Azure alone or am I going to need admin rights / Visual Studio? If it is possible any directions that you could point me in would be greatly received!
Thanks
Dave
Based on your description, one approach would be to create a Blob Triggered Azure Function through the Azure Portal (Visual Studio is not required), unzip/process the file and save desired data into Azure SQL. Moreover, considering the there is only one new file per day, prefer the Consumption Plan to optimize cost.
Find more details about Azure Function Blob Binding at https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob.
Based on your description, one approach would be to create a Blob Triggered Azure Function through the Azure Portal (Visual Studio is not required), unzip/process the file and save desired data into Azure SQL. Moreover, considering the there is only one new file per day, prefer the Consumption Plan to optimize cost.
Find more details about Azure Function Blob Binding at https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob.
Spin up your data factory on Azure, unzip function is available on adf
Is there any API or powershell cmdlet available for configuring the Azure stroage account for Monitoring?I mean monitoring blobs,queues and tables.
There are many ways by which you can configure storage analytics on a storage account:
Using Storage REST API.
Using Storage Client Library. Look for GetServiceProperties and SetServiceProperties.
If you're looking for a 3rd party tool, there's a free tool available from Cerebrata - http://blog.cerebrata.com/articles/2011/08/11/cerebrata-windows-azure-storage-analytics-configuration-utility-a-free-utility-to-configure-windows-azure-storage-analytics/. Using this tool, you can configure storage analytics.
If you're looking for a tool to view this data, try this tool: https://github.com/mogliang/Azure-Storage-Analytics-Viewer.
Can I check if my understanding is correct here.
To create an Azure Storage table I have to C# or Javacript, PHP etc.
There is no GUI for simply creating a table? And if there is a GUI, is it popular/recommended approach or a niche thing?
What you're looking for is a Storage Explorer. There are many storage explorers available in the market today - There are both open source and commercial (both paid and free) storage explorers available. Please see this blog post from Windows Azure Storage Team about the list of storage explorers: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/17/windows-azure-storage-explorers.aspx.
Apart from these, Visual Studio also has a storage explorer built into it. You can find that in the Server Explorer. I haven't used Eclipse but I have heard that there's a storage explorer there as well.
I am trying to implement azure storage metrics code in my role but i am checking if there is easy way to get azure storage metric data about my files usage. my code is stable and i do not want to change code again.
Actually if you already have Windows Azure role running, then you don't need to make any changes to your code and you still can get Windows Azure Blob storage Metrics data.
I have written a blog about it last time as Collecting Windows Azure Storage REST API level metrics data without a single line of programming, just by using tools
Please try above and see if this works for you.
Storage analytics is disabled by default, so any operations against your storage up til now has not been logged for analysis.
You may choose to enable analytics at any time, for both logging (detailed access information for every single object) and metrics (hourly rollups). Further, you may choose which specific storage service to track (blobs, tables, queues) and which operations to track (read, write, delete). Once analytics are enabled, you may access the resulting analytics data from any app (as long as you have the storage account name + key).
Persistent Systems just published a blog post on enabling storage analytics for Java apps. The same principles may be applied to a .net app (and the sdk's are very similar).
Additionally, Full Scale 180 published a sample app encapsulating storage analytics (based on REST API, as it was written before SDK v1.6 came out).
Someone explain to me what it is and how the Windows Azure Storage Analytics works and how I can use it in conjunction with storage.
Windows Azure Storage Analytics offers profiler like capabilities for Azure Storage (which includes Tables, Blobs and Queues).
You can find a very good overview of Azure Storage Analytics directly on MSDN. Then, for digging deeper and also getting some code samples, you can go here related to the logging functionality and here for the metrics one.
Check out the new Windows Azure Storage Analytics App for Windows 8 in the app store:
http://apps.microsoft.com/windows/en-US/app/azure-storage-monitor/e9292e05-c469-403d-a787-63645b861593