I want to view logs in azure, I mean logs that I have in the console in localhost where I can find them in web site deployed in azure? I am consuming an external API and I want to see what I send and what I received from the prod env
thank you
There are couple of ways to track logs in Azure
Azure API Management
Azure Monitor
Azure API Management helps you track all kinds of requests including
View activity logs
View resource logs
View metrics of your API
Set up an alert rule when your API gets unauthorized calls
Azure Monitor on the other hand helps it possible to programmatically retrieve the available default metric definitions, granularity, and metric values.
The data can be saved in a separate data store such as Azure SQL Database, Azure Cosmos DB, or Azure Data Lake. From there additional analysis can be performed as needed.
Related
Where does data logs of Azure Pipeline v2 gets stored, I would like to retrieve data of failed pipelines for specific date.( Dont want to use azure portal to view these data). Is there any table/view holds such datalogs from database.
To my knowledge, to obtain diagnostic logs you can use Azure Monitor, Operations Management Suite (OMS), or monitor those pipelines visually.
By Azure Pipeline v2, you mean Azure Data Factory v2. Alert and Monitor data factories using Azure Monitor
Diagnostic logs:
Save them to a Storage Account for auditing or manual inspection. You can specify the retention time (in days) using the diagnostic settings.
Stream them to Event Hubs for ingestion by a third-party service or custom analytics solution such as PowerBI.
Analyze them with Log Analytics
The logs are stored on Azure Data Factory web server for 45 days. If you want to get the pipeline run and activity run metadata, you can use Azure Data Factory SDK to extract the information you need and save it somewhere you want.
Recommended approach on this for log term analysis as well as limiting access to a production data factory would be to configure logs to be sent to log analytics. Be sure to enable dedicated logging tables as this will help on the backend in terms of organizing your logs.
From there you can also set up alerts and access groups running off of log analytics queries for better monitoring.
I'd like to use Kibana to create views that display log and metric information output by our Azure Cloud Service Web Roles and Worker Roles. In particular, we'd like to store performance counter information, as described here: https://learn.microsoft.com/en-us/azure/cloud-services/cloud-services-dotnet-diagnostics-performance-counters
Microsoft provides a few tools to view this data if it's stored in Azure diagnostic tables and blobs, but the tools don't have the formatting & visualizing flexibility the Elk stack does. Is anyone aware of how we might be able to get Azure Cloud Service Performance Counter information into Elasticsearch, preferably via logstash?
I've achieved this by enabling Azure diagnostics (writing to Azure table storage) and using ConveyorBelt to send the logs to ElasticSearch.
As the GitHub page states, ConveyorBelt is:
A horizontally scalable headless cluster to shovel Azure diagnostic
data (and other custom data) to ElasticSearch
I have some VMs running on Azure Service. I'd like to redirect logs from them (Windows Event Logs and MS SQL server logs) to a specific log concentrator (like Graylog). For Windows logs, I'm using Nxlog (https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#quickstart_windows). However, for specific (PaaS) applications such as SQL Server (PaaS in general) Nxlog does not apply.
Is there a way to redirect logs (VMs and PaaS) just using Azure (web) tools?
Most services keep their logs in a Storage Account so you can tap into that source and forward logs to your own centralized log database. You generally define the storage account at the place you enable diagnostics for the service.
Don't know what king of logs you are looking for in SQL DB, but for example the audit logs are saved in a storage account.
Azure Operations Management Suite (OMS) can ingest from dozens of services as well as custom logs. As itaysk mentioned, most services in Azure write service related diagnostic information to a storage account. It's really easy to ingest these from within OMS.
https://azure.microsoft.com/en-us/services/log-analytics/
For Azure Web Sites, you can use Application Insights and store custom metrics as well. There's also an option to continuously write these metrics to a storage account.
Here's a similar option for Azure SQL:
https://azure.microsoft.com/en-us/documentation/articles/sql-database-auditing-get-started/
I'm trying to monitor all our resources in a single place and after reading the msdn pages on monitoring web apps and cloud services and azure sql databases i can't seem to understand how to query azure (thorugh the SDK or powershell) to give me the same data i can see in the azure monitoring page.
is there some programattic way to get this data?
I'm afraid there isn't an easy answer to this. There isn't one API where you can fetch all the data: it's mostly service specific.
A couple of pointers:
Metrics for web sites, web/worker roles and VMs can be accessed through the metrics API. See here: https://convective.wordpress.com/2014/06/22/using-azure-monitoring-service-with-azure-virtual-machines/
Metrics for SQL Database are available through the sys.resource_stats and sys.dm_db_resource_stats DMVs.
Windows Azure Diagnostics (for web/worker roles) performance counters can be fetched though the table storage API, in the wadperformancecounterstable table.
There are probably API's for other services as well. I've built a tool myself that fetches data from a couple of these services, and can both plot it and expose it through a unified API, see: https://github.com/WadGraphEs/AzurePlot
Azure keeps a bunch of VM (and cloud service) related logs in WAD* tables. The question is about actions which do not necessarily affect VMs. Say one deleted a Table Storage. Does Azure keep a log record about that? If yes, where? How to fetch them using a program/script?
The Service Management REST API can be used to retrieve the operation logs programmatically.
List Subscription Operations
https://msdn.microsoft.com/library/azure/gg715318.aspx