We are planning to host our NServiceBus application in Azure web and worker roles (using Azure Service Bus as transport). Existing documentation is a bit unclear when it comes to monitoring and logging of Azure based endpoints.
What would be recommended practices to set up logging and monitoring of such setup?
Would you recommend creating a custom logging module inheriting from ILog interface?
Does ServiceControl work well for monitoring endpoints in Azure? If yes, where do we deploy it?
EDIT:
Igorek's answer helped me to solve logging issue. Can anyone comment on ServiceControl part of the question?
NServiceBus can hook into and utilize log4net. log4net can send its data to Windows Event Logs thru standard log4net appender. Azure Diagnostics can move the data from Event Logs to WADLogs table for inspection by on-demand tools such as Cerebrata's Diagnostic Manager or 24x7 monitoring services such as AzureWatch. http://www.paraleap.com
Alternatively, log4net can also send the logs directly to a custom Azure Table if you want to build your own monitoring tools.
Also, consider monitoring queue/topic depths of your Service Bus, to make sure that stuff is moving thru correctly. Also, consider possibly scaling by the number of messages in SB queues or topics if your NSB setup supports dynamically adding/removing instances.
Disclaimer: I work for company that makes AzureWatch and we happen to have a number of customers who utilize AzureWatch to monitor their NSB-enabled apps.
ServiceControl can indeed be used to monitor your Azure endpoints, just set up ServiceControl for the transport you use, drop the ServiceControl plugins into your endpoint bin folder, and you're good to go.
ServiceControl is best deployed locally, or on an azure virtual machine.
Related
We are hosting Confluent Platform services(broker, zookeeper, schema-registry etc.) in Azure VMs. What is the best option to scrape metrics from these and visualize in Azure monitor or application insights?
I am thinking to scrape JMX metrics and visualize in azure monitor/app insights. Monitoring will include things like health status, message flow in topics, produce/consume rate, VM health etc.
What are some good options to implement this? Any better ideas?
PS- I only want to use Azure native services for monitoring/dashboards.
You can add java agents by exporting the environment variable KAFKA_OPTS. Through that, you can add any additional JVM monitoring
Azure docs - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent
I have a local machine that reads RabbitMQ queue messages.
I wish to move it to cloud. Which Azure service can be used in this case?
I went through event hubs, but I am not sure, if it would read messages from rabbitMQ continuously.
Any suggestions for the service that should be put to use.
You should take a look at Azure Service Bus. It has got FIFO queues as well as publish/subscribe capabilities. However if using Azure managed service is not a strict requirement you can use RabbitMQ on a VM (or a cluster for high availability) as well.
UPDATE: Your response means you want a managed service. There are 2 options - if you want to go with RabbitMQ but do not want to manage the infrastructure you can go for 3rd party service provider like CloudAMQP who will manage it on your behalf. The other option is to go for Cloud native messaging - meaning if you are on Azure you change your messaging service to Azure Service Bus. This would mean changing you code as well.
I want to create logging for API and azure functions.
As thinking to use, "service bus" to create logging.
Logging needs for each request, response and error.
Would it be correct approach to do logging for api management and azure functions through service Bus. and appreciate if any example to create service bus and call from azure or api management to log the requests/responses.
Note: Regarding Application Insights, as found its hit the performance and its more for performance monitoring then logging. https://blogs.msdn.microsoft.com/apimanagement/2018/01/12/application-insights-integration/
I would still use Application Insight for that. You want to track requests and errors which Applicatin Insight will offer out of the box and also provides you a query language to query your logs or to build dashboards. Regarding your performance concern, you should just test the impact on your system - it most likely isn't that relevant.
I have implemented a Proof of Concept in Azure Service fabric that uses Azure Service Bus as a message queue. I'm using nServiceBus within the application to send and respond to messages which makes it very easy to change from one queuing technology to another.
I'd like to know if it's possible to use MSMQ instead of Service Bus in Azure Service Fabric as the nodes that are created are just windows 2016 servers and I'm not sure I need something like Service Bus. It's a question I want to answer with my POC.
Does anyone know whether MSMQ is included in an Azure Service Fabric node or how I could turn it on and if it's a viable solution?
Short answer - MSMQ is not suitable for Azure Service Fabric.
MSMQ is store and forward technology. It's using local file system to persist messages and then forward to another machine. When Service Fabric is going to move service from one node to another, it will not move the file system along. Meaning you'll lose messages.
I would recommend to stay with Azure Service Bus unless there's a good reason you're looking for an alternative.
Is Azure diagnostics only implemented through code? Windows has the Event Viewer where various types of information can be accessed. ASP.Net websites have a Trace.axd file at the root that can viewed for trace information.
I was thinking that something similar might exist in Azure. However, based on the following url, Azure Diagnostics appears to require a custom code implementation:
https://azure.microsoft.com/en-us/documentation/articles/cloud-services-dotnet-diagnostics/#overview
Is there an easier, more built-in way to access Azure diagnostics like I described for other systems above? Or does a custom Worker role need to be created to capture and process this information?
Azure Worker Roles have extensive diagnostics that you can configure up.
You get to them via the Role configuration:
Then, through the various tabs, you can configure up specific types of diagnostics and have them periodically transferred to a Table Storage account for later analysis.
You can also enable a transfer of application specific logs, which is handy and something that I use to avoid having to remote into the service to view logs:
(here, I transfer all files under the AppRoot\logs folder to a blob container named wad-processor-logs, and do so every minute.)
If you go through the tabs, you will find that you have the ability to extensively monitor quite a bit of detail, including custom Performance Counters.
Finally, you can also connect to your cloud service via the Server Explorer, and dig into the same information:
Right-click on the instance, and select View Diagnostics Data.
(a recent deployment, so not much to see)
So, yes, you can get access to Event Logs, IIS Logs and custom application logs without writing custom code. Additionally, you can implement custom code to capture additional Performance Counters and other trace logging if you wish.
"Azure diagnostics" is a bit vague since there are a variety of services in Azure, each with potentially different diagnostic experiences. The article you linked to talks about Cloud Services, but are you restricted to using Cloud Services?
Another popular option is Azure App Service, which allows you many more options for capturing logs, including streaming them, etc. Here is an article which goes into more details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/