We have a Azure based system which is growing in complexity, and we need to monitor chains of events and ensure they arrive where we expect them to arrive.
We have a on-prem Java application, which sends events to an IoT Hub. The IoT hub routes to service bus queues. We have functions that update a cosmos database, trigger other functions or route to additional queues. Some functions are also callable through an API Management instance.
Our functions are already connected to Application Insights, and here the Application Insights instance is named the same as the Function App (IIRC this naming was suggested through the form that created the AI resource)
The application map in Application Insights make me lean toward one AI per environment, to have a complete map of the system. Log Analytics also seems logical to use one per environment to be able to potentially correlate data if needed.
What is the correct path for Log Analytics and Application Insights, respectively?
If it is not as clear-cut as stated in my title, what factors do I need to consider when I start to use these services?
The correct number of instances is the one that works best for you, whether that exactly follows recommended practices or not.
The recommendation is to use one workspace per environment and make sure the cloud_RoleName in App Insights to distinguish parts of the system. Log Analytics has similar considerations.
Functions defaults to spinning up an App Insights instance along with the app because if you don't use App Insights you loose most of the logging ability- it's important to connect it to App Insights, but overriding the default behavior and connecting to a centralized workspace is common in larger systems.
There are certainly reasons you might want to split the workspaces, and you can union data across workspaces as needed to pull data together from both Log Analytics and App Insights instances.
Data access control or geographic locations. If you need to keep a portion of the data within certain geographic boundaries or limit access to certain people, then split that portion off.
Similar to the security concern is a billing one. If for whatever reason, billing for different portions of the application needs to be split, then you would also want to split the logging portion.
Different portions of the system rarely interact, or are maintained by different teams, and organizing the data into separate workspaces will provide more benefits over the hassle of cross-
You are going to surpass the limitations on a single resource. Very few applications actually hit these limits, but they are there.
Related
We are planning on using Azure Application Insights for our web app. It has been suggested that we use two instances: one for error logging and the other for user analytics. While these are different needs, it seems like one instance can accommodate both needs. What is best practice?
[I'm from Application Insights team]
The best practice is that telemetry from one app should go to the same Application Insights resource.
The might be advanced (and rare) scenarios (for instance, one stream represents audit log and should be retained way longer and / or have different RBAC requirements) where it makes sense to send to different Application Insights resources.
Is there any easy way to find out which applications are using a particular application insights from azure portal?
I have checked the various options in the portals but don't find any easy to understand interface where I can find the list of applications which are sending data to that particular application insights.
The application map should provide you with a good view of various resources using the app insights resource
The application map is good. You can also go to Performance, then choose Roles. Roles is in the same tab group as Operations and Dependencies. This will give you a listing of all services that use that Application Insights instance. This has the added benefit of allowing you to expand a particular node and see the actual instances.
This same approach also works for the Failures tab. You can see the number of calls and failures rolled up per service, and also see the breakout metrics per instance.
I have an Azure Logic App that monitors an SFTP site for new files, and if it finds one, it sends a message to an Azure Queue for subsequent processing, then deletes the file. My application has grown in scale and a single logic app seems to only be grabbing 5-10 files a minute.
Is it possible to setup a second (third, fourth, etc.) Logic App that monitors the same SFTP site, without the two apps conflicting/colliding with each other. I also see that there is a "High Throughput" setting that seems interesting, but I'm not sure it is what I need. My ultimate goal is to process more files faster, and I am considering changing the Logic App out for a scheduled Web Job that monitors the SFTP site. Since I am live and files are pouring in, I am a little reluctant to change anything until I know it's safe.
Any insight would be appreciated.
Thanks!!
Logic app comes under server less architecture, IF we select the pricing model based on 'number of executions' then it impact on the performance since Microsoft allocates the resources for such kind of pricing model as shared one and which server is free up the processing. I would recommend to attache service plan to it and select the pricing model 'per minute'
One more point, If you want longer operations to be done then Azure logic app is not appropriate one but since you are connecting to enterprise integration then logic app is good choice. I would recommend to divide this functionality between logic app with Azure function OR Microsoft flow.
I have multiple services running on Service Fabric. I would like to add Application Insight for logging. I'm just wondering whether I have to add an Application Insight resource for each microservice or only one is common for all. What is the best practice?
There is no such thing a the best practice for this. It really depends. Some considerations:
Pricing: depending on the level (basic or enterprise) you will get an amount of data for free / included in the base price. See the docs. So in some cases, depending on the amount of traffic you can reduce costs by having a dedicated AI resource per service. AI resources for services that send data below the threshold of the AI pricing plan are then (almost) free.
Querying: if you split up services per AI resource getting an overview of the whole system is difficult since at the moment you cannot create queries spanning multiple AI resources.
Responsibility: If you have multiple teams working on multiple services it might be an option to have an AI resource per team so they have a good insight in only the parts they are responsible for.
If you do decide to use a shared AI resource there are options like custom telemetry initializers to include custom data that further identify which ASF application or service is sending the data if it is not included by default.
See also Add Application Insight to a existing Azure Service Fabric cluster for more info about how to integrate AI.
Now, when it comes to bring data together you do have some additional options that may or may not need additional services or configuration. For example:
PowerBi: You can visualize data of AI resources using dashboards, see https://learn.microsoft.com/en-us/azure/application-insights/app-insights-export-power-bi
OMS: Operation Management Suite, See https://blogs.technet.microsoft.com/msoms/2016/09/26/application-insights-connector-in-oms/. As Jesse mentions you can link multiple AI Resources
Custom dashboards: Using the rest api you can create your own solution that displays data for one or more AI resources.
Iam relatively new to Cloud Computing and azure. I was wondering whether you can have more than one web and worker role in an Azure application. If so what advantages can I get using multiple roles and where do they apply?
Yes, you can have more than 1 web or worker role in an Azure Cloud Service. You can have up to 25 different roles per deployment I believe in any mix of Web and Worker roles. See the Azure Subscription and Service Limits, Quotas and Constraints link for more information.
The advantage of having the roles within the same cloud service is simply that within that cloud service they can see all the other roles and instances easily (unless you configure them otherwise). They will all be relatively close to each other within a data center because a cloud service is assigned to a stamp of machines and controlled by a Fabric Controller assigned to that stamp. You can watch this video by Mark Russinovich which sheds more light on the inner workings of Azure and talks a bit about stamps I think. A cloud service is a security boundary as well, so you get some benefits from that encapsulation if you need to do a lot of inter machine communication that ISN'T going across a queue for some reason.
The disadvantage of batching a whole bunch of roles together is that they are tied pretty closely together at that point. You can certainly scale them separately, and you can do updates that target only a single role at a time. However, if you want to deploy changes to multiple roles you may end up having to do a full deployment to all roles (even those that haven't changed) or do updates to single roles one at a time until all the ones you need updated are, which can take some time. Of course, it could be argued that having them in separate cloud services would still have you doing updates concurrently depending on your architecture and/or dependencies.
My suggestion is to group only roles that REALLY belong together in the same solution. These are role that have workloads that are interrelated. Even then, there's nothing stopping you from separating these as well into separate deployments (though you may benefit from the security boundaries that being within the same cloud service). Think about how each role will be updated, and if they would generally be updated together or not. There are many factors in thinking about how to package roles together.