How can I use appInsights with HDInsight cluster - azure

Question

I have a spark cluster on azure hdinsight. Is there any way I can integrate azure applicationInsights with it to get all the monitoring and log analysis capabilties. This can be done through Azure Monitor Logs according to Microsoft docs but for some reason I need to specifically integrate my spark app on hdinsight with applicationInsights only. Couldn't find any documentation or example for the same anywhere.

Answer 1

Unfortunately, you cannot use Application Insight to get logs of HDInsight cluster.
Azure Monitor logs enable data generated by multiple resources, such as HDInsight clusters, to be collected and aggregated in one place to achieve a unified monitoring experience.
As a prerequisite, you'll need a Log Analytics Workspace to store the collected data. If you haven't already created one, you can follow instructions here: Create a Log Analytics Workspace.
What's the difference between Azure Monitor, Log Analytics, and
Application Insights?
In September 2018, Microsoft combined Azure Monitor, Log Analytics, and Application Insights into a single service to provide powerful end-to-end monitoring of your applications and the components they rely on. Features in Log Analytics and Application Insights have not changed, although some features have been rebranded to Azure Monitor in order to better reflect their new scope. The log data engine and query language of Log Analytics is now referred to as Azure Monitor Logs. See Azure Monitor terminology updates.
Can I use Application Insights with ...?
Web apps on an IIS server in Azure VM or Azure virtual machine scale
set Web apps on an IIS server - on-premises or in a VM
Java web apps
Node.js apps
Web apps on Azure
Cloud Services on Azure
App servers running in Docker
Single-page web apps
SharePoint
Windows desktop app
Other platforms

Answer 2

Found a way of making ApplicationInsights work with HdInsight (Spark) cluster. The application deployed on the cluster is a Spark application written in Scala (maven based). Though Microsoft doesn't have an SDK for Scala at this point, I was able to use the applicationinsights-logging-log4j dependency to send app logs as well as spark yarn logs to AppInsights, which was the end goal in my case.
Here's how to do it:
Add these dependencies to pom.xml
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>applicationinsights-core</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>applicationinsights-logging-log4j1_2</artifactId>
<version>2.6.1</version>
</dependency>
Use the ApplicationInsightsAppender class
import org.apache.log4j.{ Logger, Level }
import com.microsoft.applicationinsights.log4j.v1_2.ApplicationInsightsAppender
object AppInsightLogger {
var rootLogger = Logger.getRootLogger()
var ai = new ApplicationInsightsAppender()
ai.setInstrumentationKey("your-key")
ai.activateOptions()
#transient lazy val logger = Logger.getLogger(this.getClass)
logger.setLevel(Level.INFO)
rootLogger.addAppender(ai)
def info(message: String): Unit = {
logger.info(message)
}
}
Finally, this can be used anywhere in the application such as:
AppInsightLogger.info("Streaming messages from EH")
I was able to get Spark yarn logs as well as custom logs from the application deployed on HdInsight to AppInsights without using an SDK for scala! (The dashboards and telemetry data cannot be seen using this approach. We get the logs as "Trace" having different severity levels. We can see exceptions as well if any. Use the "Search" option on the portal for viewing the logs)
Logs seen on AppInsights
Exception logs seen on AppInsights