Is there a way to specify target log files for microsoft monitoring agent to listen and pick up the logs from code? - azure

Question

I am considering the use of Microsoft monitoring agent to collect some log records from log files on the system and send them to a log analytics workspace.
Is there a way specifying target files(custom log files) the agent would listen to and stream the logs directly to azure workspace.
I know this is possible to do through azure portal by adding an additional data source in the workspace(as specified by this link https://learn.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs).
I am looking for a way to configure these data sources from c# code/powershell script.(possibily api or sdk that i am not aware of ).

Answer 1

To add custom logs Use New-AzOperationalInsightsCustomLogDataSource.
Here are theother powershell commandlets which can be handy to query and create LogAnalytics Datasource.
get-azoperationalinsightsdatasource
New-AzOperationalInsightsApplicationInsightsDataSource
New-AzOperationalInsightsAzureActivityLogDataSource
New-AzOperationalInsightsComputerGroup
New-AzOperationalInsightsCustomLogDataSource
New-AzOperationalInsightsLinuxPerformanceObjectDataSource
New-AzOperationalInsightsLinuxSyslogDataSource
New-AzOperationalInsightsSavedSearch
New-AzOperationalInsightsStorageInsight
New-AzOperationalInsightsWindowsEventDataSource
New-AzOperationalInsightsWindowsPerformanceCounterDataSource
https://learn.microsoft.com/en-us/powershell/module/az.operationalinsights/get-azoperationalinsightsdatasource?view=azps-2.7.0
Also find the link for the Log analytics Rest API's which can be used easily with C# code.
https://learn.microsoft.com/en-us/rest/api/loganalytics/
https://learn.microsoft.com/en-us/rest/api/loganalytics/datasources/createorupdate
Powershell
Custom Log to collect
Link : https://learn.microsoft.com/en-us/azure/azure-monitor/platform/powershell-workspace-configuration
$CustomLog = #"
{
"customLogName": "sampleCustomLog1",
"description": "Example custom log datasource",
"inputs": [
{
"location": {
"fileSystemLocations": {
"windowsFileTypeLogPaths": [ "e:\\iis5\\*.log" ],
"linuxFileTypeLogPaths": [ "/var/logs" ]
}
},
"recordDelimiter": {
"regexDelimiter": {
"pattern": "\\n",
"matchIndex": 0,
"matchIndexSpecified": true,
"numberedGroup": null
}
}
}
],
"extractions": [
{
"extractionName": "TimeGenerated",
"extractionType": "DateTime",
"extractionProperties": {
"dateTimeExtraction": {
"regex": null,
"joinStringRegex": null
}
}
}
]
}
"#
# Custom Logs
New-AzOperationalInsightsCustomLogDataSource -ResourceGroupName $ResourceGroup -WorkspaceName $WorkspaceName -CustomLogRawJson "$CustomLog" -Name "Example Custom Log Collection"
ARM Template
For the Arm template format for the custom logs will be as below. See the detailed link https://learn.microsoft.com/en-us/azure/azure-monitor/platform/template-workspace-configuration
{
"apiVersion": "2015-11-01-preview",
"type": "dataSources",
"name": "[concat(parameters('workspaceName'), parameters('customlogName'))]",
"dependsOn": [
"[concat('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
],
"kind": "CustomLog",
"properties": {
"customLogName": "[parameters('customlogName')]",
"description": "this is a description",
"extractions": [
{
"extractionName": "TimeGenerated",
"extractionProperties": {
"dateTimeExtraction": {
"regex": [
{
"matchIndex": 0,
"numberdGroup": null,
"pattern": "((\\d{2})|(\\d{4}))-([0-1]\\d)-(([0-3]\\d)|(\\d))\\s((\\d)|([0-1]\\d)|(2[0-4])):[0-5][0-9]:[0-5][0-9]"
}
]
}
},
"extractionType": "DateTime"
}
],
"inputs": [
{
"location": {
"fileSystemLocations": {
"linuxFileTypeLogPaths": null,
"windowsFileTypeLogPaths": [
"[concat('c:\\Windows\\Logs\\',parameters('customlogName'))]"
]
}
},
"recordDelimiter": {
"regexDelimiter": {
"matchIndex": 0,
"numberdGroup": null,
"pattern": "(^.*((\\d{2})|(\\d{4}))-([0-1]\\d)-(([0-3]\\d)|(\\d))\\s((\\d)|([0-1]\\d)|(2[0-4])):[0-5][0-9]:[0-5][0-9].*$)"
}
}
}
]
}
}