azure webhook to stream splunk data - azure

I want to schedule splunk report to an azure web-hook and persist it into Cosmos DB.(after from processing ) This tutorial gave me some insight on how to process and persist data into cosmos db via the azure functions ( in java ). To solve the next part of the puzzle I"m reaching out for some advise on how to go about:
How to setup and host a webhook on Azure ?
Should I set a HttpTrigger , inside the EventHubOutput function and deploy it into the function app.? Or should I use the Webhook from Azure Event Grid ?(not clear on how to do this ). I'm NOT looking to stream any heavy volumes of data and want to keep the consumption cost low. So , which route should I take here?. Any pointers to tutorials will be of help here.
How do I handle a webhook data processing on #EventHubOutput ( referring the java example in the tutorial) ?. What is the setup and configuration I need to do here ? Any working examples will be of help .

I ended up using just the #HttpTrigger and binding the output using #CosmosDBOutput to persist the data. Something like this , would like to know if there are any better approaches.
public class Function {
#FunctionName("PostData")
public HttpResponseMessage run(
#HttpTrigger(
name = "req",
methods = {HttpMethod.GET, HttpMethod.POST},
authLevel = AuthorizationLevel.ANONYMOUS)
HttpRequestMessage<Optional<String>> request,
#CosmosDBOutput( name = "databaseOutput", databaseName = "SplunkDataSource",
collectionName = "loginData",
connectionStringSetting = "CosmosDBConnectionString")
OutputBinding<String> document,
final ExecutionContext context) {
context.getLogger().info("Java HTTP trigger processed a request.");
// Parse the payload
String data = request.getBody().get();
if (data == null) {
return request.createResponseBuilder(HttpStatus.BAD_REQUEST).body(
"Please pass a name on the query string or in the request body").build();
} else {
// Write the data to the Cosmos document.
document.setValue(data);
context.getLogger().info("Persisting payload to db :" + data);
return request.createResponseBuilder(HttpStatus.OK).body(data).build();
}
}

Related

How to trigger Synapse Analytics Workspace pipeline using C# or any other language?

Is there any way through which we can trigger the Azure Synapse Analytics Pipeline (built-in Azure Data Factory) using C# or any language?
Using the following URL and code I am able to trigger general (not part of Synapse) Azure Data Factory successfully. But when I call the same method I am not sure what will go under name of Data Factory (property: dataFactoryName)? I tried giving workspace name but it does not work.
Built-in ADF can be triggered using Blob trigger but thing is that I have many parameters and those cannot be passed through files stored in Blob.
URL: https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers
Code: client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName, parameters)
According to the ADF team, there is a different SDK for Synapse Analytics. I'm in the same position, but haven't had a chance to generate a code sample yet.
It looks like you'll need the PipelineClient class to create a run, and the PipelineRunClient class to monitor it.
If you get this working, please post the sample code for future searchers.
I have taken guidance from the following URL
https://techcommunity.microsoft.com/t5/azure-synapse-analytics/how-to-start-synapse-pipeline-from-rest-api/ba-p/1684836
and created a C# version of it here -
https://github.com/pankajsingh23/TriggerSynapsePipeline
public async Task<string> ExecuteSynapsePipeline(IDictionary<string, object?> parameters, string? synapseEndPoint = null, string? pipeline = null)
{
synapseEndPoint ??= _Settings.SynapseExtractEndpoint;
pipeline ??= _Settings.SynapseExtractPipeline;
_Logger.LogInformation($"!!Execute synapse pipeline called with parameters: {parameters}");
Response<CreateRunResponse?>? result = null;
try
{
PipelineClient client = new(new Uri(synapseEndPoint), new DefaultAzureCredential());
_Logger.LogInformation($"!!Pipeline {client} client resolved");
result = await client.CreatePipelineRunAsync(
pipeline,
parameters: parameters);
_Logger.LogInformation($"!!{pipeline} {result?.Value?.RunId ?? "Failed to get RunId"} called");
}
catch(Exception e)
{
string errorToLog = $"{e.Message} \n";
if(e.InnerException is not null && !string.IsNullOrEmpty(e.InnerException.Message))
{
errorToLog += e.InnerException.Message;
}
_Errors.LogError(options.OrchestratorId, errorToLog);
_Logger.LogError(errorToLog);
}
return result?.Value?.RunId ?? "";
}

Azure function inserting but not updating cosmosDB

I have an azure function taking in messages from an Azure service bus queue and sending documents to cosmosDB. I'm using Azure functions 1.x:
public static class Function1
{
[FunctionName("Function1")]
public static void Run([ServiceBusTrigger("ServiceBusQueue", AccessRights.Manage, Connection = "ServiceBusQueueConnection")]BrokeredMessage current, [DocumentDB(
databaseName: "DBname",
collectionName: "Colname",
ConnectionStringSetting = "CosmosDBConnection")]out dynamic document, TraceWriter log)
{
document = current.GetBody<MyObject>();
log.Info($"C# ServiceBus queue triggered function processed the message and sent to cosmos");
}
}
This inserts to cosmos successfully, but when updating I get errors:
Microsoft.Azure.Documents.DocumentClintException: Entity with the specified id already exists in the system.
They key I'm trying to update on is the partition key of that collection.
I saw this question: Azure function C#: Create or replace document in cosmos db on HTTP request
But It seems like my usage is similar to the one in Matias Quarantas answer. Also he mentioned that using an out parameter causes an upsert on cosmos.
How can I create this "upsert" function, while still using azure function 1.x?
The binding does indeed do an Upsert operation.
I created this sample Function that takes an Http payload (JSON) and stores it in Cosmos DB as-is:
[FunctionName("Function1")]
public static HttpResponseMessage Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req,
[DocumentDB("MyDb", "MyCollection", ConnectionStringSetting = "MyCosmosConnectionString")] out dynamic document,
TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
dynamic data = req.Content.ReadAsAsync<object>().GetAwaiter().GetResult();
document = data;
return req.CreateResponse(HttpStatusCode.OK);
}
If I send a JSON payload to the Http endpoint, the output binding works as expected:
When I check the Data Explorer, I see:
If I send a second payload, this time, adding a property (same id):
The Data Explorer shows the document was updated, with the same Function code:
Can you add the full Exception/Error trace? Is your Service Bus Message including an "id"? Is your collection partitioned?
If your collection is partitioned and you are changing the value of the Partition key property, then the binding won't update the existing document, it will create a new one because the Upsert operation won't find an existing document (based on the id/partition key). But it won't throw an exception.

Data From REST API In Azure

I have implemented REST API calls using a standalone c# console application. The API returns JSON which i'm deserializing and then storing it in the database.
Now i want to implement the entire logic in Azure platform so that it can invoked by passing start date and an end date and store location (it should run for three location) Below is the code:
static void Main()
{
MakeInventoryRequest();
}
static async void MakeInventoryRequest()
{
using (var client = new HttpClient())
{
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "5051fx6yyy124hhfyuscf34f57ce9");
// Request parameters
queryString["query.locationNumbers"] = "4638";
queryString["availableFromDate"] = "2019-01-01";
queryString["availableToDate"] = "2019-03-07";
var uri = "https://api-test.location.cloud/api/v1/inventory?" + queryString;
using (var request = new HttpRequestMessage(HttpMethod.Get, uri))
using (var response = await client.SendAsync(request))
{
var stream = await response.Content.ReadAsStreamAsync();
if (response.IsSuccessStatusCode == true)
{
List<Inventory> l1 = DeserializeJsonFromStream<List<Inventory>>(stream);
InsertInventoryRecords(l1);
}
if (response.IsSuccessStatusCode == false)
{
throw new Exception("Error Response Code: " + response.StatusCode.ToString() + "Content is: " + response.Content.ReadAsStringAsync().Result.ToString());
}
}
}
}
Please suggest the best possible design using Azure components
With the information in hand I think you have multiple options , you need to find out which works for you the best . You can use Cloud service to host the console app ( you will have to change it to worker role , Visual studio will help you to convert that ) . I am not sure about the load which you are expecting but you can always increase and decrease the instance and these can be deployed to different geographies .
I see that you are persisting the data , if you want to do that you can use many of the SQL offerings . For invoking the REST API you can also azure functions and ADF.
Please feel free to comment if you want any more details on the same.

How do I access Cosmos DB database or collection metrics via the Azure REST API?

I am attempting to use the Azure metrics API to retrieve metrics for Cosmos DB databases and collections.
I am able to use the metrics API to retrieve metrics for the Cosmos DB account itself, but I cannot figure out the resource URL for the databases or collections.
So this works:
public static async Task GetMetricsForCollection(ICosmosDBAccount cosmos, IDocumentClient client)
{
var uriBuilder = new System.Text.StringBuilder();
uriBuilder.Append($"https://management.azure.com{cosmos.Id}");
uriBuilder.Append($"/providers/microsoft.insights/metricDefinitions?api-version=2018-01-01");
//...Use uri to access API over HTTP
But I can't figure out how to get more specific metrics at deeper levels.
I found this post on the MSDN Community that says that this should work
public static async Task GetMetricsForCollection(ICosmosDBAccount cosmos, IDocumentClient client)
{
var db = client.CreateDatabaseQuery().AsEnumerable().First();
var uriBuilder = new System.Text.StringBuilder();
//Use the database resource Id to retrieve the metrics
uriBuilder.Append($"https://management.azure.com{cosmos.Id}/databases/{db.ResourceId}");
uriBuilder.Append($"/providers/microsoft.insights/metricDefinitions?api-version=2018-01-01");
//...Use uri to access API over HTTP
But it returns an error
Response status code does not indicate success: 400
Microsoft.DocumentDB/databaseAccounts/databases is not a supported platform metric namespace, supported ones are
Microsoft.LocationBasedServices/accounts,Microsoft.EventHub/namespaces,Microsoft.EventHub/clusters,Microsoft.ServiceBus/namespaces,
Microsoft.KeyVault/vaults,Microsoft.ClassicCompute/domainNames/slots/roles,Microsoft.ClassicCompute/virtualMachines,
Microsoft.Network/publicIPAddresses,Microsoft.Network/networkInterfaces,Microsoft.Network/loadBalancers,
Microsoft.Network/networkWatchers/connectionMonitors,Microsoft.Network/virtualNetworkGateways,Microsoft.Network/connections,
Microsoft.Network/applicationGateways,Microsoft.Network/dnszones,Microsoft.Network/trafficmanagerprofiles,
Microsoft.Network/expressRouteCircuits,Microsoft.EventGrid/eventSubscriptions,Microsoft.EventGrid/topics,Microsoft.EventGrid/extensionTopics,
Microsoft.Batch/batchAccounts,Microsoft.TimeSeriesInsights/environments,Microsoft.TimeSeriesInsights/environments/eventsources,
Microsoft.OperationalInsights/workspaces,Microsoft.Maps/accounts,Microsoft.Sql/servers,Microsoft.Sql/servers/databases,
Microsoft.Sql/servers/elasticpools,Microsoft.AnalysisServices/servers,Microsoft.Compute/virtualMachines,
Microsoft.Compute/virtualMachineScaleSets,Microsoft.Compute/virtualMachineScaleSets/virtualMachines,Microsoft.DataFactory/dataFactories,
Microsoft.DataFactory/factories,Microsoft.Storage/storageAccounts,Microsoft.Storage/storageAccounts/blobServices,
Microsoft.Storage/storageAccounts/tableServices,Microsoft.Storage/storageAccounts/queueServices,
Microsoft.Storage/storageAccounts/fileServices,Microsoft.Logic/workflows,Microsoft.Automation/automationAccounts,
Microsoft.ContainerService/managedClusters,Microsoft.StorageSync/storageSyncServices,Microsoft.ApiManagement/service,
Microsoft.DBforMySQL/servers,Microsoft.DocumentDB/databaseAccounts,Microsoft.ContainerRegistry/registries,Microsoft.Search/searchServices,
microsoft.insights/components,microsoft.insights/autoscalesettings,Microsoft.DataLakeStore/accounts,Microsoft.Web/serverFarms,
Microsoft.Web/sites,Microsoft.Web/sites/slots,Microsoft.Web/hostingEnvironments/multiRolePools,Microsoft.Web/hostingEnvironments/workerPools,
Microsoft.HDInsight/clusters,Microsoft.NetApp/netAppAccounts/capacityPools,Microsoft.NetApp/netAppAccounts/capacityPools/volumes,
test.shoebox/testresources,test.shoebox/testresources2,Microsoft.NotificationHubs/namespaces/notificationHubs,Microsoft.CustomerInsights/hubs,
CloudSimple.PrivateCloudIaaS/virtualMachines,Microsoft.StreamAnalytics/streamingjobs,Microsoft.CognitiveServices/accounts,
Microsoft.Cache/Redis,Microsoft.Devices/IotHubs,Microsoft.Devices/ElasticPools,Microsoft.Devices/ElasticPools/IotHubTenants,
Microsoft.Devices/ProvisioningServices,Microsoft.SignalRService/SignalR,Microsoft.DataLakeAnalytics/accounts,
Microsoft.DBforPostgreSQL/servers,Microsoft.ContainerInstance/containerGroups,Microsoft.Relay/namespaces,
Microsoft.PowerBIDedicated/capacities
(So you don't have to read all of that, I can confirm that it doesn't mention collections or databases as being usable with this API.)
I've also tried it with db.Id instead of db.ResourceId with the same error.
I've also tried going to the collection with uriBuilder.Append($"https://management.azure.com{cosmos.Id}/databases/{db.ResourceId}/collections/{collection.ResourceId}"); but it also generates the same message.
I'm stumped.
After poking around a bit with network traces while exploring in the Azure Portal, it looks like there are two types of Cosmos metrics: those that use the microsoft.insights provider and those that don't.
For those that use the provider, you can add the database and collection (the human readable names, aka the .Id property) as filters to the metrics API:
public static async Task GetMetricsForCollection(ICosmosDBAccount cosmos, IDocumentClient client)
{
var db = client.CreateDatabaseQuery().AsEnumerable().First();
var dbUri = UriFactory.CreateDatabaseUri(db.Id);
var collection = client.CreateDocumentCollectionQuery(dbUri).AsEnumerable().First();
var uriBuilder = new System.Text.StringBuilder();
//Use the database resource Id to retrieve the metrics
uriBuilder.Append($"https://management.azure.com{cosmos.Id}");
uriBuilder.Append($"/providers/microsoft.insights/metricDefinitions?api-version=2018-01-01");
uriBuilder.Append($"&$filter=DatabaseName eq '{db.Id}' and CollectionName eq '{collection.Id}'");
For those that don't, you can add /metrics to the resource URI suggested by the linked forum post. In this case, you need to use the .ResourceId properties.
It also looks like a filter parameter is required. I'm just copying and pasting from the Azure portal's network traces as I don't believe this is documented anywhere but it ends up looking something like
public static async Task GetMetricsForCollection(ICosmosDBAccount cosmos, IDocumentClient client)
{
var db = client.CreateDatabaseQuery().AsEnumerable().First();
var dbUri = UriFactory.CreateDatabaseUri(db.Id);
var collection = client.CreateDocumentCollectionQuery(dbUri).AsEnumerable().First();
var uriBuilder = new System.Text.StringBuilder();
uriBuilder.Append($"https://management.azure.com{cosmos.Id}");
uriBuilder.Append($"/databases/{db.ResourceId}/collections/{collection.ResourceId}/metrics?api-version=2014-04-01");
uriBuilder.Append($"&$filter=(name.value eq 'Available Storage' or name.value eq 'Data Size' or name.value eq 'Index Size') and endTime eq 2018-06-22T12%3A35%3A00.000Z and startTime eq 2018-06-22T11%3A35%3A00.000Z and timeGrain eq duration'PT5M'");

Azure Durable Function (external functions)

I developed a couple of microservice using Azure functions, every service has independent use case and different programming language.
Now I have a use case to use all service in below order, So I developed one more Azure function to use all service in given order. below code running well.
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
string returnValue = string.Empty;
dynamic data = await req.Content.ReadAsStringAsync();
if (data == null)
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a value in the request body");
}
else
{
string body = data.ToString();
var transformResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.TransformServiceEndPoint, body, HttpMethod.POST);
var validationResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.ValidationServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
if (validationResult.Result != null && Convert.ToBoolean(validationResult.Result))
{
var encryptionResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.EncryptionServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
var storageResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.StorageServiceEndPoint, encryptionResult.Result.ToString(), HttpMethod.POST);
returnValue = storageResult.Result.ToString();
}
else
{
returnValue = "Validation Failed";
}
return req.CreateResponse(HttpStatusCode.OK, returnValue, "text/plain");
}
}
Question
If every microservice takes 1 min to execution, I have to wait ~4min in my Super Service and billed for 4+ min. (We don't need to pay for waiting time :) https://www.youtube.com/watch?v=lVwWlZ-4Nfs)
I want to use Azure Durable functions here but didn't get any method to call external url.
Please help me or suggest a better solution.
Thanks In Advance
Durable Orchestration Functions don't work with arbitrary HTTP endpoints. Instead, you need to create individual functions as Activity-triggered.
Orchestration will use message queues behind the scenes rather than HTTP. HTTP is request-response in nature, so you have to keep the connection and thus pay for it.
Queue-based orchestrator can also give you some extra resilience in face of intermittent failures.

Resources