Having difficulties outputting from my function to an eventhub and finally into ADX when I want to target a table to route to. I have had no issues hitting target tables with the Node SDK via EventHubProducerClient in which case, you simply specify those properties next to the body of the event your sending:
{
body: {
some:"fieldValue"
},
properties: {
Table: 'table_name',
Format: 'JSON',
IngestionMappingReference: 'table_name_mapping'
}
}
But doing this in the same manner when using the output binding documented in Azure Event Hubs output binding for Azure Functions where the messages that I would push would take the form of the above JS object does not work. The SDK documentation is non-helpful.
I can confirm that the data is in fact flowing from the Function to the Eventhub and into ADX if and only if I change the adx data connection for said eventhub to target a specific table (the opposite of the behavior I want) which is documented in Ingest data from event hub into Azure Data Explorer.
Any help would be greatly appreciated, this seems so silly!
Edit: grammar
The returned object is set as the data payload of the Event Hub Message that the Azure Functions runtime returns. Unfortunately, there is no way to change this from the JS Function itself.
In a C# Function you can return an EventData object but this isn't supported in non-C# languages.
You're only option, if you need your function to be in JS, is to use the Event Hub SDK directly.
Related
I have function app where I have one HttpTrigger and 3 BlobTrigger functions. After I deployed it, http trigger is working fine but for others functions which are blob triggers, it gives following errors
"Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function " for one function
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener' for function
" for another two
I verified with other environments and config values are same/similar so not sure why we are getting this issue in one environment only. I am using consumption mode.
Update: When file is placed in a blob function is not getting triggered.
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function
I was observed the same message when working on the Azure Functions Queue Trigger:
This message doesn't mean the error in function. Due to timeout of Function activity, this message will appear in the App Insights > Traces.
I have stopped sending the messages in the Queue for some time and has been observed the traces like Web Job Host Stopped and if you run the function again or any continuous activity is present in the Function, then this message will not appear in the traces.
If you are using elastic Premium and has VNET integrated, the non-http trigers needs Runtime scale monitoring enabled.
You can find Function App-->Configuration--> Function runtime settings and turn on Runtime scale monitoring.
If function app and storage account which holds the metadata of the function Private linked, you will need to add the app settings WEBSITE_CONTENTOVERVNET = 1.
Also, make sure you have private linked for blob, file, table and queue on storage account.
I created ticket with MS to fix this issue. After analysis I did some code changes as
Function was async but returning void so changed to return Task.
For the trigger I was using connection string from app settings. But then I changed it to azureWebJobStorage(even though bobth were same) in function trigger attribute param
It started working. So posting here in case it is helpful for others
I am looking to connect Azure data bricks to Event hub and read it through splunk . Initially I was able to send a test message and was able to receive the events in splunk(It was possible using scala as per https://learn.microsoft.com/en-us/azure/databricks/scenarios/databricks-stream-from-eventhubs --> Send tweets to event hubs). Now I am trying to implement the same using python using the reference from https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send ---> Send Event . When I try to pass the object-parameters, it throws an error like unable to import from Azure Event hubs
Can anyone help me understand how can I connect Azure databricks with Azure Eventhubs and include sending object -parameters ?
PS : I have added the necessary libraries required to the cluster as below :
com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.18
azure-eventhubs-spark_2.11
com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22
org.apache.spark:spark-avro_2.12:3.1.1
I have checked it with different versions of libraries too.
Can someone help me with the syntax part on how to pass the object parameters in the format of key value pairs ?
It's simple to connect to EventHubs from Azure Databricks - just follow official documentation, specifically the section Writing Data to Event Hubs (example is for batch write):
writeConnectionString = "SharedAccessSignatureFromAzurePortal"
ehWriteConf = {
'eventhubs.connectionString' :
sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(writeConnectionString)
}
# Write body data from a DataFrame to EventHubs.
# Events are distributed across partitions using round-robin model.
ds = df \
.select("body") \
.write \
.format("eventhubs") \
.options(**ehWriteConf) \
.save()
You need to construct the body column somehow - by encoding your data as JSON using the to_json(struct("*")), or encoding data as Avro...
But you also have a problem in your cluster configuration - specifically these two libraries: azure-eventhubs-spark_2.11 and com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.22 - they are for Spark 2, but you use Spark 3. Uninstall them, and restart the cluster.
I'm using the Functions Framework with Python alongside Google Cloud Pub/Sub Emulator. I'm having issues with an event triggered from a published message to a topic, where there's no event data found for the function. See more details below.
Start Pub/Sub Emulator under http://localhost:8085 and project_id is local-test.
Spin up function with signature-type: http under http://localhost:8006.
Given a background cloud function with signature-type: event:
Topic is created as test-topic
Function is spinned up under http://localhost:8007.
Create push subscription test-subscription for test-topic for endpoint: http://localhost:8007
When I publish a message to test-topic from http://localhost:8006 via POST request in Postman, I get back a 200 response to confirm the message was published successfully. The function representing http://localhost:8007 gets executed as an event as shown in the logs from the functions-framework. However, there's no actual data for event when debugging the triggered function.
Has anyone encountered this? Any ideas/suggestions on this?Perhaps, this is true? #23 Functions Framework does not work with the Pub/Sub emulator
Modules Installed
functions-framework==2.1.1
google-cloud-pubsub==2.2.0
python version
3.8.8
I'll close this post, since the issue is an actual bug that was reported last year.
Update: As a workaround until this bug is fixed, I copied the code below locally to functions_framework/__init__.py within view_func nested function, inside _event_view_func_wrapper function.
if 'message' in event_data:
if 'data' not in event_data:
message = event_data['message']
event_data['data'] = {
'data': message.get('data'),
'attributes': message.get('attributes')
}
I'm trying to solve for having thousands of IoT devices deployed, all logging events to Azure IoT hub, then being able to read events created by a single deviceid only.
I have been playing with EventProcessorHost to get something like this working, but so far I can only see a way to read all messages from all devices.
Its not a feasible solution to read all the messages and filter client side as there may be millions of messages.
The major purpose of the Azure IoT Hub is an ingestion of mass events from the devices to the cloud stream pipeline for their analyzing in the real-time manner. The default telemetry path (hot way) is via a built-in Event Hub, where all events are temporary stored in the EH partitions.
Besides that default endpoint (events), there is also capability to route an event message to the custom endpoints based on the rules (conditions).
Note, that the number of custom endpoints is limited to 10 and the number of rules to 100. If this limit is matching your business model, you can very easy to stream 10 devices individually, like is described in the Davis' answer.
However, splitting a telemetry stream pipeline based on the sources (devices) over this limit (10+1), it will require to use additional azure entities (components).
The following picture shows a solution for splitting a telemetry stream pipeline based on the devices using a Pub/Sub push model.
The above solution is based on forwarding the stream events to the Azure Event Grid using a custom topic publisher. The event schema for Event Grid eventing is here.
The Custom Topic Publisher for Event Grid is represented by Azure EventHubTrigger Function, where each stream event is mapped into the Event Grid event message with a subject indicated a registered device.
The Azure Event Grid is a Pub/Sub loosely decoupled model, where the events are delivered to the subscribers based on their subscribed subscriptions. In other words, if there is no match for delivery, the event message is disappeared.
Note, that the capable of Event Grid routing is 10 millions events per second per region. The limit of the number of subscriptions is 1000 per region.
Using the REST Api, the subscription can be dynamically created, updated, deleted, etc.
The following code snippet shows an example of the AF implementation for mapping the stream event to the EG event message. As you can see it is very straightforward implementation:
run.csx:
#r "Newtonsoft.Json"
#r "Microsoft.ServiceBus"
using System.Configuration;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.ServiceBus.Messaging;
using Newtonsoft.Json;
// reusable client proxy
static HttpClient client = HttpClientHelper.Client(ConfigurationManager.AppSettings["TopicEndpointEventGrid"], ConfigurationManager.AppSettings["aeg-sas-key"]);
// AF
public static async Task Run(EventData ed, TraceWriter log)
{
log.Info($"C# Event Hub trigger function processed a message:{ed.SequenceNumber}");
// fire EventGrid Custom Topic
var egevent = new EventGridEvent()
{
Id = ed.SequenceNumber.ToString(),
Subject = $"/iothub/events/{ed.SystemProperties["iothub-message-source"] ?? "?"}/{ed.SystemProperties["iothub-connection-device-id"] ?? "?"}",
EventType = "telemetryDataInserted",
EventTime = ed.EnqueuedTimeUtc,
Data = new
{
sysproperties = ed.SystemProperties,
properties = ed.Properties,
body = JsonConvert.DeserializeObject(Encoding.UTF8.GetString(ed.GetBytes()))
}
};
await client.PostAsJsonAsync("", new[] { egevent });
}
// helper
class HttpClientHelper
{
public static HttpClient Client(string address, string key)
{
var client = new HttpClient() { BaseAddress = new Uri(address) };
client.DefaultRequestHeaders.Add("aeg-sas-key", key);
return client;
}
}
function.json:
{
"bindings": [
{
"type": "eventHubTrigger",
"name": "ed",
"direction": "in",
"path": "<yourEventHubName>",
"connection": "<yourIoTHUB>",
"consumerGroup": "<yourGroup>",
"cardinality": "many"
}
],
"disabled": false
}
project.json:
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.Azure.EventGrid": "1.1.0-preview"
}
}
}
}
Finally, the following screen snippet shows an event grid event message received by AF subscriber for Device1:
If you're ok with Java/Scala, this example shows how to create a client and filter messages by device Id:
https://github.com/Azure/toketi-iothubreact/blob/master/samples-scala/src/main/scala/A_APIUSage/Demo.scala#L266
The underlying client reads all the messages from the hub though.
You could also consider using IoT Hub message routing, more info here:
https://azure.microsoft.com/blog/azure-iot-hub-message-routing-enhances-device-telemetry-and-optimizes-iot-infrastructure-resources
https://azure.microsoft.com/blog/iot-hub-message-routing-now-with-routing-on-message-body
I am using the relatively new ILogger (vs. TraceWriter) option in Azure functions and trying to understand how logs are captured.
Here's my function:
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, ILogger log)
{
log.LogTrace("Function 1 {level}", "trace");
log.LogWarning("Function 1 {level}", "warning");
log.LogError("Function 1 {level}", "error");
return req.CreateResponse(HttpStatusCode.OK, "Success!!!!");
}
When I look at the server logs, the LogFiles directory has a hierarchy.
The yellow highlighted file includes my log statements:
2017-08-19T13:58:31.814 Function started (Id=d40f2ca6-4cb6-4fbe-a05f-006ae3273562)
2017-08-19T13:58:33.045 Function 1 trace
2017-08-19T13:58:33.045 Function 1 warning
2017-08-19T13:58:33.045 Function 1 error
2017-08-19T13:58:33.075 Function completed (Success, Id=d40f2ca6-4cb6-4fbe-a05f-006ae3273562, Duration=1259ms)
The structured directory contains nothing here, but it seems to have various "codeddiagnostic" log statements in my real function applications directory.
What should I expect here? Ultimately, I would like to have a single sink for logging from all of my application components and take advantage of structured logging across the board.
The best way to collect structured logging from Azure Functions is to use Application Insights. One you defined that your Logger is based on ILogger, you can define a Template that specifies the properties you want to log. Then on Application Insights traces, using the Application Insights Query Language (aka Kusto) you can access the values of each of these properties with the name customDimensions.prop__{name}.
You can find a sample of how to do this with Azure Functions v2 on this post https://platform.deloitte.com.au/articles/correlated-structured-logging-on-azure-functions
Just FYI: Azure Functions running in isolated mode (.NET5 and .NET6) don't support structured logging using the ILogger from DI or provided in the FunctionContext. As of November 2021, there is an open bug about it: https://github.com/Azure/azure-functions-dotnet-worker/issues/423
As I understand it, the bug happens because all ILogger calls goes through the GRPC connection between the host and the isolated function, and in that process the message gets formatted instead of sending the original format and arguments. The Azure Insights connection that would record the structured log runs on the host and only receives the final message.
I plan on investigating some king of workaround, playing with direct access to Azure Insights inside the isolated process. If it works, I'll post the workaround in a comment at the bug linked above.
I had the same question. The log file logger doesn't really respect structured logging, but if you use AppInsights for Azure Functions it will in fact add custom properties for your structured logging
I had a conversation going about it here
https://github.com/Azure/azure-webjobs-sdk-script/issues/1675