I've been trying to get started with Azure Event Hubs & Azure functions.
While following the exact documentation - I'm unable to get the event hub trigger to work.
I have the following setup:
Function I call in an HTTP function (to generate event logs):
#EventHubOutput(name = "message-new", eventHubName = "KCETest1", connection = "KCETest1_all_EVENTHUB", dataType = "string" )
public String sendOrder(ExecutionContext context) {
return "foobar";
}
Snapshot of my event hub showing events are being triggered:
Event Hub Receiving messages
However, I'm not able to get anything to work to trigger an azure function.
I tried all the below just to get at least something written to the logs.
(as you can see - I've been playing around with all variables - thinking this might be the rootcause)
public class EventHubTriggerJava1 {
/**
* This function will be invoked when an event is received from Event Hub.
*/
#FunctionName("EventHubTriggerJava1")
public void run(
#EventHubTrigger(name = "msg", eventHubName = "KCETest1", connection = "KCETest1_all_EVENTHUB", consumerGroup = "$Default", cardinality = Cardinality.ONE , dataType = "string") String message,
final ExecutionContext context
) {
context.getLogger().info("Java Event Hub trigger 1function executed.");
context.getLogger().info("Length:" + message);
}
#FunctionName("EventHubTriggerJava2")
public void run(
#EventHubTrigger(name = "msg2", eventHubName = "testhub1", connection = "KCETest1_all_EVENTHUB", consumerGroup = "$Default", cardinality = Cardinality.ONE) EventData message,
final ExecutionContext context
) {
context.getLogger().info("Java Event Hub trigger 2function executed.");
context.getLogger().info("Length:" + message.toString());
}
#FunctionName("EventHubTriggerJava3")
public void run(
#EventHubTrigger(name = "msg3", eventHubName = "testhub1", connection = "KCETest1_all_EVENTHUB", consumerGroup = "$Default") List<String> message,
final ExecutionContext context
) {
context.getLogger().info("Java Event Hub trigger 3 function executed.");
context.getLogger().info("Length:" + message.get(0).toString());
}
}
I'm running out of ideas - any suggestions from your side?
Thx a lot!
Related
When I create a scheduled service bus message, both in Azure Portal and in my app using the Service bus producer code (below) and I receive a sequence number. I save it in my db.
Problem - When my Service bus consumer code is triggered by the dequeue of the scheduled message the sequence number is different than the one that was initially given to me by both the service bus producer code and through the Azure portal.
Shown here, where '13' is the sequnce number shown in Azure Portal screen.
Here is the code that receives the scheduled message and you can see the sequence number is different!
Here is my consumer code (don't think it matters)
private async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
JObject jsonObject = JObject.Parse(body);
var eventStatus = (string)jsonObject["EventStatus"];
await args.CompleteMessageAsync(args.Message);
// fetch row here by sequence number
// edit some data from entity, then save
int result = await dbContext.SaveChangesAsync();
}
Here is my producer code
public async Task<long> SendMessage(string messageBody, DateTimeOffset scheduledEnqueueTimeUtc)
{
await using (ServiceBusClient client = new ServiceBusClient(_config["ServiceBus:Connection"]))
{
ServiceBusSender sender = client.CreateSender(_config["ServiceBus:Queue"]);
ServiceBusMessage message = new ServiceBusMessage(messageBody);
var sequenceNumber = await sender.ScheduleMessageAsync(message, scheduledEnqueueTimeUtc);
return sequenceNumber;
}
}
From the documentation:
The SequenceNumber for a scheduled message is only valid while the message is in this state. As the message transitions to the active state, the message is appended to the queue as if had been enqueued at the current instant, which includes assigning a new SequenceNumber.
This is the code on my side:
using System;
using System.Threading;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus;
namespace ConsoleApp3
{
class Program
{
static string connectionString = "xxxxxx";
static string queueName = "myqueue";
static ServiceBusClient client;
static ServiceBusProcessor processor;
static async Task Main(string[] args)
{
client = new ServiceBusClient(connectionString);
processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
try
{
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
await processor.DisposeAsync();
await client.DisposeAsync();
}
}
static async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine($"Received: {body}");
Console.WriteLine($"ID: {args.Message.MessageId}");
await args.CompleteMessageAsync(args.Message);
}
static Task ErrorHandler(ProcessErrorEventArgs args)
{
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
}
}
And it seems no problem on my side:
Message Id changed should be the message be thrown back by some reasons.
I need to trigger an Azure Function based on Service Bus message that will do some logic and will write back to Service Bus some message that will potentially trigger another Azure function etc..
I have lack of understanding how to do it properly in standard way.
Based on this document Azure Service Bus trigger for Azure Functions we can do first part: trigger azure function by Service Bus message.
Code:
#FunctionName("sbtopicprocessor")
public void run(
#ServiceBusTopicTrigger(
name = "message",
topicName = "mytopicname",
subscriptionName = "mysubscription",
connection = "ServiceBusConnection"
) String message,
final ExecutionContext context
) {
context.getLogger().info(message);
}
Based on this document Azure Service Bus output binding for Azure Functions we can do second part: trigger output message to Service Bus.
Code:
#FunctionName("sbtopicsend")
public HttpResponseMessage run(
#HttpTrigger(name = "req", methods = {HttpMethod.GET, HttpMethod.POST}, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request,
#ServiceBusTopicOutput(name = "message", topicName = "mytopicname", subscriptionName = "mysubscription", connection = "ServiceBusConnection") OutputBinding<String> message,
final ExecutionContext context) {
String name = request.getBody().orElse("Azure Functions");
message.setValue(name);
return request.createResponseBuilder(HttpStatus.OK).body("Hello, " + name).build();
}
But I need both input / output functionalities within one function ? Should I call second function from the first one via http which seems for me a little bit awkward or should I use Service bus sdk within fist function.
Thanks for any help.
I don't work with Java but you can combine the Trigger and the Output in one function.
#FunctionName("sbtopicprocessor")
public void run(
#ServiceBusTopicTrigger(
name = "message",
topicName = "mytopicname",
subscriptionName = "mysubscription",
connection = "ServiceBusConnection"
) String messageRequest,
#ServiceBusTopicOutput(name = "message", topicName = "mytopicname", subscriptionName = "mysubscription", connection = "ServiceBusConnection") OutputBinding<String> message, final ExecutionContext context
) {
message.setValue(messageRequest.name);
}
You can combine any type of Trigger with any type of Output in one function.
I am working with Azure Event Hubs. My requirement is to fetch the events from Azure Event Hub, using azure function on a daily basis. Basically my azure function will be timer enabled. It should be able to fetch the data from azure event hubs. Is there a mechanism for this ?
I am aware that we can trigger a azure function whenever an event is received at event hub. This i don't want as the function will execute n number of time. I want to just fetch the events on a daily basis.
You can still create a timer triggered function and create consumer clients in your code to receive events. See sample code below. Let me know if you have any questions.
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Azure.Messaging.EventHubs.Consumer;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace FunctionApp7
{
public static class Function1
{
const string EventHubsConnectionString = "your connection string";
const string EventHubName = "evethub name";
const string ConsumerGroupName = "cgname";
[FunctionName("Function1")]
public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
// You better dissover partitions by eventhub client. I am just hardcoding them here for now.
var partitions = new List<string> { "0", "1" };
var receiveTasks = new List<Task>();
foreach(var p in partitions)
{
receiveTasks.Add(ReadEventsFromPartition(p));
}
// Wait until all reads complete.
Task.WhenAll(receiveTasks);
}
public static async Task ReadEventsFromPartition(string partitionId)
{
await using (var consumer = new EventHubConsumerClient(ConsumerGroupName, EventHubsConnectionString, EventHubName))
{
EventPosition startingPosition = EventPosition.FromOffset(CheckpointStore.ReadOffsetForPartition(partitionId));
long lastOffset = -1;
await foreach (PartitionEvent receivedEvent in consumer.ReadEventsFromPartitionAsync(partitionId, startingPosition))
{
// Process received events here.
// Break if no events left.
if (receivedEvent.Data == null)
{
break;
}
lastOffset = receivedEvent.Data.Offset;
}
// Persist last event's offset so we can continue reading from this position next time function is triggered.
if (lastOffset != -1)
{
// Write offset into some durable store.
CheckpointStore.WriteOffsetForPartition(partitionId, lastOffset);
}
}
}
}
}
I have an Azure function that is triggered by IOThub. So in the Azure function, I have
public static async Task Run(EventData myIoTHubMessage1, TraceWriter log)
How do I get the device id from the Event Data.
I tried
log.Info("devid="+myIoTHubMessage1.SystemProperties["ConnectionDeviceId"]);
It gave an error saying
The given key was not present in the dictionary.
the following document says that
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct
ConnectionDeviceId contains the deviceId. Would anybody know how to retrieve the deviceid from EventData or should I use some other class.
You can get device ID from SystemProperties:
public static async Task Run(EventData myIoTHubMessage1, TraceWriter log)
{
var deviceId = myIoTHubMessage1.SystemProperties["iothub-connection-device-id"];
// ....
}
for (EventData receivedEvent : receivedEvents) {
String deviceId = (String) receivedEvent.getProperties().get("deviceId");
log.info("From:" + deviceId);
}
I am using WebJob's bindings to EventHub as described here:
https://github.com/Azure/azure-webjobs-sdk/wiki/EventHub-support
While the webjob is running, trying to run the Azure Service Bus Explorer on the same hub result in this exception:
Exception: A receiver with a higher epoch '14' already exists. A new receiver with epoch 0 cannot be created.
Make sure you are creating receiver with increasing epoch value to ensure connectivity, or ensure all old epoch receivers are closed or disconnected.
From what I understand, this is caused by the 2 listeners(webjob & bus explorer) using the same Consumer Group.
So my question, how can I specify a different Consumer Group in my webjob ?
My current code look like this:
Program.cs:
var config = new JobHostConfiguration()
{
NameResolver = new NameResolver()
};
string eventHubConnectionString = ConfigurationManager.ConnectionStrings["EventHub"].ConnectionString;
string eventHubName = ConfigurationManager.AppSettings["EventHubName"];
string eventProcessorHostStorageConnectionString = ConfigurationManager.ConnectionStrings["EventProcessorHostStorage"].ConnectionString; ;
var eventHubConfig = new EventHubConfiguration();
eventHubConfig.AddReceiver(eventHubName, eventHubConnectionString, eventProcessorHostStorageConnectionString);
config.UseEventHub(eventHubConfig);
var host = new JobHost(config);
host.RunAndBlock();
Functions.cs:
public class Functions
{
public static void Trigger([EventHubTrigger("%EventHubName%")] string message, TextWriter log)
{
log.WriteLine(message);
}
}
[Edit - Bonus Question]
I don't fully grasp the use of Consumer Group and 'epoch' thing. One Consumer Group is limited to one receiver ?
The EventHubTrigger has an optional ConsumerGroup property (source). So, based on that modify the trigger like this:
public class Functions
{
public static void Trigger([EventHubTrigger("%EventHubName%", ConsumerGroup = "MyConsumerGroup")] string message, TextWriter log)
{
log.WriteLine(message);
}
}