Elk Logstash not able to connect to Event Hub Azure - azure

Question

I am trying to connect to Azure Event hub from ElasticSearch logstash. Here is the details of azure module I mentioned in /etc/logstash/logstash.yml file.
modules:
- name: azure
var.elasticsearch.hosts: localhost:9200
var.kibana.host: localhost:5601
var.input.azure_event_hubs.consumer_group: "default"
var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=ABC;AccountKey=XXXXX==;EndpointSuffix=core.windows.net"
var.input.azure_event_hubs.threads: 9
var.input.azure_event_hubs.event_hub_connections:
- "Endpoint=sb://elk-event.servicebus.windows.net/;SharedAccessKeyName=share-log;SharedAccessKey=YYYYY=;EntityPath=elk-log-event"
But from when I launch logstash using command:
./logstash --path.settings=/etc/logstash
I used consumer_group as logstash as well as default. Is it an issue?
it throws error as:
[2019-10-04T09:43:46,332][ERROR][com.microsoft.azure.eventprocessorhost.PartitionPump] host logstash-8331671e-0dea-4f8d-88de-ba3ebc974dca: 1: PartitionReceiver creation failed
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_222]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_222]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_222]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_222]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_222]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_222]
at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessageReceiver.cancelOpen(MessageReceiver.java:357) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessageReceiver.onOpenComplete(MessageReceiver.java:347) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:414) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:736) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:74) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:113) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) ~[azure-eventhubs-2.2.0.jar:?]
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) ~[proton-j-0.33.0.jar:?]
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.33.0.jar:?]
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.33.0.jar:?]
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.33.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) ~[azure-eventhubs-2.2.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
at com.microsoft.azure.eventhubs.impl.ExceptionUtil.distinguishNotFound(ExceptionUtil.java:86) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:31) ~[azure-eventhubs-2.2.0.jar:?]
at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:735) ~[azure-eventhubs-2.2.0.jar:?]
... 15 more
Is it related with consumer group or something else? If so, I am not sure how to create my own consumer group.
Any feedback? Thanks in advance.

Answer 1

The name for the consumer group needs to be $Default
//edit: I should note that it is generally not advisable to use the $Default consumer group. It is best practice to create a new consumer group for each application that is reading data from your Event Hub - even it there is only one app. You never know when somebody might have another idea and also wants to read the data as well... Plus, having named consumer groups gives you an easy overview of what apps are reading from your hub.