Some streams terminated before this command could finish! Structured Streaming - azure

Question

I am trying to read streaming data into Azure Databricks coming from Azure Eventhubs.
This is the code i've been using:
connectionString = "Connection string"
ehConf = {
'eventhubs.connectionString' : connectionString
}
df = spark \
.readStream \
.format("eventhubs") \
.options(**ehConf) \
.load()
query = df \
.writeStream \
.outputMode("append") \
.format("console") \
.start()
And its giving me an error saying:
ERROR: Some streams terminated before this command could finish!
I understood that we have to give the Jar file of Azure Eventhub according to the Databricks run time and also according to the spark version.
My spark version is 2.4.5 and Databricks runtime is 6.6, and the jar file i used is azure-eventhubs-spark_2.12-2.3.17.jar for this combination as specified
But i'm still facing this issue as "Some streams terminated before this command could finish!".Can anyone please help me on this.
Thanks

Answer 1

As I started working on this issue: First faced the same issue as you are experiencing.
ERROR: Some streams terminated before this command could finish!
After making this changes, it works perfectly with the below configuration:
Databrick Runtime: 6.6 (includes Apache Spark 2.4.5, Scala 2.11)
Azure EventHub library: com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.17
Step1: Install libraries using Library.
You can try to install "com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.17" using Install Library option.
Step2: Change the configuration related to Azure Event Hubs.
If you are using "ehConf = {'eventhubs.connectionString' : connectionString}" with version above 2.3.15, you will receive the below error message.
java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit
Note: All configuration relating to Event Hubs happens in your Event Hubs configuration dictionary. The configuration dictionary must contain an Event Hubs connection string:
connectionString = "YOUR.CONNECTION.STRING"
ehConf = {}
ehConf['eventhubs.connectionString'] = connectionString
For **2.3.15** version and above, the configuration dictionary requires that connection string be encrypted.
ehConf['eventhubs.connectionString'] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)