Pyspark write dataframe to kafka topic in avro format - apache-spark

Question

I am trying to do a POC where I need to read a csv file and write to a kafka topic.
I am using Aws Glue to do this job,
df = gc.create_data_frame_from_options("s3",{"paths": ["s3://glue-dev-us-west-1/gluetestjobs/kafka_avro_test/input.csv"]},format='csv')
df.select(to_avro(struct([col(c).alias(c) for c in df.columns])).alias("value")) \
.write \
.format("kafka") \
.option("encoding", "utf-8") \
.option("kafka.bootstrap.servers", kafka_ser) \
.option("kafka.sasl.mechanism", "PLAIN") \
.option("kafka.security.protocol", "SEC_SSL") \
.option("kafka.sasl.jaas.config", SSL_CONF)) \
.option("kafka.ssl.endpoint.identification.algorithm", "https") \
.option("topic", kafka_topic) \
.save()
{
"Event": "GlueETLJobExceptionEvent",
"Timestamp": 16343435456875,
"Failure Reason": "Traceback (most recent call last):\n File \"/tmp/kafka_avro_test.py\", line 61, in <module>\n .select(to_avro(struct([col(c).alias(c) for c in df.columns])).alias(\"value\")) \\\n File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/avro/functions.py\", line 117, in to_avro\n jc = sc._jvm.org.apache.spark.sql.avro.functions.to_avro(_to_java_column(data))\nTypeError: 'JavaPackage' object is not callable",
"Stack Trace": [
{
"Declaring Class": "to_avro",
"Method Name": "jc = sc._jvm.org.apache.spark.sql.avro.functions.to_avro(_to_java_column(data))",
"File Name": "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/avro/functions.py",
"Line Number": 117
},
{
"Declaring Class": "<module>",
"Method Name": ".select(to_avro(struct([col(c).alias(c) for c in df.columns])).alias(\"value\")) \\",
"File Name": "/tmp/kafka_avro_test.py",
"Line Number": 61
}],
"Last Executed Line number": 61,"script": "kafka_avro_test.py"}