I'm new to Kafka, using confluent kafka and trying to write messages to existing kafka topic from AWS EC2 instance using python producer code with 'sasl.mechanism': 'PLAIN','security.protocol': 'SASL_SSL'.I tried the producer example from this link. Getting following error.If any can guide me what would leads to this issue would be great help.
Failed to deliver message: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}
Related
I am gettign below error while running a Glue Streaming job which fails to connect to Kinesis data source :
Error:
WARNING:root:StreamingQueryException caught. Retry number 10 ERROR:root:Exceeded maximuim number of retries in streaming interval,
exception thrown Parse yarn logs get error message:
StreamingQueryException: 'Error while Describe Streams\n=== Streaming Query ===\nIdentifier: [id = 60exxxxxxxxxxxxx
Following are the set of jars i used :
spark-tags_2.11-2.4.0.jar,
spark-streaming-kinesis-asl_2.11-2.4.0.jar,
spark-streaming_2.11-2.4.0.jar,
aws-java-sdk-sts-1.11.271.jar,
amazon-kinesis-client-1.8.10.jar,
spark-sql_2.11-2.4.0.jar
#####################################################################
spark-tags_2.11-2.4.3.jar,
spark-streaming-kinesis-asl_2.11-2.4.3.jar,
aws-java-sdk-sts-1.11.271.jar,
jackson-dataformat-cbor-2.6.7.jar,
unused-1.0.0.jar,
spark-sql_2.11-2.4.3.jar
##########################################
spark-sql-kinesis_2.11-1.1.3-spark_2.4.jar,
spark-tags_2.11-2.4.0.jar,
unused-1.0.0.jar,
scala-library-2.11.12.jar,
spark-sql_2.11-2.4.0.jar
Please suggest, since there is very less and vague information on Glue Streaming and Kinesis integration.
Glue Streaming with Kinesis as a source uses a version of qubole/kinesis-sql
The Samples on that Github Repo should be a good starting point. Also this blog by qubole.
Kinesis ASL (spark-streaming-kinesis-asl) uses older spark streaming APIs, InputDStreams etc. Glue streaming has in-built support for spark structured streaming APIs and so if using Glue Streaming with Kinesis, need not import all those dependencies. If the application has to be Kinesis ASL based, then one might try building a Jar with all these dependencies and running them as part of a normal Glue ETL Job itself instead of a Glue Streaming Job. Porting the application to use the spark structured APIs would be much easier.
Also, there is an example of using the Structured Streaming APIs with Glue Streaming on the docs - https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-example.html
Hi am getting this error when am trying to connect to kafka remotely on my prod server.My messages are not getting produced and/or consumed from my code.Let me know if any code sample is needed.Just want to know what can be the reasons for receiving this error.
Just in case some one else go through the same issue: the issue for me was i was using it with kerberos the principal name and the keytab file that I was using didn't have permissions to create topic or produce/consume message!
Not able to commit the consumer using confluent kafka. Facing an error "Segmentation fault", while commiting the consumer.
defined consumer with 'enable.auto.commit' = False; default.topic.config {auto.offset.reset:'earliest'}
polling the messages with timeout = 50
creating a TopicPartition object using msg.topic(), msg.partition(), msg.offset()
then commiting the consumer
consumer.commit(message=msg.value(),offsets=Topic_partition object, async=True)
I'm trying to create a application-level error handler for failures during the processing inside SCS application using Kafka as a message broker. I know that SCS already provides the DLQ functionality, but in my case I want to wrap failed messages with a custom wrapper type (providing the failure context (source, cause etc.))
In https://github.com/qabbasi/Spring-Cloud-Stream-DLQ-Error-Handling you can see two approaches for this scenario: one is using SCS and the other one directly Spring Integration. (both are atm not working)
According to the current reference (https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#_producing_and_consuming_messages) SCS will allow to publish error messages received from the Spring Integration error channel, but unfortunately this not the case, at least for me. Although the application logs the following upon startup
o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 2 subscriber(s).
You shouldn't use #StreamListener("errorChannel") - that is consuming from a binder destination; to capture messages sent to the errorChannel use #ServiceActivator(inputChannel = "errorChannel").
EDIT
There were several problems with your app...
The new error handling code was added in version 1.3
The autoCommitOnError is a kafka binder property
You needed an #EnableBinding(CustomDlqMessageChannel.class)
You don't really need #EnableIntegration - boot does that for you
See my commit here.
and...
$ kafka-console-producer --broker-list localhost:9092 --topic testIn
>foo
and...
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic customDlqTopic --from-beginning
?
contentType>"application/x-java-object;type=com.example.demo.ErrorWrapper"
I'm trying to setup Spark Streaming to get messages from Kafka queue. I'm getting the following error:
py4j.protocol.Py4JJavaError: An error occurred while calling o30.createDirectStream.
: org.apache.spark.SparkException: java.nio.channels.ClosedChannelException
org.apache.spark.SparkException: Couldn't find leader offsets for Set([test-topic,0])
at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)
at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$checkErrors$1.apply(KafkaCluster.scala:366)
at scala.util.Either.fold(Either.scala:97)
Here is the code I'm executing (pyspark):
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test-topic"], {"metadata.broker.list": "host.domain:9092"})
ssc.start()
ssc.awaitTermination()
There were a couple of similar posts with the same error. In all cases the cause was the empty kafka topic. There are messages in my "test-topic". I can get them out with
kafka-console-consumer --zookeeper host.domain:2181 --topic test-topic --from-beginning --max-messages 100
Does anyone know what might be the problem?
I'm using:
Spark 1.5.2 (apache)
Kafka 0.8.2.0+kafka1.3.0 (CDH 5.4.7)
You need to check 2 things:
check if this topic and partition exists , in your case is topic is test-topic and partition is 0.
based on your code, you are trying consume message from offset 0 and it might be possible message is not available from offset 0, check what is you earliest offset and try consume from there.
Below is command to check earliest offset:
sh kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list "your broker list" --topic "topic name" --time -1
1) You have to make sure that you have already created topic test-topic
Run following command to check list of the topic
kafka-topics.sh --list --zookeeper [host or ip of zookeeper]:[port]
2) After checking your topic, you have to configure your Kafka configuration in Socket Server Settings section
listeners=PLAINTEXT://[host or ip of Kafka]:[port]
If you define short host names in /etc/hosts and use them in your kafka servers' configurations, you should change those name to ip. Or register the same short host name in your local PC or client's /etc/hosts.
Error occurred because Spark streaming lib can't resolve short hostname in the PC or client.
Another option to force creating topic if it doesn't exist. You can do this by setting property "auto.create.topics.enable" to "true" in kafkaParams map like this.
val kafkaParams = Map[String, String](
"bootstrap.servers" -> kafkaHost,
"group.id" -> kafkaGroup,
"auto.create.topics.enable" -> "true")
Using Scala 2.11 and Kafka 0.10 versions.
One of the reason for this type of error where leader cannot be found for specified topic is Problem with one's Kafka server configs.
Open your Kafka server configs :
vim ./kafka/kafka-<your-version>/config/server.properties
In the "Socket Server Settings" section , provide IP for your host if its missing :
listeners=PLAINTEXT://{host-ip}:{host-port}
I was using Kafka setup provided with MapR sandbox and was trying to access the kafka via spark code. I was getting the same error while accessing my kafka since my configuration was missing the IP.