I was installing kafka in Quickstart Cloudera VM using following link
but when i am running below command
kafka-topics --zookeeper quickstart.cloudera:2181 --create --topic test --partitions 1 --replication-factor 1
I am getting following error
19/09/21 11:28:36 INFO zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/10.0.2.15:2181, sessionid = 0x16d54d21037009d, negotiated timeout = 30000
19/09/21 11:28:38 INFO zookeeper.ZooKeeperClient: [ZooKeeperClient] Connected.
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
19/09/21 11:28:40 ERROR admin.TopicCommand$: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
I tried to resolve this issue from here. but when running below command
bin/zookeeper-server-start.sh config/zookeeper.properties
but getting error.
19/09/21 11:54:42 ERROR quorum.QuorumPeerMain: Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:131)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:106)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Caused by: java.lang.IllegalArgumentException: config/zookeeper.properties file is missing
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:115)
... 2 more
Invalid config, exiting abnormally
Any lead is appreciated.
Actually I want to connect kafka to spark. If there is another way to do it. that is also fine.
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
Your first error message says it all. It seems like you start only the zookeeper process, but not the broker. The error says that no brokers in the cluster.
Make sure that you start a Kafka broker with important broker configuration, make sure the zookeeper is the one you started!.
Related
We have 10 kafka machines with kafka version - 1.X
this kafka cluster version is part of HDP version - 2.6.5
We noticed that under /var/log/kafka/server.log the following message
ERROR Error while accepting connection {kafka.network.Accetpr}
java.io.IOException: Too many open files
We saw also additionally
Broker 21 stopped fetcher for partition ...................... because they are in the failed log dir /kafka/kafka-logs {kafka.server.ReplicaManager}
and
WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. this implies messages have arrived out of order. New: {epoch:0, offset:2227488}, Currnet: {epoch 2, offset:261} for Partition: cars-list-75 {kafka.server.epochLeaderEpocHFileCache}
so regarding to the issue -
ERROR Error while accepting connection {kafka.network.Accetpr}
java.io.IOException: Too many open files
how to increase the MAX open files , in order to avoid this issue
update:
in ambari we saw the following parameter from kafka --> config
is this is the parameter that we should to increase?
It can be done like this:
echo "* hard nofile 100000
* soft nofile 100000" | sudo tee --append /etc/security/limits.conf
Then you should reboot.
We have a 15 node cassandra setup across 3 DC's.
Using cassandra 3.0.9
One of the nodes in one of our DCs has died with the below startup error
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF 111111-111111111-11111111
What we have tried -
Bootstrap replacing the node -https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsReplaceNode.html
This works for a while and then the bootstrap process hangs with the SAME error in the logs -
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF
Provision a new blank node,
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html
and try to join this to the cluster (with the intention of removing the dead node after). It refuses to start, with the same error.
has anyone ever ran across this before?
I'm trying to send spark job to yarn (without HDFS) in HA mode.
For submitting I'm using org.apache.spark.deploy.SparkSubmit.
When I send request from machine with active Resource Manager, it works well. But if I' trying to send from machine with standby Resource Manager, job fails with error:
DEBUG org.apache.hadoop.ipc.Client - Connecting to spark2-node-dev/10.10.10.167:8032
DEBUG org.apache.hadoop.ipc.Client - Connecting to /0.0.0.0:8032
org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep
However, when I send request via command line (spark-submit), it works well through both active and standby machine.
What can cause the problem?
P.S. Use the same parameters for both type of sending job: org.apache.spark.deploy.SparkSubmit and spark-submit command line request. And properties yarn.resourcemanager.hostname.rm_id defined for all rm hosts
The problem was with absence of yarn-site.xml within class path for spark-submitter jar. Actually spark submitter jar does not take to account YARN_CONF_DIR or HADOOP_CONF_DIR env var, so cannot see yarn-site.
One solution that I found was to put yarn-site into classpath of jar.
I'm using Spark Standalone Mode tutorial page to install Spark in Standalone mode.
1- I have started a master by:
./sbin/start-master.sh
2- I have started a worker by:
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu:7077
Note: spark://ubuntu:7077 is my master name, which I can see it in Master-WebUI.
Problem: By second command, a worker started successfully. But it couldn't associate with master. It tries repeatedly and then give this message:
15/02/08 11:30:04 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster#ubuntu:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: ubuntu/127.0.1.1:7077
15/02/08 11:30:04 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#-1296628173] to Actor[akka://sparkWorker/deadLetters] was not delivered. [20] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
15/02/08 11:31:15 ERROR Worker: All masters are unresponsive! Giving up.
What is the problem?
Thanks
I usually start from spark-env.sh template. And I set, properties that I need. For simple cluster you need:
SPARK_MASTER_IP
Then, create a file called "slaves" in the same directory as spark-env.sh and slaves ip's (one per line). Assure you reach all slaves through ssh.
Finally, copy this configuration in every machine of your cluster. Then start the entire cluster executing start-all.sh script and try spark-shell to check your configuration.
> sbin/start-all.sh
> bin/spark-shell
You can set export SPARK_LOCAL_IP="You-IP" #to set the IP address Spark binds to on this node in $SPARK_HOME/conf/spark-env.sh
In my case, using spark 2.4.7 in standalone mode, I've created a passwordless ssh key using ssh-keygen, but still got asked for worker password when starting the cluster.
What I did was follow the instructions here
https://www.cyberciti.biz/faq/how-to-set-up-ssh-keys-on-linux-unix/
This line solved the problem:
ssh-copy-id -i $HOME/.ssh/id_rsa.pub user#server-ip
I have a cluster running on datastax-cassandra 1.2.5, it works fine, because of vnodes adn leveled compaction strategy issue i tried promoting it to 1.2.6.
So upgrading involved -
1 - stopping all the nodes
2 - deleting 1.2.5 rpm
3 - installing 1.2.6 rpm
4 - fixing cassandra.yaml
5 - starting cassandra.
Problem Statement - The problem now is that all the nodes are up and running, but not in one cluster. They all are operating in their own cluster even though the seeds in yaml points to the original seed.
nodetool status also just shows the one node (the node on which we are on)
system log shows one error
ERROR [WRITE-/10.93.3.46] 2013-10-21 19:43:29,101 CassandraDaemon.java (line 192)
Exception in thread Thread[WRITE-/10.10.10.10,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:351)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
**** 10.10.10.10 is the seed ip
Any help on how to pass through it
Try to set the internode_compression to none. It will disable compression between nodes, which is failing because snappy cannot initialize
internode_compression: none