From Node.js application I am trying to connect to Apache Kafka broker using node-rdkafka client .Since the kafka broker lists are SSL enabled hence configuring the node-rdkafka producer with ssl options as below :
I have already tried with different valid certificates and keys tried adding CA also using the option ssl.ca.certificate:<CA-location>
but still no luck.
Searched in librdkafka github page and found one similar issue where is the proposed solution was to use api.verison.request:false , tried this also but no luck Still getting the same error
Error: broker transport failure
Tried using another kafka client named no-kafka with the same ssl certificate and keys to connect to the same broker lists and able to establish the connection .
We have to use node-rdkafka only.
The producer configuration using node-rdkafka :
var producer = new Kafka.Producer({
'debug':'All',
'metadata.broker.list': 'comma separated list of ssl enabled broker hosts and port',
'dr_cb': true,
'security.protocol': 'ssl',
'ssl.certificate.location': path.join(__dirname, 'server.crt'),
'ssl.key.location': path.join(__dirname, 'server.key'),
'ssl.ca.location' : path.join(__dirname,'DigiCertSHA2SecureServerCA-int.cer'),
});
I expect a SUCCESS MESSAGE saying connection set up, but the actual result is Error : Broker transport Failure
Related
I have a basic Pulsar app, and when I try to connect to Pulsar, I get this exception:
2021-03-10 14:38:26.107 WARN 7 --- [r-client-io-1-1]
o.a.pulsar.client.impl.ConnectionPool : Failed to open connection
to my-pulsar-server-ms-tls.domain.com:6651 :
io.netty.channel.ConnectTimeoutException: connection timed out:
my-pulsar-server-ms-tls.domain.com/10.80.13.38:6651 2021-03-10
14:38:26.212 WARN 7 --- [al-listener-3-1]
o.a.pulsar.client.impl.PulsarClientImpl : [topic:
persistent://myTenant/myNamespace/myTopic]
Could not get connection while getPartitionedTopicMetadata -- Will try
again in 100 ms
My Pulsar client is pretty basic:
PulsarClient.builder()
.serviceUrl(serviceUrl)
.authentication(AuthenticationFactory.token(authToken))
.tlsTrustCertsFilePath(serverCertificateFilePath.toString())
.enableTlsHostnameVerification(false)
.allowTlsInsecureConnection(false)
.build();
The producer is also pretty basic and looks like this:
pulsarClient.newProducer(Schema.STRING)
.topic(topic)
.create();
I've verified that the token and TLS cert are correct. I've also tried connecting a consumer from this same environment and got a similar exception, and I know that others with the same code are able to connect to the same Pulsar cluster from other environments. What is the issue?
Your connection is getting blocked by a firewall or network issue.
Verify that you can establish a connection to your endpoint my-pulsar-server-ms-tls.domain.com:6651 from your environment.
If you're able to run a network packet dump (like tcpdump), that should make it obvious if you're not able to establish a connection.
You can also try running curl my-pulsar-server-ms-tls.domain.com:6651, and if you get back some html, that means you were able to reach the server. However, if you get Could not resolve host, then you were blocked by the network configuration (such as a missing route) or firewall.
Regarding Kafka-Zookeeper Security using DIGEST MD5 Authentication, I am trying to rotate/change credentials/password for both server(zookeeper) and client(kafka) jaas config file.
We have a 3 node cluster of 3 zookeepers and 3 kafka broker nodes with below jaas configuration file.
kafka.conf
org.apache.zookeeper.server.auth.DigestLoginModule required
username="super"
password="password";
};
zookeeper.conf
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_super="password";
};
To rotate we do a rolling restart of server(zookeeper) instances after updating the credential(password) and during the process of rolling restart after updating the same credential/password for super user for client(kafka instances) one at a time, we notice
[2019-06-15 17:17:38,929] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-06-15 17:17:38,929] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
these info level in server logs, which eventually results in unclean shutdown and restart of the broker which impacts the writes and reads for longer than expected. I have tried commenting requireClientAuthScheme=sasl in zookeeper zoo.cfg https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication to allow any clients authenticate to zookeeper but no success.
Also, alternative approach - tried to update the credential/password in jaas config file dynamically using sasl.jaas.config and do get the same exception documented in this jira (reference: https://issues.apache.org/jira/browse/KAFKA-8010).
can someone have any suggestions? Thanks in advance.
I'm writing a script to get some sensor-data using MQTT, transform the incoming value is a specific way and forward it to a cloud which uses an encrypted communication.
The script runs on RaspberryPi 3 (Raspbian) where Mosquitto runs as MQTT-broker ( The sensors send the data to this broker)
So far I get the information from the the sensors and I can transform the content like desired.
The problem is when I try to connect to the remote broker, I got following error-message
Error: error:0906D06C:PEM routines:PEM_read_bio:no start line
at Error (native)
at Object.createSecureContext (_tls_common.js:67:17)
at Object.TLSSocket._init.ssl.onclienthello.ssl.oncertcb.exports.connect (_tls_wrap.js:1015:46)
at Object.buildBuilder (/home/pi/node_modules/mqtt/lib/connect/tls.js:13:20)
at MqttClient.wrapper [as streamBuilder] (/home/pi/node_modules/mqtt/lib/connect/index.js:135:36)
at MqttClient._setupStream (/home/pi/node_modules/mqtt/lib/client.js:246:22)
at new MqttClient (/home/pi/node_modules/mqtt/lib/client.js:227:8)
at Object.connect (/home/pi/node_modules/mqtt/lib/connect/index.js:138:10)
at Object.<anonymous> (/home/pi/GATT_server/MQTT_module.js:3:25)
I have tried to configure Mosquitto as MQTT-bridge to use the config-file and it looks like it'd work (There are no error messages when the service is restarted).
The problem when I use Mosquitto as a bridge the script can't subscribe to the topics published by the sensors.
I'm not familiar with those certificates but from reading the doc of the file 'client-options.cs' I wrote following lines:
var mqtt = require('mqtt')
var client = mqtt.connect('mqtt://localhost', 'port:1883');
var remoteBroker = mqtt.connect('mqtts://xxx.xxx.io',
{port:8883,
protocol:'ssl',
username:'username',
password:'123password123',
cert:'/etc/mosquitto/certs/ca.pem'} );
/*....
.... many lines later
....*/
remoteBroker.publish(topicToUseOnRemoteMqttBroker, PayloadToForward);
Am I doing it right? Or did I miss something?
If the certificate is OK for the Mosquitto-Bridge it should be OK for the remote broker, too. - Is that right?
If you have additional links/papers about this topic - I like to learn ;P
Google gave me so many information - I don't know where to start :(
The entry for the cert should not be the path to the certificate file, but the actual cert it's self. You need to read the file in and pass that.
Somethings like this:
var remoteBroker = mqtt.connect('mqtts://xxx.xxx.io',
{port:8883,
protocol: 'ssl',
username: 'username',
password: '123password123',
cert: fs.readFileSync('/etc/mosquitto/certs/ca.pem')} );
I'm running node-rdkafka as a Node.js application. The consumer hangs indefinitely without pulling any messages from kafka (works on localhost).
Emits the below error,
{ Error: Local: Host resolution failure
origin: 'local',
message: 'host resolution failure',
code: -1,
errno: -1,
stack: 'Error: Local: Host resolution failure' }
The application works to the point of receiving data from kafka. The kafka instance is fine, validated by producing and consuming messages using the console.
Any help with debugging why this is occurring is much appreciated.
Sample consumer code here - https://github.com/Blizzard/node-rdkafka/blob/master/examples/consumer-flow.md
This issue happens due to the different networks of your client and broker.
The simple hack is to make host entry of advertised.listeners
For example,
advertised.listeners=PLAINTEXT://kafka:9092
Then add an entry in /etc/hosts with your kafka-broker-IP. For e.g. kafka-borker-IP is 192.168.1.1
192.168.1.1 kafka
You can use kafkacat utility to check your broker's IP.
kafkacat -b kafka:9092 -L
It will return metadata about the brokers.
You need to check that returned broker's IP is reachable or not from your machine.
For a better understanding of this issue.
You can refer https://www.confluent.io/blog/kafka-listeners-explained/
I had this exact problem when running kafka locally using the quick start instructions from https://kafka.apache.org/quickstart
For me, adding the following two lines to config/server.properties before starting kafka-server has solved the issue -
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
I'm trying some exercise with spark streaming with kafka. If I use kafka producer and consumer in command line, I can publish and consume the messages in kafka. When I try to do it using pyspark in jupyter notebook. I am getting zookeeper connection timeout error.
Client session timed out, have not heard from server in 6004ms for sessionid 0x0, closing socket connection and attempting reconnect
[2017-08-04 15:49:37,494] INFO Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#158da8e (org.apache.zookeeper.ZooKeeper)
[2017-08-04 15:49:37,524] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2017-08-04 15:49:37,527] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-08-04 15:49:37,533] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2017-08-04 15:49:38,637] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-08-04 15:49:38,639] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
`
Zookeeper has issues when using localhost (127.0.0.1). Described in https://issues.apache.org/jira/browse/ZOOKEEPER-1661?focusedCommentId=13599352
This little program explains the following things:
ZooKeeper does call InetAddress.getAllByName (see StaticHostProvider:60) on the connect string "localhost:2181" => as a result it gets 3 different addresses for localhost which then get shuffled (Collections.shuffle(this.serverAddresses): L72
Because of the shuffling (random), the call to StaticHostProvider.next will sometime return the fe80:0:0:0:0:0:0:1%1 address which as you can see from this small program times out after 5s => this explains the randomness I am experiencing.
It really seems to me that what I am experiencing is a reverse dns lookup issue with IPv6. Whether this reverse dns lookup is actually useful and required by ZooKeeper, I do not know. It did not behave this way in 3.3.3.
Solution, specify your zookeeper address as a FQDN and make sure the reverse lookup works or use 0.0.0.0 instead of localhost.