I have been working on MQTT protocol for quite some time. This protocol would be used by the organisation for sending acknowledgment messages to each client separately.
In the use-case, there is only 1 'publisher' client which publishes acknowledgment messages to 'subscriber' clients subscribed to their unique topics
To make sure that the broker is largely bug-free and can scale easily in future, I have been trying to test Emqx and Vernemq open-source brokers by trying to connect at least 50,000 clients to them.
However, I am not able to create those many connections. My Ubuntu 18.04
instance (8 core CPU, 15 GB RAM) on the Google Cloud fails to make any more successful connection after around 300-400.
I have tried making the following changes:
ulimit -n 64500 (for allowing these many file descriptors since every socket connection would require a file descriptor)
Please help me in making over 50,000 connections. Should I run 'n' number of threads and run the total_clients/total_threads clients under loop on each thread?
or should I create one thread for every client connection?
What should I do?
The following message appear on the "$SYS/#" topic once the clients start getting disconnected even though there is no disconnect data packet being sent by the client side.
$SYS/brokers/emqx#127.0.0.1/clients/112/disconnected {"clientid":"112","username":"undefined","reason":"closed","ts":1536587647}
import paho.mqtt.client as mqtt
from threading import Lock
import time
print_lock = Lock()
def s_print(str):
print_lock.acquire()
print(str)
print_lock.release()
def on_connect(client, userdata, flags, rc):
client_id = userdata["client_id"]
if (rc == 0):
s_print("connected " + client_id)
client.subscribe(client_id, 2)
def on_disconnect(client, userdata, rc):
client_id = userdata["client_id"]
s_print("disconnected: " + client_id + " reason " + str(rc))
def on_message(client, userdata, message):
topic = message.topic
payload = str(message.payload)
s_print("Recieved " + payload + " on " + topic)
if __name__ == '__main__':
n_clients = int(input("Enter no. of clients: "))
for i in range(n_clients):
client_id = str(i)
s_print(client_id)
userdata = {
"client_id" : client_id
}
client = mqtt.Client(client_id=client_id, clean_session=True, userdata=userdata)
client.on_connect = on_connect
client.on_disconnect =on_disconnect
client.on_message = on_message
client.connect("35.228.57.228", 1883, 60)
client.loop_start()
time.sleep(0.5)
while(1):
time.sleep(5)
Related
I am trying to setup a durable topic using ActiveMQ. Here is my publisher:
import time
import stomp
conn = stomp.Connection()
conn.connect('admin', 'password', wait=True)
for i in range(1, 21):
message = "message %d"%i
print(message)
conn.send(body=message, destination='/topic/test', persistent=True)
time.sleep(1)
conn.disconnect()
and here is my subscriber:
import time
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, frame):
print('received an error "%s"' % frame.body)
def on_message(self, frame):
print('received a message "%s"' % frame.body)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.connect('admin', 'admin', wait=True, headers = {'client-id': 'testname'})
conn.subscribe(destination='/topic/test', id=1, ack='auto', persistent=True)
Messages are sent and received ok, but if the subscriber is not listening when a message is sent, it is lost, so the topic is not durable. How do I fix this? My understanding is if you have a client-id set and the subscriber id set, messages should be durable in the case of a dropped connection, but this is not working with the current code I have.
EDIT:
Using the accepted answer below here is the working subscriber in case anyone else runs into this issue:
import time
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, frame):
print('received an error "%s"' % frame.body)
def on_message(self, frame):
print('received a message "%s"' % frame.body)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.connect('admin', 'admin', wait=True, headers = {'client-id': 'testname'})
conn.subscribe(destination='/topic/test', id=1, ack='auto', persistent=True, headers = {"activemq.subscriptionName":"testsub"})
To subscribe as a durable topic subscriber to ActiveMQ 5.x brokers the subscriber must include the header "activemq.subscriptionName:" where the unique subscription name is provided by the client each time it reconnects to the broker. The subscription must be created first before messages are sent and only then will it accumulate messages when the subscriber is offline.
The frame should look as follows:
frame = "SUBSCRIBE\n" +
"destination:/topic/myTopic" + "\n" +
"ack:auto\n" + "receipt:1\n" + "id:durablesub-1\n" +
"activemq.subscriptionName:test1\n\n" + "\u0000";
Refer to the documentation for all the frame headers needed when working with ActiveMQ "Classic"
The stomp subscription Id value identifies a unique subscription to the remote so that an application that creates multiple subscriptions to the same address (topic or queue) can manage their subscriptions. Without the Id the remote wouldn't know which subscription an unsubscribe was for etc. Not every subscription is meant to be a durable subscriber (JMS terminology here) e.g. a Queue subscription. The remote needs more context to create a durable subscriber and in this case ActiveMQ 5 requires the "activemq.subscriptionName" value to make that determination. A different messaging system will likely have different property names to accomplish this.
There is a code in which I publish and receive messages from the device. The problem is that if the device suddenly loses connection, the lwt arrives only 5 minutes after the device's connection is broken. Life time 30 sec.
test.py:
import paho.mqtt.client as mqtt
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("test/#")
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
imei = msg.topic.split('test/')[1]
data = msg.payload.decode()
print(imei)
print(data)
publish(client, imei)
def publish(client,imei):
topic = 'test/'+ imei
client.publish(topic,'hello')
print('SEND')
client = mqtt.Client()
user = 'test'
passw = '1111'
client.username_pw_set(user,passw)
client.on_connect = on_connect
client.on_message = on_message
client.enable_bridge_mode()
client.connect("localhost", 1883, 30)
client.loop_forever()
mosquitto.conf:
persistence true
persistence_location /var/lib/mosquitto/
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
include_dir /etc/mosquitto/conf.d
max_keepalive 30
this is how the client send LWT:
modem_setWillMessage(0,0,topic,"no connect")
where first 0-qos,second 0-retained messages
I just can't find the setting that set the lwt time to 60 seconds
With mqtt subscribe client I am subscribing to lots of threads (over 6000) but not getting results that change on the fly. I'm lagging. Does mqtt give possibility to subscribe too many threads in parallel in background? loop_start would that be enough?
What should I pay attention to when subscribing to more topics?
import logging
import paho.mqtt.client as mqtt
import requests
import zmq
import pandas as pd
PORT=1351
def set_publisher():
context = zmq.Context()
socket_server = context.socket(zmq.PUB)
socket_server.bind(f"tcp://*:{PORT}")
return socket_server
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
#logging.warning(f"Connected with result code :: code : {rc}")
print(f"Connected with result code :: code : {rc}")
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(topics)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
msg = msg.payload
# logging.info(f"message:: {msg}\n")
print(f"message:: {msg}\n")
if msg:
publisher.send(f"{msg}")
def on_disconnect(client, userdata, rc):
if rc != 0:
# logging.warning(f"Unexpected disconnection :: code: {rc}")
print(f"Unexpected disconnection :: code: {rc}")
#todo: if rc is change hostname raise except
client = mqtt.Client(protocol=mqtt.MQTTv31, transport="tcp")
client.username_pw_set(******, password=******)
topics = [(f"topic{i}", 0) for i in 6000]
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect
if client.connect(hostname= *****, port= **** , keepalive=300) != 0:
# logging.info("Could not connect to MQTT Broker !")
print("Could not connect to MQTT Broker !")
client.loop_forever(timeout=3000)
You are describing a situation of compute power (either at the client or the broker or in-between) not sufficient to handle your scenario. It is a common occurrance and that is what performance testing is for: does your setup handle your scenario for your requirements? Capacity planning then expands that question to: ... in the future.
I am using Ubuntu mate on the Pine A64+ 1Gb, I have installed paho mqtt library with python3, I tested library by creating local mosquito server and it is working fine. Now I need to connect to external broker having username and password, I tried with following code but it didn't worked for me. With this I am not even able to connect to the broker.
import paho.mqtt.client as mqtt
import time
broker_address = "121.242.232.175.xip.io"
port = 1883
def on_connect(client, userdata, flags, rc):
if rc==0:
client.connected_flag = True
print("connected OK Returned code=",rc)
else:
print("Bad connection Returned code=",rc)
mqtt.Client.connected_flag = False
client = mqtt.Client("SWAHVACAHU00000600")
client.username_pw_set(username="#####",password="#####")
client.on_connect = on_connect
client.loop_start()
client.connect(broker_address, port)
#while not client.connected_flag:
# print("inthe while")
# time.sleep(1)
client.loop_stop()
client.publish("pine", "Hello from Pinr A64",0)
client.disconnect()
I am checking on hivemqtt using as websocket client and subscribing to the same topic.
Check again what loop_start() does:
These functions implement a threaded interface to the network loop. Calling loop_start() once, before or after connect*(), runs a thread in the background to call loop() automatically. This frees up the main thread for other work that may be blocking. This call also handles reconnecting to the broker. Call loop_stop() to stop the background thread
paho-mqtt
That means that you start a thread which frequently handles all your networking actions (also sending your connection attempt). In your code you immediateley stop this thread again by calling loop_stop() - so there is a high chance that your connection attempt wasn't even send out.
In addition your main program terminates right after client.disconnect() without any delay - so the networking thread (if running) has absoluetly no time to deal anything at all
I recommend to restructure your code so your actions are properly timed and connection is closed after all work is done:
def on_connect(client, userdata, flags, rc):
if rc==0:
print("Connected.")
client.publish("mytopic/example", "")
else:
print("Connection refused, rc=",rc)
def on_disconnect(client, userdata, rc):
print ("Disconnected")
if rc != 0:
# otherwise loop_forever won't return
client.disconnect()
def on_publish(client, userdataq, mid):
print ("Message delivered - closing down connection")
client.disconnect()
print ("Program started")
client = mqtt.Client("MyClient")
client.username_pw_set(username=user,password=pw)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_publish = on_publish
client.connect(broker_address, port)
client.loop_forever()
print("Program finished")
The blocking loop loop_forever() automatically returns if disconnect() is called. When using loop_start() / loop_stop() you need a loop by your own in order to prevent your program from terminating and you also have to handle when to break the loop and when to close the networking thread.
Also consider putting client.connect() within a try...except block
I am trying to write a simple chat server, that takes message from 1 user and sends to all other users. The other users can simultaneously send messages and receive them.
On client side one thread continuously waits for messages and other thread sends messages.
import socket
import threading
def getMessage(s):
while True:
s.send(raw_input("Message: "))
#Main
port = 1041
host = 'localhost'
s = socket.socket()
s.connect((host, port))
background_thread = threading.Thread(target=getMessage, args=(s,))
background_thread.daemon = True
background_thread.start()
while True:
print s.recv(1024)
On server side the server, takes the incoming connection request, opens a new thread for each new request and waits for their messages, when a connection sends a message, the server broadcasts it to all other connections that are open.
import socket
from thread import *
def ClientThread(connection, clients):
while True:
message = connection.recv(1024)
connection.send("Ack\n")
broadcast(connection, clients, message)
def broadcast(connection, clients, message):
for conn in clients:
if(conn != connection):
conn.send(message)
def AcceptConnections(clients):
while True:
connection, address = s.accept()
print "Got connection from ",connection
clients.append(connection)
start_new_thread(ClientThread, (connection, clients))
#Main
print "Started server"
port = 1041
host = 'localhost'
s = socket.socket()
s.bind((host, port))
s.listen(5)
clients = []
AcceptConnections(clients)
Expected: When one client sends a message, all other connected clients receive that message, irrespective of them typing and sending a message.
Reality: Other clients receive the message only after they send 1 or 2 messages.