I have a basic MQTTListener class in Python which listens for messages on certain topics and should start or stop an async process imported from another script. This process runs forever, unless it is manually stopped. Let's assume the Listener looks like this:
import paho.mqtt.client as mqtt
import json
from python_file.py import async_forever_function
class MqttListener:
def __init__(self, host, port, client_id):
self.host = host
self.port = port
self.client_id = client_id
self.client = mqtt.Client(client_id=self.client_id)
self.client.connect(host=self.host, port=self.port)
def on_connect(self, client, userdata, flags, rc):
self.client.subscribe(topic=[("start", 1), ])
self.client.subscribe(topic=[("stop", 1), ])
logging.info(msg="MQTT - connected!")
def on_disconnect(client, userdata, rc):
logging.info(msg="MQTT - disconnected!")
def on_message(self, client, userdata, message, ):
print('PROCESSING MESSAGE', message.topic, message.payload.decode('utf-8'), )
if message.topic == 'start':
async_forever_function(param='start')
print('process started')
else:
async_forever_function(param='stop')
print('process removed')
def start(self):
self.client.on_connect = lambda client, userdata, flags, rc: self.on_connect(client, userdata, flags, rc)
self.client.on_message = lambda client, userdata, message: self.on_message(client, userdata, message)
self.client.on_disconnect = lambda client, userdata, rc: self.on_disconnect(client, userdata, rc)
self.client.loop_start()
def stop(self):
self.client.loop_stop()
Now, this works for starting a new async process. That is, async_function is correctly triggered when a message is posted on the start MQTT topic. However, once this async process is started, the listener is no longer able to receive/process messages from the stop MQTT topic and the async process will continue to run forever, when in fact it should have been stopped.
My question: how can I adapt the code of this class such that it can also process messages when an active async process is running in the background?
You can not do blocking tasks in the on_message() callback.
This call back runs on the MQTT client thread (the one started by the loop_start() function. This thread handles all network traffic and message handling, if you block it then it can't do any of that.
If you want to call long running tasks from the on_message() callback you need to start a new thread for the long running task so it doesn't block the MQTT client loop.
Related
I have a main App that initializes fastAPI and uses fastAPI MQTT. Now, this file is running in an event loop and is looking for regular MQTT port.
I have a second app that uses MQTT Websocket and is running in an event loop. Now, I want the two apps to exchange information with each other. How can I make them interchange the information.One is using websocket MQTT and another one is using fast API MQTT. What is the way to make them talk to each other. Here is the
main.py - which runs fastapi and mqtt - regular one.
from fastapi import FastAPI
from fastapi_mqtt import FastMQTT, MQTTConfig
from pydantic import BaseModel
from ipaddress import IPv4Address
import jsonpickle
app = FastAPI()
class Nmap(BaseModel):
host: IPv4Address
portRange: str
class Config:
schema_extra = {
"example" : {
"host": "10.0.2.15",
"portRange": "22-80",
"description": "Scan the port from 22 to 80 of the ip address 10.0.2.15"
}
}
## for docker - compose using mqtt - broker name from docker compose.
## for normal leave that blank to fall off to localhost.
mqtt_config = MQTTConfig(host = "mqtt")
mqtt = FastMQTT(config=mqtt_config)
mqtt.init_app(app)
#mqtt.on_connect()
def connect(client, flags, rc, properties):
mqtt.client.subscribe("/mqtt/toModel/#") # subscribing mqtt topic wildcard- multi-level
print("connected: ", client, flags, rc, properties)
#mqtt.on_message()
async def message(client, topic, payload, qos, properties):
print("received message: ", topic, jsonpickle.decode(payload.decode()), qos, properties)
return 0
#mqtt.on_disconnect()
def disconnect(client, packet, exc=None):
print("Disconnected")
#mqtt.on_subscribe()
def subscribe(client, mid, qos, properties):
print("subscribed", client, mid, qos, properties)
#app.get("/")
async def func():
mqtt.client.publish("/mqtt", "Hello from fastApi")
return {"result": True, "message": "Published"}
#app.post("/scan/{host}")
async def scan_host_port(nmap_details : Nmap):
results = {"got_val" : nmap_details}
print(type(nmap_details))
mqtt.client.publish("/mqtt/fromModel/nmap", jsonpickle.encode(nmap_details))
return results
and here is the websocket based one mqtt.
import paho.mqtt.client as paho
import time
broker="10.0.2.15"
#port= 80
#port=1883
port= 9001
sub_topic="ws/pythonApp"
def on_subscribe(client, userdata, mid, granted_qos): #create function for callback
print("subscribed with qos",granted_qos, "\n")
pass
def on_message(client, userdata, message):
print("message received " ,str(message.payload.decode("utf-8")))
def on_publish(client,userdata,mid): #create function for callback
print("data published mid=",mid, "\n")
pass
def on_disconnect(client, userdata, rc):
print("client disconnected ok")
client= paho.Client("client-socks",transport='websockets') #create client object
#client= paho.Client("control1")
client.on_subscribe = on_subscribe #assign function to callback
client.on_publish = on_publish #assign function to callback
client.on_message = on_message #assign function to callback
client.on_disconnect = on_disconnect
print("connecting to broker ",broker,"on port ",port)
client.connect(broker,port) #establish connection
client.loop_start()
print("subscribing to ",sub_topic)
client.subscribe(sub_topic)
time.sleep(3)
client.publish("ws/jsclient","on") #publish
time.sleep(4)
client.disconnect()
You don't have to do anything, the MQTT broker will bridge both input protocols (native MQTT & MQTT over WebSockets) into the same topic space.
Anything published will be published to any subscribers no matter if they are connected via native MQTT or MQTT over WebSockets.
Do i need to subscribe to all topics i am interested in before i execute the loop_forever() function? E.g. to somehow dynamically add more subscriptions during the life time of the MQTT client.
I found out that it is possible. I implemented a class which hides the mqtt stuff. In the constructor i connect to the broker and start the loop_forever() with a seperate thread. And afterwards that class subscribes to some topics and register a callback for each topic, which are called from on_message.
c = comm("localhost")
c.register_handler("topic1", first_handler)
c.register_handler("topic2", second_handler)
class comm:
def __init__(self, broker_address):
self.client = mqtt.Client("")
self.client.on_message = self.on_message
self.callbacks = dict()
self.client.connect(broker_address)
threading.Thread(target=self.__run, daemon=True).start()
def __run(self):
self.client.loop_forever()
def on_message(self, client, userdata, msg):
self.callbacks[msg.topic](jsonpickle.decode(msg.payload))
def on_connect(self, client, userdata, flags, rc):
for topic in self.callbacks:
self.client.subscribe(topic)
def register_handler(self, topic: str, handler: Callable[[Dict], None]):
self.callbacks[topic] = handler
self.client.subscribe(topic)
With mqtt subscribe client I am subscribing to lots of threads (over 6000) but not getting results that change on the fly. I'm lagging. Does mqtt give possibility to subscribe too many threads in parallel in background? loop_start would that be enough?
What should I pay attention to when subscribing to more topics?
import logging
import paho.mqtt.client as mqtt
import requests
import zmq
import pandas as pd
PORT=1351
def set_publisher():
context = zmq.Context()
socket_server = context.socket(zmq.PUB)
socket_server.bind(f"tcp://*:{PORT}")
return socket_server
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
#logging.warning(f"Connected with result code :: code : {rc}")
print(f"Connected with result code :: code : {rc}")
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(topics)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
msg = msg.payload
# logging.info(f"message:: {msg}\n")
print(f"message:: {msg}\n")
if msg:
publisher.send(f"{msg}")
def on_disconnect(client, userdata, rc):
if rc != 0:
# logging.warning(f"Unexpected disconnection :: code: {rc}")
print(f"Unexpected disconnection :: code: {rc}")
#todo: if rc is change hostname raise except
client = mqtt.Client(protocol=mqtt.MQTTv31, transport="tcp")
client.username_pw_set(******, password=******)
topics = [(f"topic{i}", 0) for i in 6000]
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect
if client.connect(hostname= *****, port= **** , keepalive=300) != 0:
# logging.info("Could not connect to MQTT Broker !")
print("Could not connect to MQTT Broker !")
client.loop_forever(timeout=3000)
You are describing a situation of compute power (either at the client or the broker or in-between) not sufficient to handle your scenario. It is a common occurrance and that is what performance testing is for: does your setup handle your scenario for your requirements? Capacity planning then expands that question to: ... in the future.
I am writing a django application and I need a way how to ask for data on MQTT and wait for response (or timeout).
Something like this in view:
mqtt.subscribe('response/topic')
mqtt.publish('request/topic', '{my json request message}')
for m in mqtt.on_message():
if m.payload == '......'
# OK I have response, render data from response to page, ...
# timeout could be implemented by raising exception in on_message()
How can i integrate paho mqtt? Or is there any other way? I need to wait for response on several places in views, one global on_message callback is not really useful.
Where should I put mqtt.loop_forever() in django?
Thank you
EDIT: my first solution is this:
import queue
import paho.mqtt.client as mqtt
def mqtt_request(req_topic, resp_topic, payload, timeout=2.0):
q = queue.Queue()
def on_message(client, userdata, message):
q.put(message)
def on_connect(client, userdata, flags, rc):
c.subscribe(resp_topic)
def on_subscribe(client, userdata, mid, granted_qos):
c.publish(req_topic, payload=payload, qos=0, retain=False)
c = mqtt.Client()
c.on_message = on_message
c.on_connect = on_connect
c.on_subscribe = on_subscribe
c.connect_async('127.0.0.1')
c.loop_start()
m = None
# FIXME: it seems that there is some delay between on_message and return
# of the message...
try:
m = q.get(timeout=timeout)
except queue.Empty:
pass
c.loop_stop()
return m
usage in view:
r = mqtt_request('test', 'test2', 'Ahoj')
if r:
print(r.payload)
I am using Ubuntu mate on the Pine A64+ 1Gb, I have installed paho mqtt library with python3, I tested library by creating local mosquito server and it is working fine. Now I need to connect to external broker having username and password, I tried with following code but it didn't worked for me. With this I am not even able to connect to the broker.
import paho.mqtt.client as mqtt
import time
broker_address = "121.242.232.175.xip.io"
port = 1883
def on_connect(client, userdata, flags, rc):
if rc==0:
client.connected_flag = True
print("connected OK Returned code=",rc)
else:
print("Bad connection Returned code=",rc)
mqtt.Client.connected_flag = False
client = mqtt.Client("SWAHVACAHU00000600")
client.username_pw_set(username="#####",password="#####")
client.on_connect = on_connect
client.loop_start()
client.connect(broker_address, port)
#while not client.connected_flag:
# print("inthe while")
# time.sleep(1)
client.loop_stop()
client.publish("pine", "Hello from Pinr A64",0)
client.disconnect()
I am checking on hivemqtt using as websocket client and subscribing to the same topic.
Check again what loop_start() does:
These functions implement a threaded interface to the network loop. Calling loop_start() once, before or after connect*(), runs a thread in the background to call loop() automatically. This frees up the main thread for other work that may be blocking. This call also handles reconnecting to the broker. Call loop_stop() to stop the background thread
paho-mqtt
That means that you start a thread which frequently handles all your networking actions (also sending your connection attempt). In your code you immediateley stop this thread again by calling loop_stop() - so there is a high chance that your connection attempt wasn't even send out.
In addition your main program terminates right after client.disconnect() without any delay - so the networking thread (if running) has absoluetly no time to deal anything at all
I recommend to restructure your code so your actions are properly timed and connection is closed after all work is done:
def on_connect(client, userdata, flags, rc):
if rc==0:
print("Connected.")
client.publish("mytopic/example", "")
else:
print("Connection refused, rc=",rc)
def on_disconnect(client, userdata, rc):
print ("Disconnected")
if rc != 0:
# otherwise loop_forever won't return
client.disconnect()
def on_publish(client, userdataq, mid):
print ("Message delivered - closing down connection")
client.disconnect()
print ("Program started")
client = mqtt.Client("MyClient")
client.username_pw_set(username=user,password=pw)
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_publish = on_publish
client.connect(broker_address, port)
client.loop_forever()
print("Program finished")
The blocking loop loop_forever() automatically returns if disconnect() is called. When using loop_start() / loop_stop() you need a loop by your own in order to prevent your program from terminating and you also have to handle when to break the loop and when to close the networking thread.
Also consider putting client.connect() within a try...except block