How to setup durable messages using ActiveMQ and stomp.py - python-3.x

I am trying to setup a durable topic using ActiveMQ. Here is my publisher:
import time
import stomp
conn = stomp.Connection()
conn.connect('admin', 'password', wait=True)
for i in range(1, 21):
message = "message %d"%i
print(message)
conn.send(body=message, destination='/topic/test', persistent=True)
time.sleep(1)
conn.disconnect()
and here is my subscriber:
import time
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, frame):
print('received an error "%s"' % frame.body)
def on_message(self, frame):
print('received a message "%s"' % frame.body)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.connect('admin', 'admin', wait=True, headers = {'client-id': 'testname'})
conn.subscribe(destination='/topic/test', id=1, ack='auto', persistent=True)
Messages are sent and received ok, but if the subscriber is not listening when a message is sent, it is lost, so the topic is not durable. How do I fix this? My understanding is if you have a client-id set and the subscriber id set, messages should be durable in the case of a dropped connection, but this is not working with the current code I have.
EDIT:
Using the accepted answer below here is the working subscriber in case anyone else runs into this issue:
import time
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, frame):
print('received an error "%s"' % frame.body)
def on_message(self, frame):
print('received a message "%s"' % frame.body)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.connect('admin', 'admin', wait=True, headers = {'client-id': 'testname'})
conn.subscribe(destination='/topic/test', id=1, ack='auto', persistent=True, headers = {"activemq.subscriptionName":"testsub"})

To subscribe as a durable topic subscriber to ActiveMQ 5.x brokers the subscriber must include the header "activemq.subscriptionName:" where the unique subscription name is provided by the client each time it reconnects to the broker. The subscription must be created first before messages are sent and only then will it accumulate messages when the subscriber is offline.
The frame should look as follows:
frame = "SUBSCRIBE\n" +
"destination:/topic/myTopic" + "\n" +
"ack:auto\n" + "receipt:1\n" + "id:durablesub-1\n" +
"activemq.subscriptionName:test1\n\n" + "\u0000";
Refer to the documentation for all the frame headers needed when working with ActiveMQ "Classic"
The stomp subscription Id value identifies a unique subscription to the remote so that an application that creates multiple subscriptions to the same address (topic or queue) can manage their subscriptions. Without the Id the remote wouldn't know which subscription an unsubscribe was for etc. Not every subscription is meant to be a durable subscriber (JMS terminology here) e.g. a Queue subscription. The remote needs more context to create a durable subscriber and in this case ActiveMQ 5 requires the "activemq.subscriptionName" value to make that determination. A different messaging system will likely have different property names to accomplish this.

Related

LWT message arrives at the wrong time (broker Mosquitto)

There is a code in which I publish and receive messages from the device. The problem is that if the device suddenly loses connection, the lwt arrives only 5 minutes after the device's connection is broken. Life time 30 sec.
test.py:
import paho.mqtt.client as mqtt
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("test/#")
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
imei = msg.topic.split('test/')[1]
data = msg.payload.decode()
print(imei)
print(data)
publish(client, imei)
def publish(client,imei):
topic = 'test/'+ imei
client.publish(topic,'hello')
print('SEND')
client = mqtt.Client()
user = 'test'
passw = '1111'
client.username_pw_set(user,passw)
client.on_connect = on_connect
client.on_message = on_message
client.enable_bridge_mode()
client.connect("localhost", 1883, 30)
client.loop_forever()
mosquitto.conf:
persistence true
persistence_location /var/lib/mosquitto/
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
include_dir /etc/mosquitto/conf.d
max_keepalive 30
this is how the client send LWT:
modem_setWillMessage(0,0,topic,"no connect")
where first 0-qos,second 0-retained messages
I just can't find the setting that set the lwt time to 60 seconds

Python Paho-Mqtt: What should I pay attention to when subscribing to more topics?

With mqtt subscribe client I am subscribing to lots of threads (over 6000) but not getting results that change on the fly. I'm lagging. Does mqtt give possibility to subscribe too many threads in parallel in background? loop_start would that be enough?
What should I pay attention to when subscribing to more topics?
import logging
import paho.mqtt.client as mqtt
import requests
import zmq
import pandas as pd
PORT=1351
def set_publisher():
context = zmq.Context()
socket_server = context.socket(zmq.PUB)
socket_server.bind(f"tcp://*:{PORT}")
return socket_server
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
#logging.warning(f"Connected with result code :: code : {rc}")
print(f"Connected with result code :: code : {rc}")
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(topics)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
msg = msg.payload
# logging.info(f"message:: {msg}\n")
print(f"message:: {msg}\n")
if msg:
publisher.send(f"{msg}")
def on_disconnect(client, userdata, rc):
if rc != 0:
# logging.warning(f"Unexpected disconnection :: code: {rc}")
print(f"Unexpected disconnection :: code: {rc}")
#todo: if rc is change hostname raise except
client = mqtt.Client(protocol=mqtt.MQTTv31, transport="tcp")
client.username_pw_set(******, password=******)
topics = [(f"topic{i}", 0) for i in 6000]
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect
if client.connect(hostname= *****, port= **** , keepalive=300) != 0:
# logging.info("Could not connect to MQTT Broker !")
print("Could not connect to MQTT Broker !")
client.loop_forever(timeout=3000)
You are describing a situation of compute power (either at the client or the broker or in-between) not sufficient to handle your scenario. It is a common occurrance and that is what performance testing is for: does your setup handle your scenario for your requirements? Capacity planning then expands that question to: ... in the future.

GCP message stays in the Pub/Sub after acknowledge

I have Pub/Sub subscribe logic wrapped inside a subscribe method that is being called once during service initialization for every subscription:
def subscribe(self,
callback: typing.Callable,
subscription_name: str,
topic_name: str,
project_name: str = None) -> typing.Optional[SubscriberClient]:
"""Subscribes to Pub/Sub topic and return subscriber client
:param callback: subscription callback method
:param subscription_name: name of the subscription
:param topic_name: name of the topic
:param project_name: optional project name. Uses default project if not set
:return: subscriber client or None if testing
"""
project = project_name if project_name else self.pubsub_project_id
self.logger.info('Subscribing to project `{}`, topic `{}`'.format(project, topic_name))
project_path = self.pubsub_subscriber.project_path(project)
topic_path = self.pubsub_subscriber.topic_path(project, topic_name)
subscription_path = self.pubsub_subscriber.subscription_path(project, subscription_name)
# check if there is an existing subscription, if not, create it
if subscription_path not in [s.name for s in self.pubsub_subscriber.list_subscriptions(project_path)]:
self.logger.info('Creating new subscription `{}`, topic `{}`'.format(subscription_name, topic_name))
self.pubsub_subscriber.create_subscription(subscription_path, topic_path)
# subscribe to the topic
self.pubsub_subscriber.subscribe(
subscription_path, callback=callback,
scheduler=self.thread_scheduler
)
return self.pubsub_subscriber
This method is called like this:
self.subscribe_client = self.subscribe(
callback=self.pubsub_callback,
subscription_name='subscription_topic',
topic_name='topic'
)
The callback method does a bunch of stuff, sends 2 emails then acknowledges the message
def pubsub_callback(self, data: gcloud_pubsub_subscriber.Message):
self.logger.debug('Processing pub sub message')
try:
self.do_something_with_message(data)
self.logger.debug('Acknowledging the message')
data.ack()
self.logger.debug('Acknowledged')
return
except:
self.logger.warning({
"message": "Failed to process Pub/Sub message",
"request_size": data.size,
"data": data.data
}, exc_info=True)
self.logger.debug('Acknowledging the message 2')
data.ack()
When I run push something to the subscription, callback runs, prints all the debug messages including Acknowledged. The message, however, stays in the Pub/Sub, the callback gets called again and it takes exponential time after each retry. The question is what could cause the message to stay in the pub/sub even after the ack is called?
I have several such subscriptions, all of them work as expected. Deadline is not an option, the callback finishes almost immediately and I played with the ack deadline anyways, nothing helped.
When I try to process these messages from locally running app connected to that pub-sub, it completes just fine and acknowledge takes the message out of the queue as expected.
So the problem manifests only in deployed service (running inside a kubernetes pod)
Callback executes buck ack does seemingly nothing
Acking messages from a script running locally (...and doing the exact same stuff) or through the GCP UI works as expected.
Any ideas?
Acknowledgements are best-effort in Pub/Sub, so it's possible but unusual for messages to be redelivered.
If you are consistently receiving duplicates, it might be due to duplicate publishes of the same message contents. As far as Pub/Sub is concerned, these are different messages and will be assigned different message IDs. Check the Pub/Sub-provided message IDs to ensure that you are actually receiving the same message multiple times.
There is an edge case in dealing with large backlogs of small messages with streaming pull (which is what the Python client library uses). If you are running multiple clients subscribing on the same subscription, this edge case may be relevant.
You can also check your subscription's Stackdriver metrics to see:
if its acks are being sent successfully (subscription/ack_message_count)
if its backlog is decreasing (subscription/backlog_bytes)
if your subscriber is missing the ack deadline (subscription/streaming_pull_ack_message_operation_count filtered by response_code != "success")
If you're not missing the ack deadline and your backlog is remaining steady, you should contact Google Cloud support with your project name, subscription name, and a sample of the duplicate message IDs. They will be able to investigate why these duplicates are happening.
I did some additional testing and I finally found the problem.
TL;DR: I was using the same google.cloud.pubsub_v1.subscriber.scheduler.ThreadScheduler for all subscriptions.
Here are the snippets of the code I used to test it. This is the broken version:
server.py
import concurrent.futures.thread
import os
import time
from google.api_core.exceptions import AlreadyExists
from google.cloud import pubsub_v1
from google.cloud.pubsub_v1.subscriber.scheduler import ThreadScheduler
def create_subscription(project_id, topic_name, subscription_name):
"""Create a new pull subscription on the given topic."""
subscriber = pubsub_v1.SubscriberClient()
topic_path = subscriber.topic_path(project_id, topic_name)
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
subscription = subscriber.create_subscription(
subscription_path, topic_path)
print('Subscription created: {}'.format(subscription))
def receive_messages(project_id, subscription_name, t_scheduler):
"""Receives messages from a pull subscription."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
def callback(message):
print('Received message: {}'.format(message.data))
message.ack()
subscriber.subscribe(subscription_path, callback=callback, scheduler=t_scheduler)
print('Listening for messages on {}'.format(subscription_path))
project_id = os.getenv("PUBSUB_PROJECT_ID")
publisher = pubsub_v1.PublisherClient()
project_path = publisher.project_path(project_id)
# Create both topics
try:
topics = [topic.name.split('/')[-1] for topic in publisher.list_topics(project_path)]
if 'topic_a' not in topics:
publisher.create_topic(publisher.topic_path(project_id, 'topic_a'))
if 'topic_b' not in topics:
publisher.create_topic(publisher.topic_path(project_id, 'topic_b'))
except AlreadyExists:
print('Topics already exists')
# Create subscriptions on both topics
sub_client = pubsub_v1.SubscriberClient()
project_path = sub_client.project_path(project_id)
try:
subs = [sub.name.split('/')[-1] for sub in sub_client.list_subscriptions(project_path)]
if 'topic_a_sub' not in subs:
create_subscription(project_id, 'topic_a', 'topic_a_sub')
if 'topic_b_sub' not in subs:
create_subscription(project_id, 'topic_b', 'topic_b_sub')
except AlreadyExists:
print('Subscriptions already exists')
scheduler = ThreadScheduler(concurrent.futures.thread.ThreadPoolExecutor(10))
receive_messages(project_id, 'topic_a_sub', scheduler)
receive_messages(project_id, 'topic_b_sub', scheduler)
while True:
time.sleep(60)
client.py
import datetime
import os
import random
import sys
from time import sleep
from google.cloud import pubsub_v1
def publish_messages(pid, topic_name):
"""Publishes multiple messages to a Pub/Sub topic."""
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(pid, topic_name)
for n in range(1, 10):
data = '[{} - {}] Message number {}'.format(datetime.datetime.now().isoformat(), topic_name, n)
data = data.encode('utf-8')
publisher.publish(topic_path, data=data)
sleep(random.randint(10, 50) / 10.0)
project_id = os.getenv("PUBSUB_PROJECT_ID")
publish_messages(project_id, sys.argv[1])
I connected to the cloud pub/sub, the server created topics and subscriptions. Then I ran the client script multiple times in parallel for both topics. After a short while, once I changed server code to instantiate new thread scheduler inside receive_messages scope, the server cleaned up both topics and functioned as expected.
Confusing thing is that in either case, the server printed out the received message for all the messages.
I am going to post this to https://github.com/googleapis/google-cloud-python/issues

sending response to particular django websocket client from rest api or a server

consumer.py
# accept websocket connection
def connect(self):
self.accept()
# Receive message from WebSocket
def receive(self, text_data):
text_data_json = json.loads(text_data)
command = text_data_json['command']
job_id = text_data_json['job_id']
if command == 'subscribe':
self.subscribe(job_id)
elif command == 'unsubscribe':
self.unsubscribe(job_id)
else:
self.send({
'error': 'unknown command'
})
# Subscribe the client to a particular 'job_id'
def subscribe(self, job_id):
self.channel_layer.group_add(
'job_{0}'.format(job_id),
self.channel_name
)
# call this method from rest api to get the status of a job
def send_job_notification(self, message, job_id):
channel_layer = get_channel_layer()
group_name = 'job_{0}'.format(job_id)
channel_layer.group_send(
group_name,
{
"type": "send.notification",
"message": message,
}
)
# Receive message from room group
def send_notification(self, event):
message = event['message']
# Send message to WebSocket
self.send(text_data=json.dumps(
message))
In the above code what I am trying to do is connect clients to the socket and subscribe clients to a particular "job_id" by creating a group called "job_1" using "subscribe" method and add it to the channel layer. Creation of groups are dynamic.
I am using below "simple websocket client extension" from Google to connect to the above websocket. I am able to make a connection with the websocket and send request to it as shown in the picture below.
Now since the client is connected and subscribed to a particular "job_id",
I am using "Postman" to send notification to the above connected client(simple websocket client extension) subscribed to particular "job_id" by passing in the job_id in the request as highlighted in yellow below.
when I do a post to the "REST-API" I am calling "send_job_notification(self, message, job_id)" method of "consumer.py" file along with the "job_id" as '1' shown in the picture below in yellow
After doing all this I don't see any message sent to the connected client subscribed to a "job_id" from the "REST-API" call.
Any help would be highly appreciated as it has been dragging on for a long time.
Edit:
thank you for the suggestion Ken its worth to make the method as "#staticmethod" but Ken how do I make the API send job status updates to the connected Clients because my long running jobs would run in some process and send update messages back to the backend via REST-API and the updates then need to be sent to the correct Client (via websockets).
My API call to the socket consumer is as below:
from websocket_consumer import consumers
class websocket_connect(APIView):
def post(self, request, id):
consumers.ChatConsumer.send_job_notification("hello",id)
My socket consumer code is as below:
Edit
`CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("localhost", 6379)],
},
},
}`
As you can see 'Redis' service is also running
Edit-1
You cannot call the method in the consumer directly from an external code because you need to get the particular consumer instance connected to your client. This is the job of the channel layer achieved by using a message passing system or broker as reddis.
From what I can see, you're already going towards the right direction, except that the send_job_notification is an instance method which will require instantiating the consumer. Make it a static method instead, so you can call it directly without a consumer instance
#staticmethod
def send_job_notification(message, job_id):
channel_layer = get_channel_layer()
group_name = 'job_{0}'.format(job_id)
channel_layer.group_send(
group_name,
{
"type": "send.notification",
"message": message,
}
And in your API view, you can simply call it as:
ChatConsumer.send_job_notification(message, job_id)

Testing a MQTT Broker By Connecting 100,000s of Clients

I have been working on MQTT protocol for quite some time. This protocol would be used by the organisation for sending acknowledgment messages to each client separately.
In the use-case, there is only 1 'publisher' client which publishes acknowledgment messages to 'subscriber' clients subscribed to their unique topics
To make sure that the broker is largely bug-free and can scale easily in future, I have been trying to test Emqx and Vernemq open-source brokers by trying to connect at least 50,000 clients to them.
However, I am not able to create those many connections. My Ubuntu 18.04
instance (8 core CPU, 15 GB RAM) on the Google Cloud fails to make any more successful connection after around 300-400.
I have tried making the following changes:
ulimit -n 64500 (for allowing these many file descriptors since every socket connection would require a file descriptor)
Please help me in making over 50,000 connections. Should I run 'n' number of threads and run the total_clients/total_threads clients under loop on each thread?
or should I create one thread for every client connection?
What should I do?
The following message appear on the "$SYS/#" topic once the clients start getting disconnected even though there is no disconnect data packet being sent by the client side.
$SYS/brokers/emqx#127.0.0.1/clients/112/disconnected {"clientid":"112","username":"undefined","reason":"closed","ts":1536587647}
import paho.mqtt.client as mqtt
from threading import Lock
import time
print_lock = Lock()
def s_print(str):
print_lock.acquire()
print(str)
print_lock.release()
def on_connect(client, userdata, flags, rc):
client_id = userdata["client_id"]
if (rc == 0):
s_print("connected " + client_id)
client.subscribe(client_id, 2)
def on_disconnect(client, userdata, rc):
client_id = userdata["client_id"]
s_print("disconnected: " + client_id + " reason " + str(rc))
def on_message(client, userdata, message):
topic = message.topic
payload = str(message.payload)
s_print("Recieved " + payload + " on " + topic)
if __name__ == '__main__':
n_clients = int(input("Enter no. of clients: "))
for i in range(n_clients):
client_id = str(i)
s_print(client_id)
userdata = {
"client_id" : client_id
}
client = mqtt.Client(client_id=client_id, clean_session=True, userdata=userdata)
client.on_connect = on_connect
client.on_disconnect =on_disconnect
client.on_message = on_message
client.connect("35.228.57.228", 1883, 60)
client.loop_start()
time.sleep(0.5)
while(1):
time.sleep(5)

Resources