I have just started using pubnub. I entered the basic code which was given in pubnub python sdk (4.0) and I get the following errors
ERROR:pubnub:Async request Exception. 'Publish' object has no
attribute 'async' ERROR:pubnub:Exception in subscribe loop: 'Publish'
object has no attribute 'async' WARNING:pubnub:reconnection policy is
disabled, please handle reconnection manually.
As far as the async() is concerned, there is a troubleshoot in which the async error can be solved be entering the following
def callback(result, status):
if status.is_error():
print("Error %s" % str(status.error_data.exception))
print("Error category #%d" % status.category)
else:
print(str(result))\
but still it doesn't work.
This is the code
from pubnub.callbacks import SubscribeCallback
from pubnub.enums import PNStatusCategory
from pubnub.pnconfiguration import PNConfiguration
from pubnub.pubnub import PubNub
pnconfig = PNConfiguration()
pnconfig.subscribe_key = 'demo'
pnconfig.publish_key = 'demo'
pubnub = PubNub(pnconfig)
def my_publish_callback(envelope, status):
# Check whether request successfully completed or not
if not status.is_error():
pass # Message successfully published to specified channel.
else:
pass # Handle message publish error. Check 'category' property to find out possible issue
# because of which request did fail.
# Request can be resent using: [status retry];
class MySubscribeCallback(SubscribeCallback):
def presence(self, pubnub, presence):
pass # handle incoming presence data
def status(self, pubnub, status):
if status.category == PNStatusCategory.PNUnexpectedDisconnectCategory:
pass # This event happens when radio / connectivity is lost
elif status.category == PNStatusCategory.PNConnectedCategory:
# Connect event. You can do stuff like publish, and know you'll get it.
# Or just use the connected event to confirm you are subscribed for
# UI / internal notifications, etc
pubnub.publish().channel("awesomeChannel").message("hello!!").async(my_publish_callback)
elif status.category == PNStatusCategory.PNReconnectedCategory:
pass
# Happens as part of our regular operation. This event happens when
# radio / connectivity is lost, then regained.
elif status.category == PNStatusCategory.PNDecryptionErrorCategory:
pass
# Handle message decryption error. Probably client configured to
# encrypt messages and on live data feed it received plain text.
def message(self, pubnub, message):
pass # Handle new message stored in message.message
pubnub.add_listener(MySubscribeCallback())
pubnub.subscribe().channels('awesomeChannel').execute()
As the error is from the publish method, it could most probably be because
async has been changed to pn_async
Note that as on date, this is applicable only for Python3 as the same has not been implemented for Python 2.
Change
pubnub.publish().channel("awesomeChannel").message("hello!!").async(my_publish_callback)
to
pubnub.publish().channel("awesomeChannel").message("hello!!").pn_async(my_publish_callback)
Reference document here
Related
We have a QLDB ingestion process that consists of a Lambda function triggered by SQS.
We want to make sure our pipeline is airtight so if a failure or error occurs during driver execution, we don't lose that data if the data fails to commit to QLDB.
In our testing we noticed that if there's a failure within the Lambda itself, it automatically resends the message to the queue, but if the driver fails, the data is lost.
I understand that the default behavior for the driver is to retry four times after the initial failure. My question is, if I wrap qldb_driver.execute_lambda() in a try statement, will that allow the driver to retry upon failure or will it instantly return as a failure and be handled by the except statement?
Here is how I've written the first half of the function:
import json
import boto3
import datetime
from pyqldb.driver.qldb_driver import QldbDriver
from utils import upsert, resend_to_sqs, delete_from_sqs
queue_url = 'https://sqs.XXX/'
sqs = boto3.client('sqs', region_name='us-east-1')
ledger = 'XXXXX'
table = 'XXXXX'
qldb_driver = QldbDriver(ledger_name = ledger, region_name='us-east-1')
def lambda_handler(event, context):
# Simple iterable to identify messages
i = 0
# Error flag
error = False
# Empty list to store message send status as well as body or receipt_handle
batch_messages = []
for record in event['Records']:
payload = json.loads(record["body"])
payload['update_ts'] = str(datetime.datetime.now())
try:
qldb_driver.execute_lambda(lambda executor: upsert(executor, ledger = ledger, table_name = table, data = payload))
# If the message sends successfully, give it status 200 and add the recipt_handle to our list
# so in case an error occurs later, we can delete this message from the queue.
message_info = {f'message_{i}': 200, 'receiptHandle': record['receiptHandle']}
batch_messages.append(message_info)
except Exception as e:
print(e)
# Flip error flag to True
error = True
# If the commit fails, set status 400 and add the message's body to our list.
# This will allow us to send the message back to the queue during error handling.
message_info = {f'message_{i}': 400, 'body': record['body']}
batch_messages.append(message_info)
i += 1
Assuming that this try/except allows the driver to retry upon failure, I've written an additional process to record message data from our batch to delete successful commits and send failures back to the queue:
# Begin error handling
if error:
count = 0
for j in range(len(batch_messages)):
# If a message was sent successfully delete it from the queue
if batch_messages[j][f'message_{j}'] == 200:
receipt_handle = batch_messages[j]['receiptHandle']
delete_from_sqs(sqs, queue_url, receipt_handle)
# If the message failed to commit to QLDB, send it back to the queue
else:
body = batch_messages[j]['body']
resend_to_sqs(sqs, queue_url, body)
count += 1
print(f"ERROR(S) DETECTED - {count} MESSAGES RETURNED TO QUEUE")
else:
print("BATCH PROCESSING SUCCESSFUL")
Thank you for your insight!
The qldb python driver can be configured for more or less retries if you need. I'm not sure if you wanted it to only try 1 time, or if you were asking that the driver will try the transaction 4 times before triggering the try/catch exception. The driver will still try up-to 4 times, before throwing the except.
You can follow the example here to modify the retry amount. Also, note the default retry timeout is a random ms jitter and not exponential. With QLDB, you shouldn't need to wait long periods to retry since it uses optimistic concurrency control.
Also, with your design of throwing the failed message back into the queue...you might want to consider throwing it into a dead letter queue. Dead-letter queues would prevent trouble messages from retrying indefinitely, unless thats your goal.
(edit/additionally)
Observe that the qldb driver exhausting retires before raising an exception.
for my backend micro-service application i am using RabbitMq as the message broker.
For the existing services i am getting events well. My question is how to pull all the old events into a new micro-service which would launch in future.
Just consider if there are three services currently
Product
Notification
Order
if i created a new product it information will be broadcasted to Notification service as well as Order service. So a record of product will be there in both Notification and Order.
so, after a while(when around 500+ products where added), if i had created new service called Analytic’s and wanted all the product created events to be listened when it is initially up.
I am using Python, RabbitMQ & Pika library.
this is my sample code
Sample Publisher code
import pika
import sys, random
connection = pika.BlockingConnection(pika.URLParameters('<rabbitmq-link>'))
channel = connection.channel()
channel.exchange_declare(exchange='group', exchange_type='fanout')
message = "info: Hello World!"
channel.basic_publish(exchange='group', routing_key='', body=message)
print(" [x] Sent %r" % message)
connection.close()
Service one code
import pika
connection = pika.BlockingConnection(pika.URLParameters('<rabbitmq-link>'))
channel = connection.channel()
channel.exchange_declare(exchange='group', exchange_type='fanout')
result = channel.queue_declare(queue='group-1', exclusive=False)
queue_name = result.method.queue
channel.queue_bind(exchange='group', queue=queue_name)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r" % body)
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
channel.start_consuming()
Service two code
import pika
connection = pika.BlockingConnection(pika.URLParameters('<rabbitmq-link>'))
channel = connection.channel()
channel.exchange_declare(exchange='group', exchange_type='fanout')
result = channel.queue_declare(queue='group-2', exclusive=False)
queue_name = result.method.queue
channel.queue_bind(exchange='group', queue=queue_name)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r" % body)
channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
channel.start_consuming()
So, i want a way where when i made the third service live, it should be able to triggered by old events
I am testing IBM MQ using JMeter and able to establish the connection with queue to send requests over it. However, I need to set "JMS_IBM_LAST_MSG_IN_GROUP" property as true for one of the message but unable to do so. I am using below piece of code while sending request or trying to set the property to true but it remains set to it's default value i.e. false when I am checking in backend. Any clue what I am missing here.
Note: Connection is being established in another sampler and making use of that connection here. This code is working fine to send any request, just that property is not getting set to true.
import java.time.Instant
import com.ibm.msg.client.jms.JmsConstants
def sess = System.getProperties().get("Session")
def destination = System.getProperties().get("Destination")
def producer = sess.createProducer(destination)
def rnd = new Random(System.currentTimeMillis())
def payload = String.format('${groupid}|${sequencenumber}|rest of the payload|')
def msg = sess.createTextMessage(payload)
println('Payload --> ' + payload)
msg.setBooleanProperty(JmsConstants.JMS_IBM_LAST_MSG_IN_GROUP,true)
def start = Instant.now()
producer.send(msg)
def stop = Instant.now()
producer.close()
SampleResult.setResponseData(msg.toString())
SampleResult.setDataType( org.apache.jmeter.samplers.SampleResult.TEXT)
SampleResult.setLatency( stop.toEpochMilli() - start.toEpochMilli())
Your code does not include anything to set the Group ID or sequence number. I assume we have all the relevant code shown, in which case, I think you are missing code something along these lines:
msg.setStringProperty("JMSXGroupID", groupid);
msg.setIntProperty("JMSXGroupSeq", sequencenumber);
As per JMS_IBM_LAST_MSG_IN_GROUP chapter of IBMMQ documentation
This property is ignored in the publish/subscribe domain and is not relevant when an application connects to a service integration bus.
In general it's not necessary to use this property, you can come up with your own custom one, i.e. for the producer:
msg.setBooleanProperty("X_CUSTOM_PROPERTY_LAST_MESSAGE",true)
and for the consumer:
msg.getBooleanProperty("X_CUSTOM_PROPERTY_LAST_MESSAGE")
More information: IBM MQ testing with JMeter - Learn How
Sharing as it might help someone else. Was able to set the property to true by making below changes, rest of the code is same as mentioned in original question
import com.ibm.msg.client.wmq.WMQConstants
def gid=String.format('${groupid}')
def msg = sess.createTextMessage()
println('Payload --> ' + payload)
msg.setStringProperty(WMQConstants.JMSX_GROUPID,gid)
msg.setBooleanProperty(WMQConstants.JMS_IBM_LAST_MSG_IN_GROUP,true)
msg.text=payload
I am trying to develop a Python 3.6 script which uses pika and threading modules.
I have a problem which I think is caused by my A) being very new to Python and coding in general, and B) my not understanding how to pass variables between functions when they are run in separate threads and already being passed a parameter in parentheses at the end of the receiving function name.
The reason I think this, is because when I do not use threading, I can pass a variable between functions simply by calling the receiving function name, and supplying the variable to be passed, in parentheses, a basic example is shown below:
def send_variable():
body = "this is a text string"
receive_variable(body)
def receive_variable(body):
print(body)
This when run, prints:
this is a text string
A working version of the code I need to to get working with threading is shown below - this uses straight functions (no threading) and I am using pika to receive messages from a (RabbitMQ) queue via the pika callback function, I then pass the body of the message received in the 'callback' function to the 'processing function' :
import pika
...mq connection variables set here...
# defines username and password credentials as variables set at the top of this script
credentials = pika.PlainCredentials(mq_user_name, mq_pass_word)
# defines mq server host, port and user credentials and creates a connection
connection = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, port=mq_port, credentials=credentials))
# creates a channel connection instance using the above settings
channel = connection.channel()
# defines the queue name to be used with the above channel connection instance
channel.queue_declare(queue=mq_queue)
def callback(ch, method, properties, body):
# passes (body) to processing function
body_processing(body)
# sets channel consume type, also sets queue name/message acknowledge settings based on variables set at top of script
channel.basic_consume(callback, queue=mq_queue, no_ack=mq_no_ack)
# tells the callback function to start consuming
channel.start_consuming()
# calls the callback function to start receiving messages from mq server
callback()
# above deals with pika connection and the main callback function
def body_processing(body):
...code to send a pika message every time a 'body' message is received...
This works fine however I want to translate this to run within a script that uses threading. When I do this I have to supply the parameter 'channel' to the function name that runs in its own thread - when I then try to include the 'body' parameter so that the 'processing_function' looks as per the below:
def processing_function(channel, body):
I get an error saying:
[function_name] is missing 1 positional argument: 'body'
I know that when using threading there is more code needed and I have included the actual code that I use for threading below so that you can see what I am doing:
...imports and mq variables and pika connection details are set here...
def get_heartbeats(channel):
channel.queue_declare(queue=queue1)
#print (' [*] Waiting for messages. To exit press CTRL+C')
def callback(ch, method, properties, body):
process_body(body)
#print (" Received %s" % (body))
channel.basic_consume(callback, queue=queue1, no_ack=no_ack)
channel.start_consuming()
def process_body(channel, body):
channel.queue_declare(queue=queue2)
#print (' [*] Waiting for Tick messages. To exit press CTRL+C')
# sets the mq host which pika client will use to send a message to
connection = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host))
# create a channel connection instance
channel = connection.channel()
# declare a queue to be used by the channel connection instance
channel.queue_declare(queue=order_send_queue)
# send a message via the above channel connection settings
channel.basic_publish(exchange='', routing_key=send_queue, body='Test Message')
# send a message via the above channel settings
# close the channel connection instance
connection.close()
def manager():
# Channel 1 Connection Details - =======================================================================================
credentials = pika.PlainCredentials(mq_user_name, mq_password)
connection1 = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, credentials=credentials))
channel1 = connection1.channel()
# Channel 1 thread =====================================================================================================
t1 = threading.Thread(target=get_heartbeats, args=(channel1,))
t1.daemon = True
threads.append(t1)
# as this is thread 1 call to start threading is made at start threading section
# Channel 2 Connection Details - =======================================================================================
credentials = pika.PlainCredentials(mq_user_name, mq_password)
connection2 = pika.BlockingConnection(pika.ConnectionParameters(host=mq_host, credentials=credentials))
channel2 = connection2.channel()
# Channel 2 thread ====================================================================================================
t2 = threading.Thread(target=process_body, args=(channel2, body))
t2.daemon = True
threads.append(t2)
t2.start() # as this is thread 2 - we need to start the thread here
# Start threading
t1.start() # start the first thread - other threads will self start as they call t1.start() in their code block
for t in threads: # for all the threads defined
t.join() # join defined threads
manager() # run the manager module which starts threads that call each module
This when run produces the error
process_body() missing 1 required positional argument: (body)
and I do not understand why this is or how to fix it.
Thank you for taking the time to read this question and any help or advice you can supply is much appreciated.
Please keep in mind that I am new to python and coding so may need things spelled out rather than being able to understand more cryptic replies.
Thanks!
On further looking in to this and playing with the code it seems that if I edit the lines:
def process_body(channel, body):
to read
def process_body(body):
and
t2 = threading.Thread(target=process_body, args=(channel2, body))
so that it reads:
t2 = threading.Thread(target=process_body)
then the code seems to work as needed - I also see multiple script processes in htop so it appears that threading is working - I have left the script processing for 24 hours + and did not receive any errors...
I have an mqtt client app that subscribes to topics based on a configuration file. Something like:
def connectMQTT():
global Connection
Connection = Client()
Connection.on_message = handleQuery
for clientid in clientids.allIDs(): # clientids.allIDs() reads files to get this
topic = '{}/{}/Q/+'.format(Basename, clientid)
print('subscription:', topic)
Connection.subscribe(topic)
I have been using it with a simple invocation like:
def main():
connectMQTT()
Connection.loop_forever()
The loop_forever will block forever. But I'd like to notice when the information read by clientids.allIDs() is out of date and I should reconnect forcing it to subscribe afresh.
I can detect a change in the files with pyinotify:
def filesChanged():
# NOT SURE WHAT TO DO HERE
def watchForChanges():
watchManager = pyinotify.WatchManager()
notifier = pyinotify.ThreadedNotifier(watchManager, FileEventHandler(eventCallback))
notifier.start()
watchManager.add_watch('/etc/my/config/dir', pyinotify.IN_CLOSE_WRITE | pyinotify.IN_DELETE)
Basically, I need loop_forever (or some other paho mqtt mechanism) to run until some signal comes from the pyinotify machinery. I'm not sure how to weld those two together though. In pseudo code, I thing I want something like
def main():
signal = setup_directory_change_signal()
while True:
connectMQTT()
Connection.loop(until=signal)
Connection.disconnect()
I'm not sure how to effect that though.
I finally circled around to the following solution which seems to work. Whereas I was trying to run the notifier in another thread and the mqtt loop in the main thread, the trick seemed to be invert that setup:
def restartMQTT():
if Connection:
Connection.loop_stop()
connectMQTT()
Connection.loop_start()
class FileEventHandler(pyinotify.ProcessEvent):
def process_IN_CREATE(self, fileEvent):
restartMQTT()
def process_IN_DELETE(self, fileEvent):
restartMQTT()
def main():
restartMQTT()
watchManager = pyinotify.WatchManager()
notifier = pyinotify.Notifier(watchManager, FileEventHandler())
watchManager.add_watch('/etc/my/config_directory', pyinotify.IN_CREATE | pyinotify.IN_DELETE)
notifier.loop()
Where connectMQTT stores a newly connected and configured MQTT client in the Connection global.