Hello i am playing around with RabbitMq and i have a question. I have many parallel nodes running and are connected on rabbitmq-server. The thing is that in conncetions are stated like ip:port Also under ip:port there is a ?. Is there a way to set up a friendly name for python 3.6 ?
P.S I have tried something like this without success
parameters = pika.ConnectionParameters(
host='localhost',
virtual_host='/',
client_properties={
'connection_name': 'random name',
},
)
connection = pika.BlockingConnection(parameters)
P.S Also i found a related question about golang tho. Here How to set connection friendly name
Versions
pika = '1.1.0'
RabbitMq = '3.8.0'
Erlang = '22.1.1'
Python = ' 3.6.8'
Any ideas? :)
This code sets the connection name as you can see here:
import functools
import logging
import pika
LOG_FORMAT = ('%(levelname) -10s %(asctime)s %(name) -30s %(funcName) '
'-35s %(lineno) -5d: %(message)s')
LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG, format=LOG_FORMAT)
def on_message(chan, method_frame, _header_frame, body, userdata=None):
"""Called when a message is received. Log message and ack it."""
LOGGER.info('Userdata: %s Message body: %s', userdata, body)
chan.basic_ack(delivery_tag=method_frame.delivery_tag)
def main():
"""Main method."""
credentials = pika.PlainCredentials('guest', 'guest')
props = { 'connection_name' : 'https://stackoverflow.com/q/58275505/1466825' }
parameters = pika.ConnectionParameters('localhost', credentials=credentials, client_properties=props)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.exchange_declare(
exchange='test_exchange',
exchange_type='direct',
passive=False,
durable=True,
auto_delete=False)
channel.queue_declare(queue='standard', auto_delete=True)
channel.queue_bind(
queue='standard', exchange='test_exchange', routing_key='standard_key')
channel.basic_qos(prefetch_count=1)
on_message_callback = functools.partial(
on_message, userdata='on_message_userdata')
channel.basic_consume('standard', on_message_callback)
try:
channel.start_consuming()
except KeyboardInterrupt:
channel.stop_consuming()
connection.close()
if __name__ == '__main__':
main()
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
I'm trying to connect a wildlife camera to my SMTP server but it keeps dropping the connection after being asked for it's username. I've verified that this server works with other wildlife cameras and email clients but always seems to fail with this specific model of wildlife camera. I've tried with no authentication, basic authentication and TLS but none of them work (The camera works with gmail SMTP though).
This is the simple code I'm using.
It seems like I need to modify the challenge_auth method. My question is how do I do that, do I just add another method to the custom handler with handle_DATA in?
import email
from email.header import decode_header
from email import message_from_bytes
from email.policy import default
from aiosmtpd.controller import Controller
from aiosmtpd.smtp import LoginPassword, AuthResult
import os
import sys
import time
import signal
import logging
##setting timezone
os.environ['TZ'] = "Europe/London"
time.tzset()
def onExit( sig, func=None):
print("*************Stopping program*****************")
controller.stop()
exit()
signal.signal(signal.SIGTERM, onExit)
# removes the spaces and replaces with _ so they're valid folder names
def clean(text):
return "".join(c if c.isalnum() else "_" for c in text)
log = logging.getLogger('mail.log')
auth_db = {
b"TestCamera1#gmail.com": b"password1",
b"user2": b"password2",
b"TestCamera1": b"password1",
}
def authenticator_func(server, session, envelope, mechanism, auth_data):
#this deliberately lets everything through
assert isinstance(auth_data, LoginPassword)
username = auth_data.login
password = auth_data.password
return AuthResult(success=True)
def configure_logging():
file_handler = logging.FileHandler("aiosmtpd.log", "a")
stderr_handler = logging.StreamHandler(sys.stderr)
logger = logging.getLogger("mail.log")
fmt = "[%(asctime)s %(levelname)s] %(message)s"
datefmt = None
formatter = logging.Formatter(fmt, datefmt, "%")
stderr_handler.setFormatter(formatter)
logger.addHandler(stderr_handler)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
logger.setLevel(logging.DEBUG)
class CustomHandler:
def handle_exception(self, error):
print("exception occured")
print(error)
return '542 Internal Server Error'
async def handle_DATA(self, server, session, envelope):
peer = session.peer
data = envelope.content # type: bytes
msg = message_from_bytes(envelope.content, policy=default)
# decode the email subject
print("Msg:{}".format(msg))
print("Data:{}".format(data))
print("All of the relevant data has been extracted from the email")
return '250 OK'
if __name__ == '__main__':
configure_logging()
handler = CustomHandler()
#update hostname to your IP
controller = Controller(handler, hostname='0.0.0.0', port=587, authenticator=authenticator_func, auth_required=True,auth_require_tls=False)
# Run the event loop in a separate thread.
controller.start()
while True:
time.sleep(10)
Here's the logs from a reolink go camera that can connect successfully. (I've updated the format 'Username' is being send .e.g from 'User Name:' to 'Username' by editing the library but that hasn't seemed to help with the suntek camera. I thought it might be more pick with the format due to cheaper, less robust firmware.
I'm trying to implement an async RPC client within a Flask server.
The idea is that each request spawn a thread with an uuid, and each request is going to wait until there is a response in the RpcClient queue attribute object with the correct uuid.
The problem is that one request out of two fails. I think that might be a problem with multi-threading, but I don't see where it comes from.
Bug can be seen here.
Using debug print, it seems that the message with the correct uuid is received in the _on_response callback and update the queue attribute in this instance correctly, but the queue attribute within the /rpc_call/<payload> endpoint doesn't synchronize (so queue[uuid] has a value of response in the RpcClient callback but still None in the scope of the endpoint).
My code:
from flask import Flask, jsonif
from gevent.pywsgi import WSGIServer
import sys
import os
import pika
import uuid
import time
import threading
class RpcClient(object):
"""Asynchronous Rpc client."""
internal_lock = threading.Lock()
queue = {}
def __init__(self):
self.connection = pika.BlockingConnection(
pika.ConnectionParameters(host='rabbitmq'))
self.channel = self.connection.channel()
self.channel.basic_qos(prefetch_count=1)
self.channel.exchange_declare(exchange='kaldi_expe', exchange_type='topic')
# Create all the queue and bind them to the corresponding routing key
self.channel.queue_declare('request', durable=True)
result = self.channel.queue_declare('answer', durable=True)
self.channel.queue_bind(exchange='kaldi_expe', queue='request', routing_key='kaldi_expe.web.request')
self.channel.queue_bind(exchange='kaldi_expe', queue='answer', routing_key='kaldi_expe.kaldi.answer')
self.callback_queue = result.method.queue
.
thread = threading.Thread(target=self._process_data_events)
thread.setDaemon(True)
thread.start()
def _process_data_events(self):
self.channel.basic_consume(self.callback_queue, self._on_response, auto_ack=True)
while True:
with self.internal_lock:
self.connection.process_data_events()
time.sleep(0.1)
def _on_response(self, ch, method, props, body):
"""On response we simply store the result in a local dictionary."""
self.queue[props.correlation_id] = body
def send_request(self, payload):
corr_id = str(uuid.uuid4())
self.queue[corr_id] = None
with self.internal_lock:
self.channel.basic_publish(exchange='kaldi_expe',
routing_key="kaldi_expe.web.request",
properties=pika.BasicProperties(
reply_to=self.callback_queue,
correlation_id=corr_id,
),
body=payload)
return corr_id
def flask_app():
app = Flask("kaldi")
#app.route('/', methods=['GET'])
def server_is_up():
return 'server is up', 200
#app.route('/rpc_call/<payload>')
def rpc_call(payload):
"""Simple Flask implementation for making asynchronous Rpc calls. """
corr_id = app.config['RPCclient'].send_request(payload)
while app.config['RPCclient'].queue[corr_id] is None:
#print("queue server: " + str(app.config['RPCclient'].queue))
time.sleep(0.1)
return app.config['RPCclient'].queue[corr_id]
if __name__ == '__main__':
while True:
try:
rpcClient = RpcClient()
app = flask_app()
app.config['RPCclient'] = rpcClient
print("Rabbit MQ is connected, starting server", file=sys.stderr)
app.run(debug=True, threaded=True, host='0.0.0.0')
except pika.exceptions.AMQPConnectionError as e:
print("Waiting for RabbitMq startup" + str(e), file=sys.stderr)
time.sleep(1)
except Exception as e:
worker.log.error(e)
exit(e)
I found where the bug came from:
Thedebug=True of the line app.run(debug=True, threaded=True, host='0.0.0.0') restart the server at the beginning.
The whole script is then restarted from the beginning. Because of it, another rpcClient is initialized and consume from the same queue. Problem is that the previous thread is also running. This cause two rpcClient to consume from the same thread, with one that is virtually useless.
With mqtt subscribe client I am subscribing to lots of threads (over 6000) but not getting results that change on the fly. I'm lagging. Does mqtt give possibility to subscribe too many threads in parallel in background? loop_start would that be enough?
What should I pay attention to when subscribing to more topics?
import logging
import paho.mqtt.client as mqtt
import requests
import zmq
import pandas as pd
PORT=1351
def set_publisher():
context = zmq.Context()
socket_server = context.socket(zmq.PUB)
socket_server.bind(f"tcp://*:{PORT}")
return socket_server
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
#logging.warning(f"Connected with result code :: code : {rc}")
print(f"Connected with result code :: code : {rc}")
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(topics)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
msg = msg.payload
# logging.info(f"message:: {msg}\n")
print(f"message:: {msg}\n")
if msg:
publisher.send(f"{msg}")
def on_disconnect(client, userdata, rc):
if rc != 0:
# logging.warning(f"Unexpected disconnection :: code: {rc}")
print(f"Unexpected disconnection :: code: {rc}")
#todo: if rc is change hostname raise except
client = mqtt.Client(protocol=mqtt.MQTTv31, transport="tcp")
client.username_pw_set(******, password=******)
topics = [(f"topic{i}", 0) for i in 6000]
client.on_connect = on_connect
client.on_message = on_message
client.on_disconnect = on_disconnect
if client.connect(hostname= *****, port= **** , keepalive=300) != 0:
# logging.info("Could not connect to MQTT Broker !")
print("Could not connect to MQTT Broker !")
client.loop_forever(timeout=3000)
You are describing a situation of compute power (either at the client or the broker or in-between) not sufficient to handle your scenario. It is a common occurrance and that is what performance testing is for: does your setup handle your scenario for your requirements? Capacity planning then expands that question to: ... in the future.
In this case I'm working with the websockets module.
A typical server implementation is this:
import websockets
start_server = websockets.serve(counter, "localhost", 6789)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
But, I want to be able to stop and restart the server without stopping the event loop. It's not clear to me from the minimal documentation that I've seen about asyncio servers, how to do this. Nor do I know if websockets is implemented in quite the same way.
For instance, if I do something like this:
def counter():
start_server = websockets.serve(connection_handler, 'localhost', 6789)
this = loop.run_until_complete(start_server)
try:
loop.run_forever()
finally:
this.close()
loop.run_until_complete(this.wait_closed())
loop = asyncio.get_event_loop()
loop.create_task(anothertask)
startcounter = counter()
I can trigger the server to stop by calling loop.stop(). How do I stop the server without stopping the loop (and upsetting another task running on the loop)?
You can use asyncio.create_task to submit a task when the loop is already running. The run_until_complete() followed by run_forever() pattern is now deprecated because it is incompatible with asyncio.run, which is now the preferred way to run asyncio code.
The recommended approach is to use asyncio.run at top-level to start an async entry point (typically defined as async def main()), and do the rest from there. run_until_complete(x) then becomes simply await x, and run_forever() is not needed because you can await things like server.serve_forever() or an asyncio.Event of your choosing.
Since serve_forever doesn't appear to exist for websockets server, here is the variant with the event (untested):
async def connection_handler(...):
...
async def test(stop_request):
# occasionally stop the server to test it
while True:
await asyncio.sleep(1)
print('requesting stop')
stop_request.set()
async def main():
stop_request = asyncio.Event()
asyncio.create_task(test(stop_request))
while True:
print('starting the server')
server = await websockets.serve(connection_handler, 'localhost', 6789)
await stop_request.wait()
stop_request.clear()
print('stopping the server')
server.close()
await server.wait_closed()
asyncio.run(main())
This answer is based on the suggestions and answer provided by user4815162342 as well as borrowing a couple things from the websockets module documentation. The reason for my answer is largely to provide a more fleshed out example. I think that one of the hardest parts of working with asyncio is integrating it with other pieces of code without breaking it.
One thing I want to point out about this implementation is that the "server" object in the line of user4815162342's code:
async def somefunction(): # requires async function
server = await websockets.serve(handler, 'localhost', 6789)
is the same "server" object that would be created with this bit of code that you will see in the websockets documentation fairly often:
def somefunction(): # non-async function
start_server = websockets.serve(handler, 'localhost', 6789)
server = asyncio.run_until_complete(start_server)
And therefore it will respond to the method close().
Furthermore, the author of websockets suggests that the server can be started with:
async def somefunction(): # requires async function
async with websockets.serve(handler, 'localhost', 6789)
await something_that_says_stop
Note which of these can be called from a non-async function and which must occur "on the loop".
In my example (a somewhat fanciful bit of script) I integrate a tkinter GUI. This is accomplished with the aiotkinter module which, aside from it's pypi page, has hardly any other mention, let alone example code, on the interwebs. So, here's an example of how to use it with the websockets module in a scenario where the websockets server[s] can be started and stopped.
Notice that I do not call root.mainloop(). Instead, the "blocking line" in this is the asyncio.run() line. Also notice where the set_event_loop_policy line is placed.
This has been tested on mac with python 3.8.2 and TclTk 8.6. Thank you user4815162342 for your help. The language of asyncio is getting clearer and I think we need to get more example code and answers out there that uses the newer language/structures...
import sys
from functools import partial
import asyncio
import aiotkinter
from tkinter import *
from tkinter import messagebox
import json
import websockets
from netifaces import interfaces, ifaddresses, AF_INET
async def register(w, label2):
USERS.remove(w) if w in USERS else USERS.add(w)
label2.config(text=f"User Count = {len(USERS)}")
async def connection_handler(websocket, path, stop_request,
ticker, start_request, label, label2):
misterDict = {"Mr.White":"white", "Mr.Orange":"orange",
"Mr.Blonde":"khaki", "Mr.Pink":"pink",
"Mr.Blue":"blue", "Mr.Brown":"brown"}
await register(websocket, label2)
try:
message = json.dumps("welcome")
await websocket.send(message)
# ignore this -> while server.get():
async for message in websocket:
msg = json.loads(message)
if msg == "end":
return
elif msg == "stoptick":
ticker.set(False)
elif msg == "starttick":
ticker.set(True)
startTicker(ticker, label)
elif msg == "stopserver":
stop_request.set()
break
elif msg in misterDict.keys():
color = misterDict[msg]
changeColor(label, color)
# ignore this -> await asyncio.sleep(.05)
except Exception as E1:
print(f"Error! {E1}")
finally:
await register(websocket, label2)
async def main():
stop_request = asyncio.Event()
start_request = asyncio.Event()
# Some tkinter setup
root = Tk()
root.title("Color Server")
root.minsize(250, 250)
ticker = BooleanVar()
ticker.set(0)
server = BooleanVar()
server.set(0)
label = Label(master=root, height=4, width=20, bg="white",
bd=5)
label.pack()
buttonS = Checkbutton(master=root, text='Mr tick', variable=ticker,
height=3, command=partial(startTicker,
ticker, label))
buttonS.pack()
buttonW = Checkbutton(master=root, text='Websocket Server',
variable=server, height=3,
command=partial(startWeb, server,
label, stop_request, start_request))
buttonW.pack()
def on_closing():
if messagebox.askokcancel("Quit", "Do you want to quit?"):
QUITTER.add("yes")
if server.get():
stop_request.set()
else:
start_request.set()
buttonX = Button(master=root, text='Shut down everything', height=2,
command=on_closing)
buttonX.pack()
label2 = Label(master=root, text='User Count = 0',
height=2, bg="white", bd=2)
label2.pack()
root.protocol("WM_DELETE_WINDOW", on_closing)
# websocket server setup code.
serverlist = set()
iplist = [ifaddresses(face)[AF_INET][0]["addr"]
for face in interfaces() if AF_INET in ifaddresses(face)]
print(f"the interfaces found = {iplist}")
while True:
await start_request.wait()
start_request.clear()
if "yes" in QUITTER:
break
server.set(1)
for ipadd in iplist: # for each IP address in the system
# setup a websocket server
bound_handler = partial(connection_handler,
stop_request=stop_request, ticker=ticker,
start_request=start_request,
label=label, label2=label2)
sockserver = await websockets.serve(bound_handler, ipadd, 6789)
serverlist.add(sockserver)
await stop_request.wait()
stop_request.clear()
for sockserver in serverlist:
sockserver.close()
await sockserver.wait_closed()
serverlist.clear()
if "yes" in QUITTER:
break
server.set(0)
def startWeb(server, label, stop_request, start_request):
if not server.get():
stop_request.set()
return
else:
start_request.set()
def changeColor(label, color):
label.config(bg=color)
async def mRtick(ticker, label):
ticktock = [("tick", "w"), ("tock", "e")]
tocker = 0
while ticker.get():
a,b = ticktock[tocker]
label.config(text=a, anchor=b)
tocker = not tocker
await asyncio.sleep(0.5)
label.config(text="")
def startTicker(ticker, label):
if not ticker.get():
return
asyncio.create_task(mRtick(ticker, label))
QUITTER = set()
USERS = set()
asyncio.set_event_loop_policy(aiotkinter.TkinterEventLoopPolicy())
asyncio.run(main())
print("we've ended...")
sys.exit()
Please note that I discovered that I had an unnecessary while loop in the code. This has now been commented out.
The HandlerWebsockets does work fine and is just replying with what has been send at the moment throught messageToSockets(msg). However both tries to send messages to the websocket from the coroutine of the web application are not working. Looks like everything is blocked by these attempts...
class webApplication(tornado.web.Application):
def __init__(self):
handlers = [
(r'/', HandlerIndexPage),
(r'/websocket', HandlerWebSocket, dict(msg='start')),
]
settings = {
'template_path': 'templates'
}
tornado.web.Application.__init__(self, handlers, **settings)
#gen.coroutine
def generateMessageToSockets(self):
while True:
msg = str(randint(0, 100))
print ('new messageToCon: ', msg)
yield [con.write_message(msg) for con in HandlerWebSocket.connections]
yield gen.sleep(1.0)
if __name__ == '__main__':
ws_app = webApplication()
server = tornado.httpserver.HTTPServer(ws_app)
port = 9090
print('Listening on port:' + str(port))
server.listen(port)
IOLoop.current().spawn_callback(webApplication.generateMessageToSockets)
IOLoop.current().set_blocking_log_threshold(0.5)
IOLoop.instance().start()
Here the WebSockets Handler
class HandlerWebSocket(tornado.websocket.WebSocketHandler):
connections = set()
def initialize(self, msg):
print('HWS:' + msg)
def messageToSockets(self, msg):
print ('return message: ', msg)
[con.write_message(msg) for con in self.connections]
def open(self):
self.connections.add(self)
print ('new connection was opened')
pass
def on_message(self, message):
print ('from WebSocket: ', message)
self.messageToSockets(message)
def on_close(self):
self.connections.remove(self)
print ('connection closed')
pass
I am a bit lost in the examples, questions here, documentation etc. So any hint how to properly start a continous calling websocket routine is greatly appreciated
generateMessageToSockets will loop endlessly, generating messages as fast as it can without waiting for those messages to be sent. Since it starts first and never yields, the HTTPServer will never actually be able to accept a connection.
If you really want to send messages as fast as you can, the minimal solution without blocking would be
yield [con.write_message(msg) for con in HandlerWebSocket.connections]
yield gen.moment
But it would probably be better to use gen.sleep to send messages at regular intervals, instead of "as fast as possible".
unfortunately all the gen.routines tries didn't work for me. Moved back to threads
def generateMessageToSockets():
while True:
msg = str(randint(0, 100))
print ('new messageToCon: ', msg)
[con.write_message(msg) for con in HandlerWebSocket.connections]
sleep(1.0)
class WebApplication(tornado.web.Application):
def __init__(self):
handlers = [
(r'/', HandlerIndexPage),
(r'/websocket', HandlerWebSocket, dict(msg='start')),
]
settings = {
'template_path': 'templates'
}
tornado.web.Application.__init__(self, handlers, **settings)
if __name__ == '__main__':
tGenarate = threading.Thread(target=generateMessageToSockets)
tGenarate.start()
ws_app = WebApplication()
server = tornado.httpserver.HTTPServer(ws_app)
port = 9090
print('Listening on port:' + str(port))
server.listen(port)
ioloop.IOLoop.instance().start()
which works