my first custom logging handler not working - python-3.x

I am trying to write custom handler for logging that would send logs to netcat
I can see on the receiving end that the connection is established and then closes, however, no messages are received and i can see no errors.
here is the code i am running
import socket
import time
import logging
hostname = '127.0.0.1'
port = '1234'
message = 'hello world!\n'
class Nc_handler(logging.Handler):
def __init__(self, hostname, port):
logging.Handler.__init__(self)
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((hostname, int(port)))
def emit(self, content):
log_entry = self.format(content)
print("Checking if emit is run")
self.socket.send(log_entry.encode())
logger = logging.getLogger(__name__)
# set format
nc_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
# create nc handler
nc_handler = Nc_handler(hostname, port)
# set handler's level
nc_handler.setLevel(logging.INFO)
# set handler's format
nc_handler.setFormatter(nc_format)
logger.addHandler(nc_handler)
logger.info(message)
If i use nc_handler.emit('Hello') it throws an error:
File "handler.py", line 35, in <module>
nc_handler.emit(message)
File "handler.py", line 17, in emit
log_entry = self.format(content)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/__init__.py", line 869, in format
return fmt.format(record)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/__init__.py", line 608, in format
record.message = record.getMessage()
AttributeError: 'str' object has no attribute 'getMessage'
So i suspect i don't use Formatter correctly but I can't figure out what exactly I am doing wrong.
I would really appreciate any help or advice.

Method emit has a wrong signature: its argument must be a LogRecord object. Not a string.
That is because format method needs a LogRecord object.
That is the cause of the AttributeError exception.
I see also a confusion I don't understand: why using message as argument for logging.info?
message is an attribute of the LogRecord object too and is taken from what you logged explicitly.
Use a hard-coded string:
logging.info("Hello")
Or use a different variable:
myvar = "Hello"
logging.info(myvar)
Does it help?

Related

Only show print statements on console & don't show any INFO in Python logging

I have configured my logging to log both File & Stream.
I just want to see the print statements in my console & see other info on my log file (except unhandled exceptions as python shows them on the console by default)
I set the StreamLogging.setLevel(level=logging.CRITICAL) but it still shows API connection messages.
I want to get the Python's default stderr like before configuring the logging, but also change the output, like adding %(asctime)s before printing.
My Logging
import logging
FileLoggingFormat = f"%(message)s"
FileLoggingDate = r"%m/%d/%Y %H:%M:%S"
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
# Add StreamHandler
StreamLogging = logging.StreamHandler()
logger.addHandler(StreamLogging)
StreamLogging.setFormatter(logging.Formatter("%(message)s"))
StreamLogging.setLevel(level=logging.CRITICAL)
# Add FileHandler
FileLogging = logging.FileHandler(
filename="Log.log",
mode="w",
encoding="utf-8"
)
logger.addHandler(FileLogging)
FileLogging.setFormatter(logging.Formatter(FileLoggingFormat, FileLoggingDate))
❌ Not suitable Result in Console
(Printed statements showed by πŸ“)
πŸ“ Port: 1523
πŸ“ NotFound.
INFO:library.connection.connection:Connecting...
Traceback (most recent call last):
File "c:\Users\test.py", line 20, in <module>
f.write(message)
TypeError: write() argument must be str
INFO:library.session.session:Session stopped
Session stopped
βœ… Suitable Result in Console
(Only display prints & unhandled exceptions)
πŸ“ Port: 1523
πŸ“ NotFound.
Traceback (most recent call last):
File "c:\Users\test.py", line 20, in <module>
f.write(message)
TypeError: write() argument must be str
βœ… Suitable Result in Console
(Only display prints & unhandled exceptions with date, added %(asctime)s to StreamHandler)
2022/10/22 10:16:03.678
πŸ“ Port: 1523
2022/10/22 10:16:03.700
πŸ“ NotFound.
2022/10/22 10:16:03.706
Traceback (most recent call last):
File "c:\Users\test.py", line 20, in <module>
f.write(message)
TypeError: write() argument must be str
There are several things that need explaining:
Using logging.basicConfig
You use logging.basicConfig(level=logging.INFO). That has for effect to create a StreamHandler and attach it to the root logger (the one you are using since logger = getLogger() and getLogger() with no argument returns the "root" logger).
At the end of your logging setup, if you do:
print(logger, logger.handlers)
It will give:
<RootLogger root (INFO)> [<StreamHandler <stderr> (NOTSET)>, <StreamHandler <stderr> (CRITICAL)>, <FileHandler /home/vvvvv/Log.log (NOTSET)>]
You have 2 StreamHandler instead of just 1. Moreover, the superfluous StreamHandler has a logging level of NOTSET meaning it will log everything. This is the cause of your bug.
You could solve it by removing this line:
logging.basicConfig(level=logging.INFO)
Doing so, gives
<RootLogger root (INFO)> [<StreamHandler <stderr> (CRITICAL)>, <FileHandler /home/vvvvv/Log.log (NOTSET)>]
Using a child logger instead of the root logger
As previously said, you are using the root logger (logger = logging.getLogger()). You should refrain from doing so and instead use a child logger, like this:
# logger = logging.getLogger()
logger = logging.getLogger("foobar")
logger.propagate = False
Set propagate to False to avoid sending the logs upwards to the root logger.
Piecing everything together gives the correct behavior:
import logging
FileLoggingFormat = f"%(message)s"
FileLoggingDate = r"%m/%d/%Y %H:%M:%S"
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("foobar")
logger.propagate = False
# Add StreamHandler
StreamLogging = logging.StreamHandler()
StreamLogging.setFormatter(logging.Formatter("%(message)s"))
StreamLogging.setLevel(level=logging.CRITICAL)
logger.addHandler(StreamLogging)
# Add FileHandler
FileLogging = logging.FileHandler(
filename="Log.log",
mode="w",
encoding="utf-8"
)
logger.addHandler(FileLogging)
FileLogging.setFormatter(
logging.Formatter(FileLoggingFormat, FileLoggingDate)
)

How to zmq.poll() some socket and some sort of variable?

I'm attempting to poll a few sockets and a multiprocessing.Event
The documentation states:
A zmq.Socket or any Python object having a fileno() method that
returns a valid file descriptor.
which means that I can't use my Event but I should be able to use a file(as returned from open(...) or an io object (anything from the io library), but I'm having no success:
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3\helpers\pydev\pydevd.py", line 1683, in <module>
main()
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3\helpers\pydev\pydevd.py", line 1677, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3\helpers\pydev\pydevd.py", line 1087, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2017.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:\work\polldamnyou.py", line 122, in <module>
p = poller.poll(1000)
File "C:\WinPython-64bit-3.6.3.0Qt5\python-3.6.3.amd64\Lib\site-packages\zmq\sugar\poll.py", line 99, in poll
return zmq_poll(self.sockets, timeout=timeout)
File "zmq\backend\cython\_poll.pyx", line 143, in zmq.backend.cython._poll.zmq_poll
File "zmq\backend\cython\_poll.pyx", line 123, in zmq.backend.cython._poll.zmq_poll
File "zmq\backend\cython\checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Unknown error
I have found the same question asked before but the solution was to use another socket which is sidestepping the problem. I am curious and want to see this working. Does any one have any clues what sort of object can be used in zmq.Poller other than a socket?
Edit: a few things I've tried
import traceback, os, zmq
def poll(poller):
try:
print('polled: ', poller.poll(3))
except zmq.error.ZMQError as e:
traceback.print_exc()
class Pollable:
def __init__(self):
self.fd = os.open('dump', os.O_RDWR | os.O_BINARY)
self.FD = self.fd
self.events = 0
self.EVENTS = 0
self.revents = 0
self.REVENTS = 0
def fileno(self):
return self.fd
def __getattribute__(self, item):
if item != '__class__':
print("requested: ", item)
return super().__getattribute__(item)
ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
poller = zmq.Poller()
poller.register(sock, zmq.POLLIN)
poll(poller) # works
file = open('dump', 'w+b')
print("fileno: ", file.fileno())
poller.register(file, zmq.POLLIN)
poll(poller) # fails
file.events = 0
file.revents = 0
file.EVENTS = 0
file.REVENTS = 0
file.fd = file.fileno()
file.FD = file.fileno()
poll(poller) # still fails
poller.unregister(file)
file.close()
poll(poller) # works
fd = os.open('dump', os.O_RDWR|os.O_BINARY)
print("fd: ", fd)
dummy = Pollable()
poller.register(dummy, zmq.POLLIN)
poll(poller) # fails
__getattribute__ shows that that fd and fileno are being accessed, but nothing else, so what is still wrong?!
In case one has never worked with ZeroMQ,one may here enjoy to first look at "ZeroMQ Principles in less than Five Seconds"before diving into further details
Q : "what sort of object can be used in zmq.Poller other than a socket?"
Welcome to the lovely lands of the Zen-of-Zero. Both the published API and the pyzmq ReadTheDocs are clear and sound on this :
The zmq_poll() function provides a mechanism for applications to multiplex input/output events in a level-triggered fashion over a set of sockets. Each member of the array pointed to by the items argument is a zmq_pollitem_t structure. The nitems argument specifies the number of items in the items array. The zmq_pollitem_t structure is defined as follows :
typedef struct
{
void //*socket//;
int //fd//;
short //events//;
short //revents//;
} zmq_pollitem_t;
For each zmq_pollitem_t item, zmq_poll() shall examine either the ØMQ socket referenced by socket or the standard socket specified by the file descriptor fd, for the event(s) specified in events. If both socket and fd are set in a single zmq_pollitem_t, the ØMQ socket referenced by socket shall take precedence and the value of fd shall be ignored.
For each zmq_pollitem_t item, zmq_poll() shall first clear the revents member, and then indicate any requested events that have occurred by setting the bit corresponding to the event condition in the revents member.
If none of the requested events have occurred on any zmq_pollitem_t item, zmq_poll() shall wait timeout milliseconds for an event to occur on any of the requested items. If the value of timeout is 0, zmq_poll() shall return immediately. If the value of timeout is -1, zmq_poll() shall block indefinitely until a requested event has occurred on at least one zmq_pollitem_t.
Last, but not least,the API is explicit in warning about implementation :
The zmq_poll() function may be implemented or emulated using operating system interfaces other than poll(), and as such may be subject to the limits of those interfaces in ways not defined in this documentation.
Solution :
For any wished-to-have-poll()-ed object, that does not conform to the given mandatory property of having an ordinary fd-filedescriptor, implement a such mandatory property mediating-proxy, meeting the published API-specifications or do not use it with zmq_poll() for successful API calls.
There is no third option.

How do I expose a signal to DBUS in Python?

I know quiet well how to create methods – however, I wish to expose some signals using Python so that severall programms can be called at once.
class Bildschirm(dbus.service.Object):
DBUS_NAME = 'hm.retro.Retro'
DBUS_OBJECT_PATH = '/hm/retro/Retro'
DBUS_INTERFACE = 'hm.retro.Retro'
def __init__(self, _Visio):
self.Visio = _Visio
self.bus = dbus.SessionBus()
bus_name = dbus.service.BusName(self.DBUS_NAME, bus=self.bus)
super().__init__(bus_name, self.DBUS_OBJECT_PATH)
#dbus.service.method(DBUS_INTERFACE, in_signature='i')
def delta(self, nr):
weiter = self.Visio.akt_sender_nr
weiter = (weiter+nr) % SENDER_AM
self.Visio.akt_sender_nr = weiter
self.Visio.update_sender_labels()
self.Visio.screen.force_update()
self.Visio.screen.draw_next_frame()
#dbus.service.signal(DBUS_INTERFACE)
def config_change_visio(self):
self.Visio.load_config()
self.Visio.update_sender_labels()
self.Visio.update_preset_labels()
self.Visio.screen.force_update()
self.Visio.screen.draw_next_frame()
However, if I try to call the signal with dbus-send:
dbus-send --session --print-reply --dest=hm.retro.Retro /hm/retro/Retro hm.retro.Retro.config_change_visio
I get an error message:
Error org.freedesktop.DBus.Error.UnknownMethod: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/dbus/service.py", line 654, in _message_cb
(candidate_method, parent_method) = _method_lookup(self, method_name, interface_name)
File "/usr/lib/python3/dist-packages/dbus/service.py", line 246, in _method_lookup
raise UnknownMethodException('%s is not a valid method of interface %s' % (method_name, dbus_interface))
dbus.exceptions.UnknownMethodException: org.freedesktop.DBus.Error.UnknownMethod: Unknown method: config_change_visio is not a valid method of interface hm.retro.Retro
What am I doing wrong?
If you pass --print-reply to dbus-send, you are implying that the message type you’re sending is a method call, as only method calls can have a reply. The dbus-send man page says:
--print-reply
Block for a reply to the message sent, and print any reply received in
a human-readable form. It also means the message type (--type=) is
method_call.
Remove the --print-reply argument from your dbus-send invocation, and add --type=signal and it should work.
Note that while it shouldn’t affect this bug, it’s more conventional for D-Bus signal names to be in CamelCase, not underscore_case. The D-Bus specification says:
It is conventional for member names on D-Bus to consist of capitalized
words with no punctuation ("camel-case"). Method names should usually
be verbs, such as GetItems, and signal names should usually be a
description of an event, such as ItemsChanged.
So perhaps your signal should be called VisioConfigChanged instead. Similarly, perhaps your method should be called something like AddDelta.

Aiomas RPC Python 3.6

My goal is to bind TCP RPC server aiomas and clients in different programming languages.
I'm setting the connection, but neither the response from the server nor the response is received.
The documentation says:
https://media.readthedocs.org/pdf/aiomas/1.0.3/aiomas.pdf
Page 26: - On the RPC level, it a three-tuple:
[function, [arg0, arg1, ...], {kwarg0: val0, kwarg1: val1}]
function is a string containing the name of an exposed functions; The type of the arguments and keyword arguments may vary depending on the function.
This simple server:
import aiomas
class Server:
router = aiomas.rpc.Service()
#aiomas.expose
def ping(self, i):
print('Ping receive data: {}'.format(i))
return i
if __name__ == '__main__':
server = aiomas.run(aiomas.rpc.start_server(
('127.0.0.1', 5000),
Server())
)
print('Server run')
aiomas.run(server.wait_closed())
And this my problem tcp client
import socket
import pickle
MESS = ['ping', [1]]
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5000))
s.settimeout(1.5)
s.send(pickle.dumps(MESS))
data = s.recv(1024)
s.close()
Tell me please what's wrong. It is necessary to understand in order to realize this in other languages.Much grateful for any hint
I read the documentation several times, But I did not understand how to compose the header of the package. As a result, I figured out that the last byte is the length of the package. Everything worked.
Here is an example package
data='[0, 0, ["methood", ["args"], {"kwargs":"if exists"}]]'
hex(len(data)) --> 0x35
b'\x00\x00\x00\x35[0, 0, ["methood", ["args"], {"kwargs":"if exists"}]]'

In PySpark, how can I log to log4j from inside a transformation

I want to log to the standard logger inside an executor during transformation with log levels and formatting respected. Unfortunately I can't get access to the log4j logger object inside the method as it's not serializable, and the spark context isn't available inside the transformation. I could just log outside of the transformation all of the objects I'm going to touch but that doesn't really help debugging or monitoring code execution.
def slow_row_contents_fetch(row):
rows = fetch_id_row_contents(row) # API fetch, DB fetch, etc
# This shows up, but not controllable by log level
print "Processed slow row with {} results".format(len(rows))
return rows
sc.parallelize(fetchable_ids).flatMap(slow_row_contents_fetch, True)
Outside of the transformation I can get the logger via:
logger = sc._jvm.org.apache.log4j.LogManager.getRootLogger()
logger.warn('This will show up as expected')
But the sc isn't available inside the transformation, for good reasons. You see the following message if you try to call the sc directly inside the transformation:
Exception: It appears that you are attempting to reference
SparkContext from a broadcast variable, action, or transforamtion.
SparkContext can only be used on the driver, not in code that it run
on workers. For more information, see SPARK-5063.
I can just print but that's not easily filterable and just get tracked as unformatted error messages to the log4j logger.
Serializing the logger itself, as excepted, fails when calling the logger within the transform function:
...
File "/usr/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
File "/usr/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 36, in deco
File "/usr/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o36.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
at py4j.Gateway.invoke(Gateway.java:252)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
Is there a way to get access to the executor logger during transformations in pyspark?
After a few hours of digging into the spark repository, it seems this is impossible to achieve currently. The executor doesn't actually have a jvm instance it's attached to, the data is just streamed over the socket without a jvm native binding to utilize.
Here's the worker creation code that streams the error messages to stderr:
private def createSimpleWorker(): Socket = {
...
val worker = pb.start()
// Redirect worker stdout and stderr
redirectStreamsToStderr(worker.getInputStream, worker.getErrorStream)
...
}
/**
* Redirect the given streams to our stderr in separate threads.
*/
private def redirectStreamsToStderr(stdout: InputStream, stderr: InputStream) {
try {
new RedirectThread(stdout, System.err, "stdout reader for " + pythonExec).start()
new RedirectThread(stderr, System.err, "stderr reader for " + pythonExec).start()
} catch {
case e: Exception =>
logError("Exception in redirecting streams", e)
}
}
And here's the worker.py code for communicating the job processing. There's no place to emit log messages or message type which indicates a log event.
try:
...
command = pickleSer._read_with_length(infile)
if isinstance(command, Broadcast):
command = pickleSer.loads(command.value)
func, profiler, deserializer, serializer = command
init_time = time.time()
def process():
iterator = deserializer.load_stream(infile)
serializer.dump_stream(func(split_index, iterator), outfile)
if profiler:
profiler.profile(process)
else:
process()
except Exception:
try:
write_int(SpecialLengths.PYTHON_EXCEPTION_THROWN, outfile)
write_with_length(traceback.format_exc().encode("utf-8"), outfile)
except IOError:
# JVM close the socket
pass
except Exception:
# Write the error to stderr if it happened while serializing
print("PySpark worker failed with exception:", file=sys.stderr)
print(traceback.format_exc(), file=sys.stderr)
exit(-1)
finish_time = time.time()
report_times(outfile, boot_time, init_time, finish_time)
write_long(shuffle.MemoryBytesSpilled, outfile)
write_long(shuffle.DiskBytesSpilled, outfile)
# Mark the beginning of the accumulators section of the output
write_int(SpecialLengths.END_OF_DATA_SECTION, outfile)
write_int(len(_accumulatorRegistry), outfile)
for (aid, accum) in _accumulatorRegistry.items():
pickleSer._write_with_length((aid, accum._value), outfile)
...
And finally the message types available:
class SpecialLengths(object):
END_OF_DATA_SECTION = -1
PYTHON_EXCEPTION_THROWN = -2
TIMING_DATA = -3
END_OF_STREAM = -4
NULL = -5
Have a look at this question
Similar situation
You can get your map function to return you an object that can contain a stack trace string or a real object, and a bool flag stating if there was an error. This can be useful to debug a task that has side effects, or if you have specific data conditions that cause failures.

Resources