I want two loggers, one for prod and one for dev. I found that it's mandatory to declare a root (because if I delete the root logger, python's raise me an error).
Here is my test.ini file:
[loggers]
keys=root,dev,prod
[handlers]
keys=handlerRoot,handlerDev,handlerProd
[formatters]
keys=formatterRoot,formatterDev,formatterProd
# Root
[logger_root]
level=WARNING
handlers=handlerRoot
[handler_handlerRoot]
class=StreamHandler
level=WARNING
formatter=formatterRoot
args=(sys.stdout,)
[formatter_formatterRoot]
format=
%(asctime)s - %(levelname)s - ROOOOT - %(message)s
datefmt=%A %d %B %Y at %H:%M:%S
# Prod
[logger_prod]
level=WARNING
handlers=handlerProd
qualname=prod
[handler_handlerProd]
class=StreamHandler
level=WARNING
formatter=formatterProd
args=(sys.stdout,)
[formatter_formatterProd]
format=
%(asctime)s - %(levelname)s - %(message)s
datefmt=%A %d %B %Y at %H:%M:%S
# Dev
[logger_dev]
level=DEBUG
handlers=handlerDev
qualname=dev
[handler_handlerDev]
class=StreamHandler
level=DEBUG
formatter=formatterDev
args=(sys.stdout,)
[formatter_formatterDev]
format=
%(asctime)s - %(name)s - %(levelname)s - %(module)s - %(filename)s
%(message)s
datefmt=%d-%m-%Y %H:%M:%S
And my main.py file:
import logging
from logging.config import fileConfig
fileConfig('test.ini')
loggerDev = logging.getLogger("dev")
loggerDev.warning("Not supposed to also log in root.")
But my console output this:
21-04-2020 23:05:41 - dev - WARNING - main - main.py
Not supposed to also log in root.
Tuesday 21 April 2020 at 23:05:41 - WARNING - ROOOOT - Not supposed to also log in root.
And I don't want the latest line to be printed. It seems that the loggerDev is a child of the root logger.
How to solve this?
Solution founded: set propagate to false.
In .ini file you can do it by adding this line in the [logger_myLogger] section:
propagate=0
Related
Added new log level verbose to the logger module. It is working, but the filename which gets logged along with it is not the correct one. It is always showing the logger module name instead of the module from where the log is getting triggered.
Below is the logger module code,
class _CustomRotatingFileHandler(logging.handlers.RotatingFileHandler):
def __init__(self, *args, **kwargs):
prev_umask = os.umask(0o000)
super().__init__(*args, **kwargs)
os.umask(prev_umask)
def doRollover(self):
prev_umask = os.umask(0o000)
super().doRollover()
os.umask(prev_umask)
def setup_logging():
reference = 'my_project'
date_fmt = '%(asctime)s %(levelname)s %(filename)s:%(lineno)d' \
' : %(message)s'
# Defining a custom log level for verbose log level
# selected a value between logging.INFO and logging.DEBUG
log_level_verbose = 15
logging.addLevelName(log_level_verbose, 'VERBOSE')
logger = logging.getLogger(reference)
logger.setLevel(logging.DEBUG)
# For custom log level verbose
log_func = lambda msg, *args, **kwargs: logger.log(log_level_verbose, msg, *args, **kwargs) # noqa: E731
setattr(logger, 'verbose', log_func)
file_handler = _CustomRotatingFileHandler(
log_file, maxBytes=10485760, backupCount=3
)
file_log_format = logging.Formatter(date_fmt, "%b %e %H:%M:%S")
file_handler.setFormatter(file_log_format)
file_handler.setLevel(log_level_verbose)
console_handler = logging.StreamHandler(sys.stdout)
console_log_format = logging.Formatter("%(message)s")
console_handler.setFormatter(console_log_format)
console_handler.setLevel(logging.INFO)
logger.handlers = [file_handler, console_handler]
return logger
The verbose level created here is to log message only to the log file and not to console.
The flow is, main.py creates the logger object by calling setup_logging and this logger object passed to test_module.py function for execution and logging.
The log file content is something like below where VERBOSE message is getting logged only in the file (as expected) but the filename logged along with it is wrong, it should be test_module.py
Feb 4 16:54:31 INFO main.py:151 : ----------------------
Feb 4 16:54:31 INFO main.py:152 : Executing command for 'task1'
Feb 4 16:54:31 INFO main.py:153 : ----------------------
Feb 4 16:54:31 INFO test_module.py:34 : Executing command 'ls -ltr'
Feb 4 16:54:31 VERBOSE log.py:88 : Some verbose log message
Feb 4 16:54:32 INFO test_module.py:37 : Result: total 24
-rwxr-xr-x. 1 root root 20606 Jan 13 13:31 main.py
Feb 4 16:54:32 INFO main.py:160 : Completed
Currently for the VERBOSE log, I'm getting the log message as below,
Feb 4 16:54:31 VERBOSE log.py:88 : Some verbose log message
The expected log message with correct filename should be as below,
Feb 4 16:54:31 VERBOSE test_module.py:35 : Some verbose log message
Finally I got the answer. Here is the line of the code that got modified,
# For custom log level verbose
log_func = lambda msg, *args, **kwargs: logger._log(log_level_verbose, msg, args, **kwargs) # noqa: E731
Changed from logger.log to logger._log and argument *args changed to args.
I found the precise reason behind this in one of the comments for an answer in How to add a custom loglevel to Python's logging facility by #rivy. Below is the explanation,
Using _log() instead of log() is needed to avoid introducing an extra level in the call stack. If log() is used, the introduction of the extra stack frame causes several LogRecord attributes (funcName, lineno, filename, pathname, ...) to point at the debug function instead of the actual caller.
And this is exactly the reason why filename was not working correctly with logger.log
I'm trying to run background task processing, redis and rabbitMQ work in separate docker containers
#dramatiq.actor(store_results=True)
def count_words(url):
try:
response = requests.get(url)
count = len(response.text.split(" "))
print(f"There are {count} words at {url!r}.")
except requests.exceptions.MissingSchema:
print(f"Message dropped due to invalid url: {url!r}")
result_backend = RedisBackend(host="172.17.0.2", port=6379)
result_broker = RabbitmqBroker(host="172.17.0.5", port=5672)
result_broker.add_middleware(Results(backend=result_backend))
dramatiq.set_broker(result_broker)
message = count_words.send('https://github.com/Bogdanp/dramatiq')
print(message.get_result(block=True))
RabbitMQ:
{"queue_name":"default","actor_name":"count_words","args":["https://github.com/Bogdanp/dramatiq"],"kwargs":{},"options":{},"message_id":"8e10b6ef-dfef-47dc-9f28-c6e07493efe4","message_timestamp":1608877514655}
Redis
1:C 22 Dec 2020 13:38:15.415 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 22 Dec 2020 13:38:15.417 * Running mode=standalone, port=6379.
1:M 22 Dec 2020 13:38:15.417 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 22 Dec 2020 13:38:15.417 # Server initialized
1:M 22 Dec 2020 13:38:15.417 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 22 Dec 2020 13:38:15.417 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 25 Dec 2020 10:08:12.274 * Background saving terminated with success
1:M 26 Dec 2020 19:23:59.445 * 1 changes in 3600 seconds. Saving...
1:M 26 Dec 2020 19:23:59.660 * Background saving started by pid 24
24:C 26 Dec 2020 19:23:59.890 * DB saved on disk
24:C 26 Dec 2020 19:23:59.905 * RDB: 4 MB of memory used by copy-on-write
1:M 26 Dec 2020 19:23:59.961 * Background saving terminated with success
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/dramatiq/message.py", line 147, in get_result
return backend.get_result(self, block=block, timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/dramatiq/results/backends/redis.py", line 81, in get_result
raise ResultTimeout(message)
dramatiq.results.errors.ResultTimeout: count_words('https://github.com/Bogdanp/dramatiq')
L.S,
i am trying to learn python(3) and am baffled by the behavior of time.sleep()
i run raspbian buster on a rpi3b.
#!/usr/bin/python3
import bluetooth
import sys
import time
_shuttermac ="11:22:33:44:5D:6A"
while True:
# check if the device is switched on by mac address lookup
print("Checking " + time.strftime("%a, %d %b %Y %H:%M:%S - ", time.gmtime()),end ="")
result = bluetooth.lookup_name(_shuttermac, timeout=5)
#print(result)
if (result != None):
print("Device detected by bluetooth mac lookup")
else:
print("Device with MAC " + _shuttermac + " NOT detected")
time.sleep(1) # check every 5 secs if device comes online
sys.exit()
output
Checking Thu, 25 Jun 2020 11:21:02 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:08 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:14 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:20 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:26 - Device with MAC 11:22:33:44:5D:6A NOT detected
as you can see, the output statement is 6 seconds apart, not 1 as coded
where did i go wrong?
The six second interval comes because each loop of while True does two time-consuming things when the lookup fails:
bluetooth.lookup_name(_shuttermac, timeout=5) has a 5 second timeout. If it is found quickly, it should complete faster
time.sleep(1) pauses the whole Python program for 1 second
You could shorten the timeout on bluetooth.lookup_name and the sleep() duration. This may have unintended consequences in the accuracy of the lookup (I'm not very familiar with it).
The returned output from pexpect.run() includes \r\n at the end of every line. Printing to the terminal using print(returnVal.decode()) correctly prints one line for each line returned. When I examine the output I see that the byte string contains \r\n. When I log that to a file I get double returns to the log file. I'm on a Mac using Python 3.7. Is there a way to set the preferred new line when writing the output? I am using pythons logging class and using the info() method to write the string. Output looks like this:
total 80
-rw-r--r-- 1 xxxx admin 1048 Nov 12 00:41 Constants.py
-rw-r--r-- 1 xxxx admin 5830 Nov 12 13:33 file1.py
-rw-r--r-- 1 xxxx admin 2255 Nov 12 00:51 file2.py
When it should look like:
total 80
-rw-r--r-- 1 xxxx admin 1048 Nov 12 00:41 Constants.py
-rw-r--r-- 1 xxxx admin 5830 Nov 12 13:33 file1.py
-rw-r--r-- 1 xxxx admin 2255 Nov 12 00:51 file2.py
Here is a simplified version of my original Logger class:
class Logger():
def __init__( self, path ):
msgFormat = '%(asctime)s.%(msecs)d\t%(message)s'
dateFormat = '%m/%d/%Y %H:%M:%S'
logging.basicConfig( format=msgFormat, datefmt=dateFormat, filename=path, level=logging.INFO )
def Log ( self, theStr ):
logging.info( str( theStr ))
The string being returned from Pexpect looks something like:
Line1\r\nLine2
Depending on how you log the output, it's advisable to format the newlines before sending to logger. However, if you must override the logging module's newline parameter for FileHandler, and as an experiment, you can do so by monkey patching its _open method as the functionality isn't available by default.
I used source code for Python version 3.8 to get _open function's definition.
import logging
def custom_open(self):
"""
Monkey patched _open function of class logging.FileHandler (Python 3.8)
"""
return open(self.baseFilename, self.mode, encoding=self.encoding, newline='')
logging.FileHandler._open = custom_open
if __name__ == "__main__":
pexpect_return = "Output\nTest"
my_log = logging.getLogger("test_logger")
my_log.setLevel(logging.INFO)
my_log.addHandler(logging.FileHandler("test.log"))
my_log.info(pexpect_return)
How it works
Python's logging module has a class FileHandler, which uses a method _open to create a file handler object to write and append to log files on disk. Its default implementation as of version 3.8 does not have the newline parameter so it uses default newlines.
Monkey patching is when you replace or update a method/function in one of your imported classes, as the program is running. This line logging.FileHandler._open = custom_open tells python to replace the _open method of the FileHandler class, with my custom_open method. Then later when I use my_log.addHandler(logging.FileHandler("test.log")), the new custom_open method is used to open the file with newline paramater.
You can further confirm that the new method is used to open the file by adding a suffix to the file name like this:
return open(self.baseFilename+"__Monkey_Patched", self.mode, encoding=self.encoding, newline='')
If you will now run that demo code, the filename will be "test.log__Monkey_Patched".
This code, however, will not replace any newline characters which you pass to the logger as part of the string to log. You need to process that beforehand.
I have a datetime from below generated code
from datetime import datetime
from pytz import timezone
tz = timezone('US/Pacific')
print(datetime.now(tz)) # 2019-06-17 05:41:22.189735-07:00
I am looking for this format of output
"Wed, 17 Jun 2019 05:41:22 -0700"
How can i achieve this?
You're looking for the datetime.strftime method:
print(datetime.now().strftime("%a, %d %b %Y %H:%M:%S.%f"))
i managed to fix it myself by
print(datetime.now(tz).strftime("%a, %d %B %Y %H:%M:%S %z")) # Mon, 17 June 2019 05:51:16 -0700