I have a module name acms and inside that have number of python files.The main.py has calls to other python files.I have added logs in those files, which are displayed on console but i also want to write these logs in a file called all.log, i tried with setting log levels and logger in a file called log.py but didnt get the expected format,since am new to python am getting difficulty in handling logs
Use the logging module and use logger = logging.getLogger(__name__). Then it will use the correct logger with the options that you have set up.
See the thinkpad-scripts project for its logging. Also the logging cookbook has a section for logging to multiple locations.
We use the following to log to the console and the syslog:
kwargs = {}
dev_log = '/dev/log'
if os.path.exists(dev_log):
kwargs['address'] = dev_log
syslog = logging.handlers.SysLogHandler(**kwargs)
syslog.setLevel(logging.DEBUG)
formatter = logging.Formatter(syslog_format)
syslog.setFormatter(formatter)
logging.getLogger('').addHandler(syslog)
Related
I am working on adding new features to project. Within the project I am logging stdout to a file because some components are printing information useful for debugging. I recently added a new feature to the project which uses papermill to run jupyter notebook. The problem I am having is that papermill is printing everything to the console even if I redirect stdout to a temporary variable.
Below you can see a sample code,
with io.StringIO() as buf, redirect_stdout(buf):
pm.execute_notebook(
path,
path,
progress_bar=False,
stdout_file=buf,
stderr_file=buf,
parameters=dict(**params)
)
print("!!! redirected !!!")
print("!!! redirected !!!")
The first print statement successfully gets redirected to the buf while everything pm.execute_notebook prints goes to the console. The last print statement prints on the console as expected.
To solve the problem I had to change the handler and logging level of the logger.
To get the logger:
logger = logging.getLogger('papermill')
To change the logging level:
logger.setLevel('WARNING')
To remove the stream handler:
logger.removeHandler(logging.StreamHandler())
Removing the stream handler and setting the right level solved my problem. Here is a link to Python logging documentation.
I am trying to learn how to use the logging module.
I want to log information to both console and to file.
I confess that I have not completed studying both https://docs.python.org/3/library/logging.html#logging.basicConfig and https://docs.python.org/3/howto/logging.html
It's a little daunting for a novice like me to learn all of it, but I am working on it.
I am trying to use a modified version of the “Logging to multiple destinations” program from https://docs.python.org/3/howto/logging-cookbook.html, to which I refer as “Cookbook_Code”.
The Cookbook_Code appears at that URL under the title "Logging to multiple destinations".
But I have two problems:
The Cookbook Code saves to a file named:
"E:\Zmani\Logging\Logging_to_multiple_destinations_python.org_aaa.py.txt",
and I cannot figure out:
A. Why the Cookbook Code does that, nor
B. How to make the logging module save instead to a the following filepath (which I stored in a var, "logfile_fullname"): "e:\zmani\Logging\2020-10-14_14_14_os.walk_script.log"
I cannot figure out how to have the log file use the following datetime format:
"YYYY-MM-DD_HH-MM-SS - INFO: Sample info."
instead of the following datetime format: "10/14/2020 03:00:22 PM - INFO: Sample info."
I would like the console output include the same datetime prefix:
"YYYY-MM-DD_HH-MM-SS -"
Any suggestions would be much appreciated.
Thank you,
Marc
Here’s the code I have been running:
log_file_fullname = "e:\zmani\Logging\2020-10-14_14_14_os.walk_script.log"
# https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook
import logging
# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%Y-%m-%d_%H-%M-%S',
filename=log_file_fullname,
filemode='w')
console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)
logging.info('Sample info.')
logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')
logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
A quick run of your code showed that it already does 1.B and 2 of your problems.
Your provided URLs showed nowhere that Logging_to_multiple_destinations_python.org_aaa.py.txt is being used. It doesn't matter anyway. It just a path to a text file provided that its parent folders exist. So 1.A is just merely a demonstration.
If you add %(asctime)s to the console's formatter, it will give you 3.
formatter = logging.Formatter('%(asctime)s %(name)-12s: %(levelname)-8s %(message)s', datefmt='%Y-%m-%d_%H-%M-%S')
You should only use basicConfig if you don't need to add any logger nor doing a complicated setup.
basicConfig will only have an effect if there aren't any handler in the root logger. If you use filename argument, it creates a FileHandler. If you use stream argument, it creates a StreamHandler. And you cannot use both arguments at once.
So you said that you need to output to file and console, just create a handler for each of them.
import logging
log_file_fullname = "2020-10-14_14_14_os.walk_script.log"
# config file handler
file_handler = logging.FileHandler(log_file_fullname)
file_handler.setLevel(logging.DEBUG)
fmt_1 = logging.Formatter('%(asctime)s %(name)-12s: %(levelname)-8s %(message)s',
datefmt='%Y-%m-%d_%H-%M-%S')
file_handler.setFormatter(fmt_1)
# config console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
fmt_2 = fmt_1 # you could add console only formatter here
console_handler.setFormatter(fmt_2)
# retrieve a root logger and add handlers
root_logger = logging.getLogger()
root_logger.setLevel(logging.NOTSET) # change from default WARNING to NOTSET,
root_logger.addHandler(file_handler) # this allows root_logger to take all propagated
root_logger.addHandler(console_handler) # messages from other child loggers.
# this line use root logger by default
logging.info('Sample info.')
# create module loggers
logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')
# logging by module loggers
logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
I am using snowflake PUT command from Python to move files from local system to snowflake staging.
I have 400 (40 MB each) files so I am using command like -> put file:///Path/file_name*
It is working and loading all the files but its taking around 30 mins.
I want to know the progress so that I can be sure its progressing, is there a way to print logs after each file is loaded ( file 1 is moved to staging, file 2 is moved to staging etc.)
Is there a way to print logs after each file is loaded?
While the statement execution is non-interactive when used from a library, the Snowflake python connector does support logging its execution work.
Here's a shortened snippet that incorporates the example from the link above:
# Assumes a 'con' object pre-exists and is connected to Snowflake already
import logging
for logger_name in ['snowflake.connector', 'botocore', 'boto3']:
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(funcName)s() - %(message)s'))
logger.addHandler(ch)
con.cursor().execute("put file:///Path/file_name* #stage_name")
# Optional, custom app log:
# logging.info("put command completed execution, exiting")
con.close()
Watching the output (to stderr) while this program runs will yield the following (filtered for just upload messages):
~> python3 your_logging_script.py 2>&1 | grep -F "upload_one_file()"
[…]
2020-06-24 04:57:06,495 - upload_one_file() - done: status=ResultStatus.UPLOADED, file=/Path/file_name1, (…)
2020-06-24 04:57:07,312 - upload_one_file() - done: status=ResultStatus.UPLOADED, file=/Path/file_name2, (…)
2020-06-24 04:57:09,121 - upload_one_file() - done: status=ResultStatus.UPLOADED, file=/Path/file_name3, (…)
[…]
You can also configure the python logger to use a file, and tail the file instead of relying on the stderr (from logging.StreamHandler) used for simplicity above.
If you need to filter the logging for only specific messages, the logging python module supports attaching your own filters that decide on each record emitted. The following filters for just the upload_one_file() function call messages (use record.message field to filter over the log message instead of on the function name used in example below):
class UploadFilter(logging.Filter):
def filter(self, record):
# Only tests one condition, but you could chain conditions here
return "upload_one_file" in record.funcName.lower()
for logger_name in ['snowflake.connector', 'botocore', 'boto3']:
logger = logging.getLogger(logger_name)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s - %(funcName)s() - %(message)s'))
ch.addFilter(UploadFilter())
# ch.addFilter(AnyOtherFilterClass())
logger.addHandler(ch)
Note: If you are changing your handlers (stream to file), ensure you add the filter to the actual new handler too, and the handler to the logger. You can read the tutorial on logging in Python to understand its mechanism better.
I'm writing a log file in my python script as below:
logfile = 'Datalog'+currtime+'.log'
logging.basicConfig(filename=logfile,level=logging.DEBUG,format='%(asctime)s %(message)s', datefmt='%d/%m/%Y %H:%M:%S')
logging.info(file+' is processed successfully')
As this job has been scheduled to run every 15 mins, its creating new log files even it has no files to process.. Is there any way to append the same log file instead of creating new files in Python 3.x? Any help would be appreciated!
From your example, you append a variable onto the log file name: I'm assuming it's a timestamp. If you target the same filename again and again, it will be appended. See the example below from the docs:
import logging
LOG_FILENAME = 'example.log'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)
logging.debug('This message should go to the log file')
The example above states: "If you run the script repeatedly, the additional log messages are appended to the file."
So if you change your filename to:
logfile = 'Datalog.log'
This will append to the existing file if Datalog.log exists, rather than create a new file.
I am trying to set python logger which always writes INFO level to the stdout and at DEBUG level to a file. Something similar to
https://stackoverflow.com/a/11111212/3516550
but without creating anotherlogger object. I tried this, but both of them get the default level of logging.WARNING. Is there someway for me to set both to the original logging object? The rest of the code uses logging, I'd like to preserve that, if possible.
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
file_handler = logging.FileHandler("my_log.log")
file_handler.setLevel(logging.DEBUG)
logging.basicConfig(handlers=[file_handler, stream_handler])
The python version I am using is
Python 3.6.3 :: Anaconda custom (64-bit)