I am using python 3.7.6. and abseil module for logging messages with absl-py 0.9.0. I am using this piece of code for my tests.
from absl import logging
from absl import app
def main(argv):
#logging.set_stderrthreshold(logging.ERROR)
#logging._warn_preinit_stderr = False
logging.set_verbosity(logging.DEBUG)
print(' 0 -----')
logging.debug(' 1 logging-debug-test')
logging.info(' 2 logging-info-test')
logging.warning(' 3 logging-warning-test')
logging.error('4 logging-error-test')
print(' 5 -----')
if __name__ == '__main__':
app.run(main)
When testing it in a Jupyter notebook, it is clear from the color code of the background that abseil messages are in the stderr stream.
Same things when executing the python code in a shell:
I tried few things with different values like:
logging.set_stderrthreshold(logging.DEBUG)
logging._warn_preinit_stderr = True
but I still see 100% the same output.
How can I redirect output abseil logging messages to stdout instead of stderr ?
Is it expected to have the logging output messages redirect to stderr and not stdout? I am probably missing something with the logging logic and I want to better understand it.
I was told that this is the standard behavior and what Python's standard logging module does. In my case adding the following line redirect the logging messages to stdout:
logging.get_absl_handler().python_handler.stream = sys.stdout
Now in my Jupyter notebook it looks like that:
This did NOT work for me for some reason:
from absl import logging
import sys
logging.get_absl_handler().python_handler.stream = sys.stdout
But this did:
import logging
import sys
logging.basicConfig(stream=sys.stdout)
Related
I've tried every method in the title to run it in the background but here are the issues i got trying to use them:
pythonw and pyw: server doesn't work, going to localhost:8000 error with ERR_EMPTY_RESPONSE.
& arg and START/B: doesn't start script in background and instead output server log
so now I'm short on ideas on how to run this script in the background.
Using pythonw, it should help to explicitly redirect stdout and stderr to a file – maybe this behavior is somehow related to the problem described here (although this seems to be specific to Python 2.7). Not capturing the output by redirecting it to os.devnull seems to work as well.
The following script produces a minimum working server example with pythonw for me (using Python 3.7.9):
import http.server
import os
import sys
if __name__ == "__main__":
sys.stdout = sys.stderr = open(os.devnull, "w")
httpd = http.server.HTTPServer(("localhost", 8000), http.server.SimpleHTTPRequestHandler)
httpd.serve_forever()
I'm using python 3.7.3, with flask version 1.0.2.
When running my app.py file without the following imports:
import logging
logging.basicConfig(filename='api.log',level=logging.DEBUG)
Flask will display relevant debug information to console, such as POST/GET requests and which IP they came from.
As soon as DEBUG logging is enabled, I no longer receive this output. I have tried running my application in debug mode:
app.run(host='0.0.0.0', port=80, debug=True)
But this produces the same results. Is there a way to have both console output, and python logging enabled? This might sound like a silly request, but I would like to use the console for demonstration purposes, while having the log file present for troubleshooting.
Found a solution:
import logging
from flask import Flask
app = Flask(__name__)
logger = logging.getLogger('werkzeug') # grabs underlying WSGI logger
handler = logging.FileHandler('test.log') # creates handler for the log file
logger.addHandler(handler) # adds handler to the werkzeug WSGI logger
#app.route("/")
def index():
logger.info("Here's some info")
return "Hello World"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
Other Examples:
# logs to console, and log file
logger.info("Some text for console and log file")
# prints exception, and logs to file
except Exception as ue:
logger.error("Unexpected Error: malformed JSON in POST request, check key/value pair at: ")
logger.error(ue)
Source:
https://docstrings.wordpress.com/2014/04/19/flask-access-log-write-requests-to-file/
If link is broken:
You may be confused because adding a handler to Flask’s app.logger doesn’t catch the output you see in the console like:
127.0.0.1 - - [19/Apr/2014 18:51:26] "GET / HTTP/1.1" 200 -
This is because app.logger is for Flask and that output comes from the underlying WSGI module, Werkzeug.
To access Werkzeug’s logger we must call logging.getLogger() and give it the name Werkzeug uses. This allows us to log requests to an access log using the following:
logger = logging.getLogger('werkzeug')
handler = logging.FileHandler('access.log')
logger.addHandler(handler)
# Also add the handler to Flask's logger for cases
# where Werkzeug isn't used as the underlying WSGI server.
# This wasn't required in my case, but can be uncommented as needed
# app.logger.addHandler(handler)
You can of course add your own formatting and other handlers.
Flask has a built-in logger that can be accessed using app.logger. It is just an instance of the standard library logging.Logger class which means that you are able to use it as you normally would the basic logger. The documentation for it is here.
To get the built-in logger to write to a file, you have to add a logging.FileHandler to the logger. Setting debug=True in app.run, starts the development server, but does not change the log level to debug. As such, you'll need to set the log level to logging.DEBUG manually.
Example:
import logging
from flask import Flask
app = Flask(__name__)
handler = logging.FileHandler("test.log") # Create the file logger
app.logger.addHandler(handler) # Add it to the built-in logger
app.logger.setLevel(logging.DEBUG) # Set the log level to debug
#app.route("/")
def index():
app.logger.error("Something has gone very wrong")
app.logger.warning("You've been warned")
app.logger.info("Here's some info")
app.logger.debug("Meaningless debug information")
return "Hello World"
app.run(host="127.0.0.1", port=8080)
If you then look at the log file, it should have all 4 lines printed out in it and the console will also have the lines.
I am running the following code in a custom Python application:
if __name__ == '__main__':
logging.basicConfig(filemode='example.log', level=logging.DEBUG)
logging.debug('This message should go to the log file')
but the output is being written to standard out. I ran the same code from a Jupyter notebook and it creates the example.log file and writes the log message to it.
I read that the order of imports may be important. Here is the order:
import logging
import argparse
import time
import os
import sys
import json
You made a typo in the arguments to basicConfig.
Instead of setting filename to example.log, you set filemode, which is something else!
It worked for me like this:
import logging
if __name__ == '__main__':
logging.basicConfig(filename='example.log', level=logging.DEBUG)
logging.debug('This message should go to the log file')
I use scrapy 1.1.0, and I have 5 spiders in the "spiders" folder.
In every spider, I try to use python3 logging module. And the code structure like this :
import other modules
import logging
class ExampleSpider(scrapy.Spider):
name = 'special'
def __init__(self):
# other initializations
# set log
self.log = logging.getLogger('special')
self.log.setLevel(logging.DEBUG)
logFormatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
# file handler
fileHandler = logging.FileHandler(LOG_PATH) # LOG_PATH has defined
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(logFormater)
self.log.addHandler(fileHandler)
# other functions
every spider has the same structure.When I run these spiders, I check the log file, they did exist, but their size are always 0 byte.
And the other question is that when I run one spider, it always generated two or more log files. Like I run a spider, and it will generate a.log and b.log.
Any answers would appreciate.
You can set log file via LOG_FILE setting in settings.py or via command line argument --logfile FILE, i.e. scrapy crawl myspider --logfile myspider.log
As described in the official docs
i can't import zxlolbot.py
have problem...
No handlers could be found for logger "zxlolbot"
It means zxlolbot uses logging module without configuring it properly. You should see:
logger = logging.getLogger(__name__)
in zxlolbot.py. Add this line after it:
logger.addHandler(logging.NullHandler())
To save logging messages to a file, you could add in bot.py:
if __name__=="__main__":
import logging
logging.basicConfig(filename='bot.log', level=logging.INFO)
This sounds like you didn't set up the logging module properly.