Python, Flask print to console and log file simultaneously - python-3.x

I'm using python 3.7.3, with flask version 1.0.2.
When running my app.py file without the following imports:
import logging
logging.basicConfig(filename='api.log',level=logging.DEBUG)
Flask will display relevant debug information to console, such as POST/GET requests and which IP they came from.
As soon as DEBUG logging is enabled, I no longer receive this output. I have tried running my application in debug mode:
app.run(host='0.0.0.0', port=80, debug=True)
But this produces the same results. Is there a way to have both console output, and python logging enabled? This might sound like a silly request, but I would like to use the console for demonstration purposes, while having the log file present for troubleshooting.

Found a solution:
import logging
from flask import Flask
app = Flask(__name__)
logger = logging.getLogger('werkzeug') # grabs underlying WSGI logger
handler = logging.FileHandler('test.log') # creates handler for the log file
logger.addHandler(handler) # adds handler to the werkzeug WSGI logger
#app.route("/")
def index():
logger.info("Here's some info")
return "Hello World"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
Other Examples:
# logs to console, and log file
logger.info("Some text for console and log file")
# prints exception, and logs to file
except Exception as ue:
logger.error("Unexpected Error: malformed JSON in POST request, check key/value pair at: ")
logger.error(ue)
Source:
https://docstrings.wordpress.com/2014/04/19/flask-access-log-write-requests-to-file/
If link is broken:
You may be confused because adding a handler to Flask’s app.logger doesn’t catch the output you see in the console like:
127.0.0.1 - - [19/Apr/2014 18:51:26] "GET / HTTP/1.1" 200 -
This is because app.logger is for Flask and that output comes from the underlying WSGI module, Werkzeug.
To access Werkzeug’s logger we must call logging.getLogger() and give it the name Werkzeug uses. This allows us to log requests to an access log using the following:
logger = logging.getLogger('werkzeug')
handler = logging.FileHandler('access.log')
logger.addHandler(handler)
# Also add the handler to Flask's logger for cases
# where Werkzeug isn't used as the underlying WSGI server.
# This wasn't required in my case, but can be uncommented as needed
# app.logger.addHandler(handler)
You can of course add your own formatting and other handlers.

Flask has a built-in logger that can be accessed using app.logger. It is just an instance of the standard library logging.Logger class which means that you are able to use it as you normally would the basic logger. The documentation for it is here.
To get the built-in logger to write to a file, you have to add a logging.FileHandler to the logger. Setting debug=True in app.run, starts the development server, but does not change the log level to debug. As such, you'll need to set the log level to logging.DEBUG manually.
Example:
import logging
from flask import Flask
app = Flask(__name__)
handler = logging.FileHandler("test.log") # Create the file logger
app.logger.addHandler(handler) # Add it to the built-in logger
app.logger.setLevel(logging.DEBUG) # Set the log level to debug
#app.route("/")
def index():
app.logger.error("Something has gone very wrong")
app.logger.warning("You've been warned")
app.logger.info("Here's some info")
app.logger.debug("Meaningless debug information")
return "Hello World"
app.run(host="127.0.0.1", port=8080)
If you then look at the log file, it should have all 4 lines printed out in it and the console will also have the lines.

Related

Python logging use a single logger for entire project where name is defined by arguments from cmd

Hey I am trying to set up a logger for python3.9 for an entire project, with multiple files. I want to just define the logger in main.py using command line arguments to define log file name.
logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s', level='INFO')
logger = logging.getLogger(__name__)
def main():
file_name = sys.argv[1]
lookback_minutes = int(sys.argv[2])
file_handler = logging.FileHandler(f'log/{file_name}.log')
logger.addHandler(file_handler)
logger.info(f'Running processing chain for: {file_name}')
processing_chain.run(lookback_minutes)
processing_chain file:
import logging
def run(lookback_minutes):
logging.info(lookback_minutes)
Which works for main, I get the info statement printed to the log file. However I do not understand how to import it into the files that main calls. How do I bring the file handler into processing_chain file? Currently from what I could understand from other places on stackoverflow, I just import logging and then use logging.info or any other level and it should follow. But it does not log to file, just to console.

How to send errors to .log file in python pandas?

Even I searched in the google, did not find.
I am struggling with the code. Please help me.
My requirement is simple:
If any error occured, need to send to .log file for the below program.But below code is not capturing in .log file.
Please help me.
Thanks in advance.
import pandas as pd
import pyodbc
import time
import logging
df= pd.read_csv('Testing.csv')
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler("Archieve\\spam.log")
fh.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ch)
logger.addHandler(fh)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
Make sure you all the imports are properly resolved with installing respective dependencies.
Then make sure you have a file named "Testing.csv" in your current working directory or PYTHONPATH.
Finally make sure, you have folder named "Archieve" in your current working directory, if it is not there, create it first.
mkdir Archieve
Finally see the log file in Archieve folder.
cat Archieve\spam.log

Python 3.7 - Logger not showing any log on GCP LOG Viewer

I've made a decorator named "logger" and I'm using it to show log on the console.
In my python 2.7 everything works fine but in my python 3.7 code, it doesn't print any log in GCP log viewer. (My code is deployed on GCP standard environment)
I've tried to set this code but it doesn't work.
logging.getLogger().setLevel(logging.INFO)
Here's my simple logger decorator
def logger(func):
def wrapper(*args, **kwargs):
logging.info('INSIDE %s PARAMETERS: %s', func.__name__, args)
res = func(*args, **kwargs)
logging.info('OUTSIDE %s OUTPUT: %s', func.__name__, res)
return res
return wrapper
Even a simple logging.info statement is not working.
I've tried everything with logging, but nothing gets printed out on GCP Log Viewer.
logging.info("Text info")
logging.debug("Text debug")
logging.warning("Text warning")
logging.error("Text error")
logging.critical("Text critical")
Here is the log viewer
I deployed the following app to App Engine:
In app.yaml:
runtime: python37
In requirements.txt:
Flask==1.0.2
In main.py:
from flask import Flask
import logging
logging.getLogger().setLevel(logging.INFO)
app = Flask(__name__)
#app.route('/')
def hello():
logging.info("Text info")
logging.debug("Text debug")
logging.warning("Text warning")
logging.error("Text error")
logging.critical("Text critical")
return 'Hello World!'
When I look at the Stackdriver logs for the application after a request, I see the INFO, WARNING, ERROR and CRITICAL log messages as expected.
Is it possible you're looking somewhere else for the logs? From the App Engine Dashboard, you can go to
Services > [your service] > Diagnose > Tools > Logs
to find the logs for your service.

scrapy using python3 logging module issue

I use scrapy 1.1.0, and I have 5 spiders in the "spiders" folder.
In every spider, I try to use python3 logging module. And the code structure like this :
import other modules
import logging
class ExampleSpider(scrapy.Spider):
name = 'special'
def __init__(self):
# other initializations
# set log
self.log = logging.getLogger('special')
self.log.setLevel(logging.DEBUG)
logFormatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
# file handler
fileHandler = logging.FileHandler(LOG_PATH) # LOG_PATH has defined
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(logFormater)
self.log.addHandler(fileHandler)
# other functions
every spider has the same structure.When I run these spiders, I check the log file, they did exist, but their size are always 0 byte.
And the other question is that when I run one spider, it always generated two or more log files. Like I run a spider, and it will generate a.log and b.log.
Any answers would appreciate.
You can set log file via LOG_FILE setting in settings.py or via command line argument --logfile FILE, i.e. scrapy crawl myspider --logfile myspider.log
As described in the official docs

'gunicorn --paster' log messages not shown

I am starting gunicorn with the --paster option for running Pyramid.
gunicorn -w 1 --paster development.ini
gunicorn's own messages show up fine on console, for example
2014-02-20 22:38:50 [44201] [INFO] Starting gunicorn 18.0
2014-02-20 22:38:50 [44201] [INFO] Listening at: http://0.0.0.0:6543 (44201)
2014-02-20 22:38:50 [44201] [INFO] Using worker: sync
However the log messages in my Pyramid app are not showing up.
If I use pserve development.ini, which uses waitress as WSGI server, the log messages show up on console just fine.
My development.ini includes a pretty vanilla logging configuration section.
[loggers]
keys = root, apipython
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console
[logger_apipython]
level = DEBUG
handlers =
qualname = apipython
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
I am at lost why the logs are not showing up when I use gunicorn.
Do not use pserve with gunicorn, it is deprecated and most likely will be removed in some of the next versions.
Gunicorn has "logconfig" setting, just set it to your config via command line argument:
gunicorn -w 1 --paster development.ini --log-config development.ini
or in the same config:
[server:main]
use = egg:gunicorn#main
logconfig = %(here)s/development.ini
It happens because of "pserve" command not only starts server and loads application, it also setups logging. While "gunicorn --paster" just loads application. To fix it you can explicitly setup logging on your application:
from pyramid.config import Configurator
from pyramid.paster import setup_logging
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application. """
setup_logging(global_config['__file__'])
config = Configurator(settings=settings)
# Configure application
return config.make_wsgi_app()
Or as you pointed in comment, change server in config file and use "pserve" command:
[server:main]
use = egg:gunicorn#main

Resources