My python script produces a log file with lines of logging when I run it from Spyder, but when I run it outside of Spyder (e.g. in the command window, or as an executable), the log file is produced but remains empty.
I confirmed that I'm using full file paths to specify the log file.
I understand that using basicConfig doesn't always work as expected, so I followed the answer provided here. A simplified version of my script is shown below:
# Example inputs
workdir = 'c:\Users\xx\work'
log_file = 'log_2.txt'
# Initialize logging functionality
fileh = logging.FileHandler(os.path.join(workdir, log_file), 'w')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fileh.setFormatter(formatter)
fileh.setLevel(logging.DEBUG)
log = logging.getLogger() # root logger
for hdlr in log.handlers[:]: # remove all old handlers
log.removeHandler(hdlr)
log.addHandler(fileh) # set the new handler
# Rest of code including log messages (removed rest of script here, only leaving logging-related text)
logging.info('Here is a log message.')
logging.debug('Here is another log message.')
# At the end of the script
logging.shutdown()
Most related questions are when someone can't find the log file that's being written. However, in this case, the log file is created but not written to.
Can someone advise on how this code needs to be modified to also work outside of Spyder?
Related
Hey I am trying to set up a logger for python3.9 for an entire project, with multiple files. I want to just define the logger in main.py using command line arguments to define log file name.
logging.basicConfig(format='%(asctime)s %(levelname)-8s %(message)s', level='INFO')
logger = logging.getLogger(__name__)
def main():
file_name = sys.argv[1]
lookback_minutes = int(sys.argv[2])
file_handler = logging.FileHandler(f'log/{file_name}.log')
logger.addHandler(file_handler)
logger.info(f'Running processing chain for: {file_name}')
processing_chain.run(lookback_minutes)
processing_chain file:
import logging
def run(lookback_minutes):
logging.info(lookback_minutes)
Which works for main, I get the info statement printed to the log file. However I do not understand how to import it into the files that main calls. How do I bring the file handler into processing_chain file? Currently from what I could understand from other places on stackoverflow, I just import logging and then use logging.info or any other level and it should follow. But it does not log to file, just to console.
I'm using python 3.7.3, with flask version 1.0.2.
When running my app.py file without the following imports:
import logging
logging.basicConfig(filename='api.log',level=logging.DEBUG)
Flask will display relevant debug information to console, such as POST/GET requests and which IP they came from.
As soon as DEBUG logging is enabled, I no longer receive this output. I have tried running my application in debug mode:
app.run(host='0.0.0.0', port=80, debug=True)
But this produces the same results. Is there a way to have both console output, and python logging enabled? This might sound like a silly request, but I would like to use the console for demonstration purposes, while having the log file present for troubleshooting.
Found a solution:
import logging
from flask import Flask
app = Flask(__name__)
logger = logging.getLogger('werkzeug') # grabs underlying WSGI logger
handler = logging.FileHandler('test.log') # creates handler for the log file
logger.addHandler(handler) # adds handler to the werkzeug WSGI logger
#app.route("/")
def index():
logger.info("Here's some info")
return "Hello World"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=80)
Other Examples:
# logs to console, and log file
logger.info("Some text for console and log file")
# prints exception, and logs to file
except Exception as ue:
logger.error("Unexpected Error: malformed JSON in POST request, check key/value pair at: ")
logger.error(ue)
Source:
https://docstrings.wordpress.com/2014/04/19/flask-access-log-write-requests-to-file/
If link is broken:
You may be confused because adding a handler to Flask’s app.logger doesn’t catch the output you see in the console like:
127.0.0.1 - - [19/Apr/2014 18:51:26] "GET / HTTP/1.1" 200 -
This is because app.logger is for Flask and that output comes from the underlying WSGI module, Werkzeug.
To access Werkzeug’s logger we must call logging.getLogger() and give it the name Werkzeug uses. This allows us to log requests to an access log using the following:
logger = logging.getLogger('werkzeug')
handler = logging.FileHandler('access.log')
logger.addHandler(handler)
# Also add the handler to Flask's logger for cases
# where Werkzeug isn't used as the underlying WSGI server.
# This wasn't required in my case, but can be uncommented as needed
# app.logger.addHandler(handler)
You can of course add your own formatting and other handlers.
Flask has a built-in logger that can be accessed using app.logger. It is just an instance of the standard library logging.Logger class which means that you are able to use it as you normally would the basic logger. The documentation for it is here.
To get the built-in logger to write to a file, you have to add a logging.FileHandler to the logger. Setting debug=True in app.run, starts the development server, but does not change the log level to debug. As such, you'll need to set the log level to logging.DEBUG manually.
Example:
import logging
from flask import Flask
app = Flask(__name__)
handler = logging.FileHandler("test.log") # Create the file logger
app.logger.addHandler(handler) # Add it to the built-in logger
app.logger.setLevel(logging.DEBUG) # Set the log level to debug
#app.route("/")
def index():
app.logger.error("Something has gone very wrong")
app.logger.warning("You've been warned")
app.logger.info("Here's some info")
app.logger.debug("Meaningless debug information")
return "Hello World"
app.run(host="127.0.0.1", port=8080)
If you then look at the log file, it should have all 4 lines printed out in it and the console will also have the lines.
Even I searched in the google, did not find.
I am struggling with the code. Please help me.
My requirement is simple:
If any error occured, need to send to .log file for the below program.But below code is not capturing in .log file.
Please help me.
Thanks in advance.
import pandas as pd
import pyodbc
import time
import logging
df= pd.read_csv('Testing.csv')
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create file handler which logs even debug messages
fh = logging.FileHandler("Archieve\\spam.log")
fh.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ch)
logger.addHandler(fh)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
Make sure you all the imports are properly resolved with installing respective dependencies.
Then make sure you have a file named "Testing.csv" in your current working directory or PYTHONPATH.
Finally make sure, you have folder named "Archieve" in your current working directory, if it is not there, create it first.
mkdir Archieve
Finally see the log file in Archieve folder.
cat Archieve\spam.log
I have a Python app that works locally, but on the server something goes wrong. So I have to do some debugging in an SSH session. The server logs tell me that something goes wrong in utils/do_something.py. I have created utils/log (rights 777), where the debugging values are supposed to go.
do_something.py looks like this:
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
def do_something(arg1, arg2):
print('XXX')
f = open(log_file, 'a')
f.write('XXX\n')
stuff = goes.wrong
Loading the page causes do_something to run (as confirmed by the error logs). Locally the expected XXX appears in the console and in the log file. But nothing happens on the server.
I created a second file utils/blah.py:
print('BLAH')
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
f = open(log_file, 'a')
f.write('BLAH\n')
When I run it with python blah.py, the expected BLAH appears in the console and the logfile.
I don't care that much about the difference between local and production server.
But I would like to understand the difference between do_something.py and blah.py.
Is there a better way to debug in an SSH session?
I work in a virtualenv in a mod_wsgi 4.6.5/Python3.7 environment on a Webfaction server. Some details about it can be seen in this question on the Webfaction forum.
Edit 1: On the server print seems to be discouraged anyway.
(See Where do things go when I ‘print’ them from my Django app?)
But what matters to me is f.write(). I just added print for comparison.
Edit 2: It is the same when I use the logging module. It works when I run blah.py, but nothing happens when loading the page runs do_something.
Edit 3: I tried the same with a simpler app, and the result is the same.I added the logging in views.py:
from django.http import HttpResponse
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
def home_view(request):
f = open(log_file, 'a')
f.write('XXX\n')
return HttpResponse("Hello from home view.")
Locally this writes XXX to the log file every time the page is loaded. But not on the server.There are no errors in the server log.
Using the logging module: I am not sure why that did not work, but now it does.
import os
import logging
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
logging.basicConfig(filename=log_file, level=logging.DEBUG)
logging.debug('This works.')
Possibly I have used it with filename='log' instead of filename=log_file.
Locally the former creates the log file in the root folder. But on the server it must already exist.
Writing rights: It seems worth mentioning that touch log gave me a file I could not wright to, and lacking sudo I could not use chmod. So I used the trick install -b -m 777 /dev/null log.
It appears that you never explicitly close the file after writing, so the output is probably buffered. Your server and your local machine may have different settings with respect to file buffering, which would explain the differences you're experiencing.
If you want to open, write, and close a file, python's context managers are the best way to do it:
def home_view(request):
with open(log_file, 'a') as f:
f.write('XXX\n')
return HttpResponse("Hello from home view.")
You might check the privileges on the utils folder, and on any parent folders. Many times an http server will be running as user 'nobody', which has practically zip privileges. So unless the utils folder itself also has 0777 privileges, that could create a problem. You also might want to put the f.write in a try ... except block so that you can catch the error specifically and create a more useful / informative message about the error. Best.
I use scrapy 1.1.0, and I have 5 spiders in the "spiders" folder.
In every spider, I try to use python3 logging module. And the code structure like this :
import other modules
import logging
class ExampleSpider(scrapy.Spider):
name = 'special'
def __init__(self):
# other initializations
# set log
self.log = logging.getLogger('special')
self.log.setLevel(logging.DEBUG)
logFormatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
# file handler
fileHandler = logging.FileHandler(LOG_PATH) # LOG_PATH has defined
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(logFormater)
self.log.addHandler(fileHandler)
# other functions
every spider has the same structure.When I run these spiders, I check the log file, they did exist, but their size are always 0 byte.
And the other question is that when I run one spider, it always generated two or more log files. Like I run a spider, and it will generate a.log and b.log.
Any answers would appreciate.
You can set log file via LOG_FILE setting in settings.py or via command line argument --logfile FILE, i.e. scrapy crawl myspider --logfile myspider.log
As described in the official docs