I have written code to add logs using logging module in python. I tried running code through Sonarqube, It is showing following error .
Make sure that this logger's configuration is safe.
python code:
from logging.config import fileConfig
import logging
#this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
logger = logging.getLogger("alembic.env")
class DefaultConfig:
DEVELOPMENT = False
DEBUG = False
TESTING = False
LOGGING_LEVEL = "DEBUG"
CSRF_ENABLED = True
Please help to resolve this hotspot. And one more question I have, Is it mandatory to look into low priority hotspots.
Related
My python script produces a log file with lines of logging when I run it from Spyder, but when I run it outside of Spyder (e.g. in the command window, or as an executable), the log file is produced but remains empty.
I confirmed that I'm using full file paths to specify the log file.
I understand that using basicConfig doesn't always work as expected, so I followed the answer provided here. A simplified version of my script is shown below:
# Example inputs
workdir = 'c:\Users\xx\work'
log_file = 'log_2.txt'
# Initialize logging functionality
fileh = logging.FileHandler(os.path.join(workdir, log_file), 'w')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fileh.setFormatter(formatter)
fileh.setLevel(logging.DEBUG)
log = logging.getLogger() # root logger
for hdlr in log.handlers[:]: # remove all old handlers
log.removeHandler(hdlr)
log.addHandler(fileh) # set the new handler
# Rest of code including log messages (removed rest of script here, only leaving logging-related text)
logging.info('Here is a log message.')
logging.debug('Here is another log message.')
# At the end of the script
logging.shutdown()
Most related questions are when someone can't find the log file that's being written. However, in this case, the log file is created but not written to.
Can someone advise on how this code needs to be modified to also work outside of Spyder?
I have requirement to refactor a K8s Python app so that it gets some configuration from a remote Giltab project because for various reasons we want to decouple applicaton settings from our pipeline/deployment environment.
In my functional testing, this works:
import configparser
config = configparser.ConfigParser()
config_file = "config.ini" # local file for testing
config.read(config_file)
['config.ini']
However, when I attempt to read the configuration from a remote file (our requirement), this DOES NOT work:
import requests
import os
token = os.environ.get('GITLAB_TOKEN')
headers = {'PRIVATE_TOKEN': token}
params = { 'ref' : 'master' }
response = requests.get('https:/path/to/corp/gitlab/file/raw', params=params,
headers=headers
config = configparser.ConfigParser()
configfile = response.content.decode('utf-8')
print(configfile) # this is good!
config.read(configfile) # this fails to load the contents into configparser
[]
I get an empty list. I can create a file and or print the contents of the configfile object from the requests.get call, and the ini data looks good. config.read() seems unable to load this as an object in memory and only seems to work by reading a file from disk. Seems like writing the contents of the requests.get to a local .ini file would defeat the whole purpose of using the remote configuration repo.
Is there a good way to read that configuration file from the remote and have configparser access it at container runtime?
I got this working with:
config.read_string(configfile)
We are planning to rotate the log that is generated by Tomcat using Logrotate for volume maintenance. When I checked for the logs I was able to find two places in which these logs were been generated "../apache-tomcat-7.0.57/logs" and in the path that is specified in the "logging.properties". I did check in the Tomcat document, from which I was able to understand that Tomcat uses the default path which is "/logs" is no path is mentioned externally in "logging.properties". I was not able to find if I have missed any configuration.
logging.properties file:
handlers = 1catalina.org.apache.juli.FileHandler, 2localhost.org.apache.juli.FileHandler, 3manager.org.apache.juli.FileHandler, 4host-manager.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = <custome path>
1catalina.org.apache.juli.FileHandler.prefix = catalina.
2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = <custome path>
2localhost.org.apache.juli.FileHandler.prefix = localhost.
3manager.org.apache.juli.FileHandler.level = FINE
3manager.org.apache.juli.FileHandler.directory = <custome path>
3manager.org.apache.juli.FileHandler.prefix = manager.
4host-manager.org.apache.juli.FileHandler.level = FINE
4host-manager.org.apache.juli.FileHandler.directory = <custome path>
4host-manager.org.apache.juli.FileHandler.prefix = host-manager.
java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager].handlers = 3manager.org.apache.juli.FileHandler
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager].handlers = 4host-manager.org.apache.juli.FileHandler
# For example, set the org.apache.catalina.util.LifecycleBase logger to log
# each component that extends LifecycleBase changing state:
#org.apache.catalina.util.LifecycleBase.level = FINE
# To see debug messages in TldLocationsCache, uncomment the following line:
#org.apache.jasper.compiler.TldLocationsCache.level = FINE
My question is why are the logs getting generated in multiple places and how to make it log in just in one directory for maintaining the same ?
Reference link
https://tomcat.apache.org/tomcat-7.0-doc/logging.html
By default - It'll log to ${catalina.base}/logs which is what you should see in ${catalina.base}/conf/logging.properties
Additionally standard output (aka exception.printStackTrace()) goes into (by default) ${catalina.base}/logs/catalina.out
${catalina.base}/logs/catalina.out can be set to a different file by setting the environment variable CATALINA_OUT or CATALINA_OUT_CMD. So see what CATALINA_OUT_CMD does - It'll be easier to read the comments in ${catalina.home}/bin/catalina.sh
I want to use two databases: Postgres and Snowflake using alembic migration tool in a single fastapi app.
Able to perform alembic migrations and alembic upgrade if using single database i.e. Postgres but on using multiple databases alembic is creating problems.
Tried using snowflake database independently on a different app with only snowflake as a database. It is working perfectly fine with revisions and alembic upgrade but not working on a single app with two databases.
Here is my directory structure as suggested in few articles:
main-project-->
postgres-migration-->
versions/
env.py
README
script.py.mako
snowflake-migration-->
versions/
env.py
README
script.py.mako
Here is my alembic.ini file generated for postgres, but I manipulated it to support snowflake:
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = db-migration
[A_SNOWFLAKE_SCHEMA]
# path to env.py and migration scripts for schema1
script_location = snowflake-db-migration
# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date
# within the migration file as well as the filename.
# string value is passed to dateutil.tz.gettz()
# leave blank for localtime
# timezone =
# max length of characters to apply to the
# "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; this defaults
# to db-migration/versions. When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat db-migration/versions
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
#sqlalchemy.url = driver://user:pass#localhost/dbname
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
Above file is able to create revision scripts inside versions/ folder of snowflake-db-migration. But not able to run the generated revision using(throwing errors):
alembic upgrade head
How to convert this .ini file to support multi-database .ini file?
Tried using command:
alembic init --template multidb snowflake-db-migration
But not sure how to integrate changes here.
Also don't want to hardcode sqlalchemy.url variable in .ini file as required to use snowflake variables from a .toml file which contains all environment variables.
There are few answers but none of them are accepted and exactly similar to my use case.
I use scrapy 1.1.0, and I have 5 spiders in the "spiders" folder.
In every spider, I try to use python3 logging module. And the code structure like this :
import other modules
import logging
class ExampleSpider(scrapy.Spider):
name = 'special'
def __init__(self):
# other initializations
# set log
self.log = logging.getLogger('special')
self.log.setLevel(logging.DEBUG)
logFormatter = logging.Formatter('%(asctime)s %(levelname)s: %(message)s')
# file handler
fileHandler = logging.FileHandler(LOG_PATH) # LOG_PATH has defined
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(logFormater)
self.log.addHandler(fileHandler)
# other functions
every spider has the same structure.When I run these spiders, I check the log file, they did exist, but their size are always 0 byte.
And the other question is that when I run one spider, it always generated two or more log files. Like I run a spider, and it will generate a.log and b.log.
Any answers would appreciate.
You can set log file via LOG_FILE setting in settings.py or via command line argument --logfile FILE, i.e. scrapy crawl myspider --logfile myspider.log
As described in the official docs