How to save log file into subdirectory using dictConfig configuration? - python-3.x

I have the following config in my Python 3.6 application:
config = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": ("[%(asctime)s] %(levelname)s - %(threadName)s - %(name)s - %(message)s")
},
},
"handlers": {
"default": {
"class": "logging.StreamHandler",
"formatter": "standard",
"level": DEFAULT_LOG_LEVEL,
},
"fh": {
"class": "logging.handlers.RotatingFileHandler",
"formatter": "standard",
"level": FILE_LOG_LEVEL,
"filename": "myfile.log",
"maxBytes": 1024*1000, # 1 MB
"backupCount": 20
}
},
"loggers": {
'': {
"handlers": ["default", "fh"],
'level': DEFAULT_LOG_LEVEL,
'propagate': True
}
}
}
This currently works; however, it saves all the .log files to the current working directory (where the executable is on Windows).
What I'd like to do is have it save this to the "logs" directory which is in the same level as the executable itself and already existing.
My thought was to modify the "filename" property int he log configuration, but I'm not sure how that impacts my script files that use:
logger = logging.getLogger(__name__)
Would that need to be modified as well to get it to log to the correct location?
Desired Structure:
app.exe
logs
myfile.log

My initial thought seems do what I need after testing. I simply put the directory name in front of the file name in the config's "filename" property value. No modification was needed to the script's logger line.

Related

Give directories nicknames in Visual Studio Code

I am currently using SSH-Remote add-on for VisualStudioCode and some directories are having numbered names due to docker usage, my question is:
"Can i somehow 'rename' them without actually making changes to directory name"
With this i would give folder a nickname or a alias shown only to me and not changing actual values on my VPS
As a workaround you can add those folders to a workspace in VS Code, save the workspace and then edit the .code-workspace file you just saved to add a new node "name" where you can add custom names to the folder without actually renaming them.
{
"folders": [
{
"path": "../path/to/your/directory1"
},
{
"path": "../path/to/your/directory2"
}
],
"settings": {}
}
from this to
{
"folders": [
{
"path": "../path/to/your/directory1",
"name": "Your Custom NickName1"
},
{
"path": "../path/to/your/directory2",
"name": "Your Custom NickName2"
}
],
"settings": {}
}

Python Logging - Configuring Rotating File Handler From Config file

I have the below config file which I am using to configure logging in my python modules , the issue with this is that it does not append the timestamp to the file-name , right now the log file created is file_handler.log but I want it to be file_handler-timestamp.log
{
"version": 1,
"disable_existing_loggers": true,
"formatters": {
"simple": {
"format": "%(asctime)s %(levelname)s %(filename)s %(lineno)s %(message)s",
"datefmt": "%Y-%m-%d %H:%M:%S"
}
},
"handlers": {
"file_handler": {
"level": "INFO",
"class": "logging.handlers.TimedRotatingFileHandler",
"formatter": "simple",
"filename": "error.log",
"backupCount": 10,
"interval" : 1,
"when": "s",
"encoding": "utf8"
}
},
"loggers": { },
"root": {
"handlers": [
"file_handler"
],
"level": "DEBUG"
}
}
Working code without config
import logging
import time
from logging.handlers import RotatingFileHandler
logger = logging.getLogger('log_1')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s', '%Y-%m-%d %H:%M:%S')
handler = RotatingFileHandler('log' + time.strftime('_%Y%m%d-%H%M') + '.log', maxBytes=2000, backupCount=10)
handler.setFormatter(formatter)
logger.addHandler(handler)
Workaround code which updated the config dynamically when program is initiated
import logging.config
import json
import time
fp = open("sample_config.json")
config = json.load(fp)
fp.close()
config['handlers']['file_handler']['filename'] += time.strftime('_%Y%m%d-%H%M') + '.log'
logging.config.dictConfig(config)
timestamp of when? When the file is rotated or when it is created?
If you use TimedRotatingFileHandler, then when the file is rotated, the timestamp is appended to the filename.
If you want the timestamp appended to the log filename, you have to subclass RotatingFileHandler and modify the __init__ method. Then point the "class" item of "file_handler" of the config file to your new class.
It is easy to do, but it may not be what you need.
Just let the timestamp be appended to the rotated log filename or if you need to limit the size too, read this answer

Is there a way to use another extension instead of .feature?

I use some files written with Gherkin mode, but they don't have the ".feature" extension. I tried to change some visual code files related to cucumber extension to be able to highlight files that are not .feature but I had no success.
For example:
Workspace settings.json:
{
"folders": [
{
"path": "/home/user/git"
},
{
"path": "/home/user/Documents/scripts"
}
],
"settings": {}
"cucumberautocomplete.steps": [
"*.myext"
],
"cucumberautocomplete.syncfeatures": "*.myext",
"cucumberautocomplete.strictGherkinCompletion": true
}
It worked changing the file /home/user/.config/Code/User/settings.json and adding this config:
"files.associations": {
"*.myext": "feature"
}
{
"editor.renderWhitespace": "all",
"window.titleBarStyle": "custom",
"editor.fontSize": 15,
"python.jediEnabled": false,
"terminal.integrated.shell.linux": "/bin/bash",
"workbench.colorTheme": "Visual Studio Dark",
"editor.suggestSelection": "first",
"vsintellicode.modify.editor.suggestSelection": "automaticallyOverrodeDefaultValue",
"[feature]": {
},
"files.associations": {
"*.myext": "feature"
}
}

Custom date in azure blob folder path

I have looked at some posts and documentation on how to specify custom folder paths while creating an azure blob (using the azure data factories).
Official documentation:
https://learn.microsoft.com/en-us/azure/data-factory/v1/data-factory-azure-blob-connector#using-partitionedBy-property
Forums posts:
https://dba.stackexchange.com/questions/180487/datafactory-tutorial-blob-does-not-exist
I am successfully able to put into date indexed folders, however what I am not able to do is put into incremented/decremented date folders.
I tried using $$Text.Format (like below) but it gives a compile error --> Text.Format is not a valid blob path .
"folderPath": "$$Text.Format('MyRoot/{0:yyyy/MM/dd}/', Date.AddDays(SliceEnd,-2))",
I tried using the PartitionedBy section (like below) but it too gives a compile error --> Only SliceStart and SliceEnd are valid options for "date"
{
"name": "MyBlob",
"properties": {
"published": false,
"type": "AzureBlob",
"linkedServiceName": "MyLinkedService",
"typeProperties": {
"fileName": "MyTsv.tsv",
"folderPath": "MyRoot/{Year}/{Month}/{Day}/",
"format": {
"type": "TextFormat",
"rowDelimiter": "\n",
"columnDelimiter": "\t",
"nullValue": ""
},
"partitionedBy": [
{
"name": "Year",
"value": {
"type": "DateTime",
"date": "Date.AddDays(SliceEnd,-2)",
"format": "yyyy"
}
},
{
"name": "Month",
"value": {
"type": "DateTime",
"date": "Date.AddDays(SliceEnd,-2)",
"format": "MM"
}
},
{
"name": "Day",
"value": {
"type": "DateTime",
"date": "Date.AddDays(SliceEnd,-2)",
"format": "dd"
}
}
]
},
"availability": {
"frequency": "Day",
"interval": 1
},
"external": false,
"policy": {}
}
Any pointers are appreciated!
EDIT for response from Adam:
I also used folder structure directly in FileName as per suggestion from Adam as per below forum post:
Windows Azure: How to create sub directory in a blob container
I used it like in below sample.
"typeProperties": {
"fileName": "$$Text.Format('{0:yyyy/MM/dd}/MyBlob.tsv', Date.AddDays(SliceEnd,-2))",
"folderPath": "MyRoot/",
"format": {
"type": "TextFormat",
"rowDelimiter": "\n",
"columnDelimiter": "\t",
"nullValue": ""
},
It gives no compile error and also no error during deployment. But it throws an error during execution!!
Runtime Error is ---> Error in Activity: ScopeJobManager:PrepareScopeScript, Unsupported unstructured stream format '.adddays(sliceend,-2))', can't convert to unstructured stream.
I think the problem is that FileName can be used to create folders but not dynamic folder names, only static ones.
you should create a blob using the following convention: "foldername/myfile.txt" , so you could also append additional blobs under that foldername. I'd recommend checking this thread: Windows Azure: How to create sub directory in a blob container , It may help you resolve this case.

Adding more than one path to logstash forwarder

I have added more than one path to logstash forwarder config file. But only the first one works. It is not sending logs for the second path.
My config file.
"network": {
"servers": [ "x.x.x.x:5000" ],
"ssl certificate": "./logstash-forwarder.crt",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/opt/app1/application.log"
],
"fields": { "type": "app1" }
},
{
"paths":[
"/opt/app2/application.log"
],
"fields":{"type": "app2"}
}
]
}
I know I can add them together in the first path block and it will work, but I want to add two different types, which I guess can't be done in one path block.
This is resolved. Actually the issue is that my second log file is not updated for more than a day. So, logstash-forwarder skipped it.
2015/08/07 04:55:14.029097 Skipping file (older than dead time of 24h0m0s):

Resources