I have a crontab entry which is running a python3 script. This python script uses a config.ini file to get some tokens for use in the script.
The crontab entry is:
*/15 * * * * /usr/bin/python3 /opt/scripts/tf_state_backup/tf_state_backup.py >> ~/cron.out 2>&1
The config.ini file has the following:
[terraform]
token = <base64 encoded API key>
[gitlab]
token = <base64 encoded API key>
The relevant part of the python script is as follows:
import configparser
## read config file and decode api keys
config = configparser.ConfigParser()
config.read(os.path.abspath('config.ini'))
tfc_token = base64.b64decode(config['terraform']['token']).decode('utf-8')
gitlab_token = base64.b64decode(config['gitlab']['token']).decode('utf-8')
When this runs I can check the cron.out file for any errors. I get the following error every time it runs.
SyntaxError: invalid syntax
Traceback (most recent call last):
File "/opt/scripts/tf_state_backup/tf_state_backup.py", line 17, in <module>
tfc_token = base64.b64decode(config['terraform']['token']).decode('utf-8')
File "/usr/lib64/python3.6/configparser.py", line 959, in __getitem__
raise KeyError(key)
KeyError: 'terraform'
I have checked the following:
Ensured script and config has correct permissions & +x permission
Ran script exactly as it is in the cron tab, it runs fine without any problems
Ensured the config.ini was being referenced by it's absolute path, not relative path
Any help on this would be excellent.
Your should use the get method of ConfigParser object. The first parameter is the section name the second is the variable name. If the raw parameter is set to True then special characters will be read as string (Eg.: %).
I have written a working version.
test.ini:
[terraform]
token = aGVsbG93b3JsZA==
[gitlab]
token = bm90X2hlbGxvd29ybGQ=
test.py:
import configparser
import base64
config = configparser.ConfigParser()
config.read("test.ini")
tfc_token = base64.b64decode(config.get('terraform', 'token', raw=True)).decode('utf-8')
gitlab_token = base64.b64decode(config.get('gitlab', 'token', raw=True)).decode('utf-8')
print(tfc_token)
print(gitlab_token)
Output:
>>> python3 test.py
helloworld
not_helloworld
FYI:
I have used Python3.6.6 and Linux OS for testing.
I have generated the base64s on this site: https://www.base64encode.org/
I managed to get this running by updating the cron job to the following:
*/15 * * * * cd /opt/scripts/tf_state_backup/ && /usr/bin/python3 /opt/scripts/tf_state_backup/tf_state_backup.py
I am perhaps not getting the path correctly for the config file?
Regardless it is now working.
Related
I need to change the output/working directory of the hydra config framework in such a way that it lies outside of my project directory. According to my understanding and the doc, config.yaml would need to look like this:
exp_nr: 0.0.0.0
condition: something
hydra:
run:
dir: /absolute/path/to/folder/${exp_nr}/${condition}/
In my code, I then tried to access and set the path like this:
import os
import hydra
from omegaconf import DictConfig
#hydra.main(config_path="../../config", config_name="config", version_base="1.3")
def main(cfg: DictConfig):
print(cfg)
cwd = os.getcwd()
print(f"The current working directory is {cwd}")
owd = hydra.utils.get_original_cwd()
print(f"The Hydra original working directory is {owd}")
work_dir = cfg.hydra.run.dir
print(f"The work directory should be {work_dir}")
But I get the following output and error:
{'exp_nr': '0.0.0.0', 'condition': 'something'}
The current working directory is /project/path/subdir/subsubdir
The Hydra original working directory is /project/path/subdir/subsubdir
Error executing job with overrides: ['exp_nr=1.0.0.0', 'condition=somethingelse']
Traceback (most recent call last):
File "/project/path/subdir/subsubdir/model.py", line 13, in main
work_dir = cfg.hydra.run.dir
omegaconf.errors.ConfigAttributeError: Key 'hydra' is not in struct
full_key: hydra
object_type=dict
I see that hydra.run.dir doesn't appear in the cfg dict printed first but how can I access the path through the config if os.getcwd() isn't set already? Or what did I do wrong?
The path is correct as I already saved files to the folder before integrating hydra and if the process isn't killed due to the error the folder also gets created but hydra doesn't save any files to it, not even the log file with the parameters it should save by default. I also tried to set the path relative to the standard output path or having an extra config parameter work_dir: ${hydra.run.dir} (returns an Interpolation error).
You can access the Hydra config via the HydraConfig singleton documented here.
from hydra.core.hydra_config import HydraConfig
#hydra.main()
def my_app(cfg: DictConfig) -> None:
print(HydraConfig.get().job.name)
I am trying to call python script using Jenkins. I am able to execute the python script over command promt successfully. However when same script is called over Jenkins, I am facing problem of dll. My script is as follow
import isystem.connect as ic
import time
import os
print('isystem.connect version: ' + ic.getModuleVersion())
# 1. connect to winIDEA Application
pathTowinIDEA = 'C:/winIDEA/iConnect.dll'
cmgr_APPL = ic.ConnectionMgr(pathTowinIDEA)
cmgr_APPL.connectMRU('U:/winIDEA/Winidea_cust_config.xjrf.xjrf')
debug_APPL = ic.CDebugFacade(cmgr_APPL)
ec = ic.CExecutionController(cmgr_APPL)
print('APPL Configuration launched')
Error is appearing at line 9:
cmgr_APPL.connectMRU('U:/winIDEA/Winidea_cust_config.xjrf.xjrf')
Error message in Jenkins:
E:6 Can not load iConnect.dll: 'C:/winIDEA/iConnect64.dll'
SystemError: 'The specified module could not be found.
I have verified that iConnect64.dll is present at this path. How to resolve it ?
I'm totally lost here. I'm trying to create a scheduler to run python script on my Mac, but I'm getting the following error:
Traceback (most recent call last):
File "/Users/Root/Desktop/Project/Data/script.py", line 148, in <module>
run(
File "/Users/Root/Desktop/Project/Data/script.py", line 121, in run
config = get_config("config")
File "/Users/Root/Desktop/Project/Data/config/__init__.py", line 3, in get_config
with open(f"config/{config_type}.json", "r") as file:
FileNotFoundError: [Errno 2] No such file or directory: 'config/config.json'
So crontab convinces me that there is no such file or a directory, which is not true. I can run my script manually without errors.
My crontab is:
00 19 21 1-12 * /Library/Frameworks/Python.framework/Versions/3.9/bin/python3/ /Users/Root/Desktop/Project/Data/script.py >> /Users/Root/Desktop/Project/Data/cron.txt 2>&1
What am I doing wrong?
I'd be grateful for any help!
And is this possible without changing the relative path to an absolute path? I am aware of this solution
I assume crontab's cwd (Current Working Directory) is not same as where the script is stored.
this would solve your problem:
import os
script_dir = os.path.dirname(os.path.realpath(__file__))
os.chdir(script_dir)
You can get the directory where you script is by calling "os.path.dirname(os.path.realpath(file))"
if you change the current working directory "os.chdir(...dir...)" you can access you config/config.json by relative path,
Otherwise you will have to use a absolute path
Try running this and check your output file:
import os
script_dir = os.path.dirname(os.path.realpath(__file__))
print (os.getcwd())
print(script_dir)
os.chdir(script_dir)
print (os.getcwd())
Try to open with full path to the json file.
the important part of the error message:
I am getting the following error
starting uWSGI 2.0.18
setting pythonHome to /var/www/demo/venv
python version :3.5.3
Fatal Python error :unable to get the locale encoding
import error : no module named 'encodings'
It shows python version :3.5.3
however inside my venv/lib folder , there is only one package python 2.7
does this have something to do with my error?
please help me out with this.
this is my demo_uwsgi.ini file
#application's base folder
base = /var/www/demo
#python module to import
app = flaskfile //flaskfile is my flask file
module = %(app)
home = %(base)/venv
pythonpath = %(base)
#socket file's location
socket = /var/www/demo/%n.sock
#permissions for the socket file
chmod-socket = 666
#the variable that holds a flask application inside the module imported at line #6
callable = app
#location of log files
logto = /var/log/uwsgi/%n.log```
Am I missing plugins or something? I added plugins = python32 in my demo_uwsgi.ini file and it shows no such file or directory. Do I need to change or unset python path or something?
figured it out myself. Delete the default Nginx configuration file and add your new configuration file at the /etc/nginx. Then follow the instructions in this link https://vladikk.com/20.13/09/12/serving-flask-with-nginx-on-ubuntu/ step by step. change the ownership from root to user. It works perfectly
I have the following code;
#!/usr/bin/env python3
import yaml
with open('config.yml', 'r') as config_file:
config = yaml.load(config_file)
The file is called __init__.py which is in the directory ~/bin/myprogram/myprogram/ and in the same directory, I have a file called config.yml
My symlink is as follows;
user$ ls -la /usr/local/bin/
lrwxr-xr-x 1 user admin 55 27 Nov 13:25 myprogram -> /Users/user/bin/myprogram/myprogram/__init__.py
Every time I run myprogram, I get the error FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'. I believe this is because the config.yml is not in /usr/local/bin/. What is the best way to work around this issue?
You can use __file__ to access the location of the __init__.py file when executing code in that file. It returns the full path, but care has to be taken as it may be the .pyc (or .pyo) version. Since you are using Python3 I would use the pathlib module:
import yaml
from pathlib import Path
my_path = Path(__file__).resolve() # resolve to get rid of any symlinks
config_path = my_path.parent / 'config.yaml'
with config_path.open() as config_file:
config = yaml.safe_load(config_file)
Please note:
If you have to use PyYAML, use safe_load(), even PyYAML's own documentation indicates .load() can be unsafe. It almost never necessary to use that. And in the unlikely event that safe_load() cannot load your config, e.g. if it has !!python/... tags, you should explicitly add register the classes that you actually need to the SafeLoader).
Since September 2006 the recommended extension for YAML files has been .yaml