I want to read some values as a parameter using configparser in Databricks
i can import configparser module in databricks but unable to read the parameters from configfile its coming up error as KEY ERROR
please check the below screenshot
config file is
The problem is that your file is located on DBFS (the /FileStore/...) and this is file system isn't understood by configparser that works with "local" file system. To get this working, you need to append the /dbfs prefix to file path: /dbfs/FileStore/....
P.S. it may not work on community edition with DBR 7.x. In this case, just copy this config file before reading using the dbutils.fs.cp, like this :
dbutils.fs.cp("/FileStore/...", "file:///tmp/config.ini")
config.read("/tmp/config.ini")
Related
I need to change the output/working directory of the hydra config framework in such a way that it lies outside of my project directory. According to my understanding and the doc, config.yaml would need to look like this:
exp_nr: 0.0.0.0
condition: something
hydra:
run:
dir: /absolute/path/to/folder/${exp_nr}/${condition}/
In my code, I then tried to access and set the path like this:
import os
import hydra
from omegaconf import DictConfig
#hydra.main(config_path="../../config", config_name="config", version_base="1.3")
def main(cfg: DictConfig):
print(cfg)
cwd = os.getcwd()
print(f"The current working directory is {cwd}")
owd = hydra.utils.get_original_cwd()
print(f"The Hydra original working directory is {owd}")
work_dir = cfg.hydra.run.dir
print(f"The work directory should be {work_dir}")
But I get the following output and error:
{'exp_nr': '0.0.0.0', 'condition': 'something'}
The current working directory is /project/path/subdir/subsubdir
The Hydra original working directory is /project/path/subdir/subsubdir
Error executing job with overrides: ['exp_nr=1.0.0.0', 'condition=somethingelse']
Traceback (most recent call last):
File "/project/path/subdir/subsubdir/model.py", line 13, in main
work_dir = cfg.hydra.run.dir
omegaconf.errors.ConfigAttributeError: Key 'hydra' is not in struct
full_key: hydra
object_type=dict
I see that hydra.run.dir doesn't appear in the cfg dict printed first but how can I access the path through the config if os.getcwd() isn't set already? Or what did I do wrong?
The path is correct as I already saved files to the folder before integrating hydra and if the process isn't killed due to the error the folder also gets created but hydra doesn't save any files to it, not even the log file with the parameters it should save by default. I also tried to set the path relative to the standard output path or having an extra config parameter work_dir: ${hydra.run.dir} (returns an Interpolation error).
You can access the Hydra config via the HydraConfig singleton documented here.
from hydra.core.hydra_config import HydraConfig
#hydra.main()
def my_app(cfg: DictConfig) -> None:
print(HydraConfig.get().job.name)
I have requirement to refactor a K8s Python app so that it gets some configuration from a remote Giltab project because for various reasons we want to decouple applicaton settings from our pipeline/deployment environment.
In my functional testing, this works:
import configparser
config = configparser.ConfigParser()
config_file = "config.ini" # local file for testing
config.read(config_file)
['config.ini']
However, when I attempt to read the configuration from a remote file (our requirement), this DOES NOT work:
import requests
import os
token = os.environ.get('GITLAB_TOKEN')
headers = {'PRIVATE_TOKEN': token}
params = { 'ref' : 'master' }
response = requests.get('https:/path/to/corp/gitlab/file/raw', params=params,
headers=headers
config = configparser.ConfigParser()
configfile = response.content.decode('utf-8')
print(configfile) # this is good!
config.read(configfile) # this fails to load the contents into configparser
[]
I get an empty list. I can create a file and or print the contents of the configfile object from the requests.get call, and the ini data looks good. config.read() seems unable to load this as an object in memory and only seems to work by reading a file from disk. Seems like writing the contents of the requests.get to a local .ini file would defeat the whole purpose of using the remote configuration repo.
Is there a good way to read that configuration file from the remote and have configparser access it at container runtime?
I got this working with:
config.read_string(configfile)
I am trying to export a space from contentful using the Export/import tool and the option config json file.
What is the file path where the config file should be?
Contentful DevRel here. Welcome to Stack Overflow! 👋
I assume you're referring to the config.json file mentioned in the docs. The config file is optional in case you don't want to pass the command-line options. The CLI options will do the job just fine.
If you want to go with this config file you choose the file path and can run the command with the path to your config as follows:
contentful space export --config example-config.json
Hope that helps. :)
My app needs configs like = app.config['LDAP_BASE_DN'] = 'OU=users,dc=example,dc=org' I want to pass this configurations to a YAML file and then make the app use it. I can load the file using PyYAML or config_with_yaml the problem is i can't set the app to use it as configurations.
It should work based on "https://exploreflask.com/en/latest/configuration.html"
I load my config with cfg = config.load("/Users/pjose/Project/dev_maintenance/backend/config.yaml") then i set my app config to get the data from the yaml file app.config.from_object(cfg) and by calling app.config["LDAP_USERNAME"] it should set the config, but it does not work.
YAML file:
LDAP_USERNAME: 'CN=Hermes Conrad,ou=people,dc=planetexpress,dc=com'
I get this error:
File "/Users/pjose/Project/dev_maintenance/backend/dev_maintenance/__init__.py", line 32, in <module>
app.config["LDAP_USERNAME"]
KeyError: 'LDAP_USERNAME'
I finally could make this work
The problem was that i was not passing the value in the YAML file to the app.config["LDAP_USERNAME"] as there are no references of this subject on the documentation that i used and i though it would fetch the value just by declaring like that.
So a example on how you could use a YAML file to set you app configurations is:
config.yaml
SQLALCHEMY_DATABASE_URI: "sqlite://"
SQLALCHEMY_TRACK_MODIFICATIONS : False
Then to get the values you need to parse the yaml using the PyYAML lib or other:
data = yaml_loader.yaml("/Users/pjose/Project/dev_maintenance/backend/config.yaml")
app.config.from_object(data)
app.config['SQLALCHEMY_DATABASE_URI'] = data.setdefault('SQLALCHEMY_DATABASE_URI')
I am just getting started with spark and I am trying out examples in local mode...
I noticed that in some examples when creating the RDD the relative path to the file is used and in others the path starts with "file:///". The second option did not work for me at all - "Input path does not exist"
Can anyone explain what the difference is between using the file path and putting 'file:///' in front of it ?
I am using Spark 2.2 on Mac running in local mode
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("test")
sc = SparkContext(conf = conf)
#This will work providing the relative path
lines = sc.textFile("code/test.csv")
#This will not work
lines = sc.textFile("file:///code/test.csv")
sc.textFile("code/test.csv") means test.csv in /<hive.metastore.warehouse.dir>/code/test.csv on HDFS.
sc.textFile("hdfs:///<hive.metastore.warehouse.dir>/code/test.csv") is equal to above.
sc.textFile("file:///code/test.csv") means test.csv in /code/test.csv on local file system.