Iterating through yml file and storing the values in variable - python-3.x

I am trying to iterate through yml files which have data in this format
this file name is artifact.yml
artifacts:
- name: feture1
version: 2.0.5
git_url: git#somethign:tmm/tmm.git
- name: feture2
version: 1.0
git_url: git#123.git
My end gole is to fetch the version and name from this file and store it in a variable for further use.
i have tried using the yml module (snap of code i am trying)
import yaml
op = open('artifact.yml')
myyml = yaml.safe_load(op)
for i in myyml['artifacts']:
print (i)
op.close()
the output i am getting is this.
{'name': 'feture1', 'version': '2.0.5', 'git_url': 'git#somethign:tmm/tmm.git'}
{'name': 'feture2', 'version': 1.0, 'git_url': 'git#123.git'}
but i am not sure how to separate out the below two dictionaries and store the name and version in a separate variable
also i have tried something like this
myyml['artifacts'][0])
the output is
{'name': 'feture1', 'version': '2.0.5', 'git_url': 'git#somethign:tmm/tmm.git'}
but the problem is i will not be aware that how many elements will be there, as i parse it so also need some way to make this effective regardless of the number of entries in the yml file (FYI, the yml source format is same what i have given on top it is just that the number of entries could be more or less)
Thanks .

using myyml['artifacts'][0]) is giving you a map
then all you have to do is giving it a key and get the desired value
myyml['artifacts'][0]["name"])
myyml['artifacts'][0]["version"])

Related

Save big lists to csv/pickle files

I'm learning to use pandas to save data to csv and pickle files, using the following script:
data = {'Product': [['Desktop Computer' * 30]],
'Price': [['850' * 30]]
}
df = pd.DataFrame(data, columns= ['Product', 'Price'])
df.to_csv('sample_csv.csv')
df.to_pickle('sample_pickle.pkl')
The csv file could be saved correctly, but the pickle file had some trash in it. Please see the attached pictures "correct_small_csv.png" and "pickle_withtrash.png".
Another thing I found is if the list size in data increases from 30 to 3000. The saved csv file would be messed up also. Basically the list of the 3000 'Desktop Computer' will be saved in two cells in the csv file. Please see the picture "Messed_big_csv.png".
If you want to have a list of strings do this 'Price': [['10'] * 30] instead of this 'Price': [['10' * 30]]. In other case you will have a list with one string element like ['10101010101....'] instead of ['10', '10', ....]
I've tried your solution with saving and it worked well for me for 30 and 3k items. Maybe there is a glitch or something like this? For sure everything is fine with code so I suggest there's something with an excel file visualization. I'm not sure but check the max length of a cell in csv/excel. Maybe it can't store 3k of 'Desktop Computer' in one cell.
Moreover pickle is a binary file so maybe you have no need to open it as text file.
Try to read both in pandas and see what will happen (everything should be just fine)

Does pytest provide a way of automatically reading properties for a test from a yml or conf file?

I'm using Python 3.8 and pytest for unit testing. I have tests similar to the following
def test_insert_record(az_sql_db_inst):
    """."""
    foreign_key_id = 10
I don't like hard-coding "10" in the file, and would prefer this be stored in some external configuration file where other tests can also access the value. Does pytest provide a way to do this (to load the values from a properties or yml file)? I could just open a file on my own, e.g.
def test_insert_record(az_sql_db_inst):
    """."""
file = open('test.txt')
for line in file:
fields = line.strip().split()
foreign_key_id = fields[0]
but it seems like there is something that can automatically do that for me.
If you simply want to initialize some constants, a separated config.py would do the trick.
from config import foreign_key_id
And PyYAML package would be good if you prefer general-purpose text files.
However, if you want to specify expected test results or even different results depending on scenarios, #pytest.mark.parametrize may be a good option.

How do I consume files in Django 2.2 for yaml parsing?

I'm trying to upgrade my site from Django 1.11 to Django 2.2, and I'm having trouble with uploading and parsing yaml files.
The error message is:
ScannerError : mapping values are not allowed here in "", line 1, column 34: b'---\n recipeVersion: 9\n name: value\n'
^
I'm getting the file contents using a ModelForm with a widget defined as:
'source': AsTextFileInput()
... using ...
class AsTextFileInput(forms.widgets.FileInput):
def value_from_datadict(self, data, files, name):
return files.get(name).read()
... and then I get the source variable to parse with:
cleaned_data = super(RecipeForm, self).clean()
source = cleaned_data.get("source")
From that error message above, it looks like my newlines are being escaped, so yaml sees the text all on a single line. I tried logging the source of this file, and here's how it shows in my log file:
DEBUG b'---\n recipeVersion: 9\n name: value\n'
So, how can I get this file content without (what looks to me like) escaped newlines so I can parse it as yaml?
Edit: my code and yaml (simplified for this question) have not changed; upgrading Python projects has broken the parsing.
Decoding the bytestring fixed it:
class AsTextFileInput(forms.widgets.FileInput):
def value_from_datadict(self, data, files, name):
return files.get(name).read().**decode('utf-8')**

How do I read Windows Registry file to check for values? [Python]

I am trying to perform auditing checks on Windows Registry file (.reg file, offline) and I am hoping that I can utilize Python to perform a check on the reg file.
For example (pseudo code):
#Configure registry policy processing: Do not apply during periodic background processing
testloc = "C:\\Users\\test.reg"
datafile = open(testloc, "r")
read = datafile.read()
find(Software\Policies\Microsoft\Windows\Group Policy\{35378EAC-683F-11D2-A89A-00C04FBBCFA2})
check(NoBackgroundPolicy) #check if dword value is correct
if(dword == correct):
print("correct")
else:
print("wrong")
I have tried looking at _winreg but it seems like it does a check on a live system using Windows API. Another issue is the large .reg file size(~200MB).
How can I perform such a check using Python?
I don't know if there's a lib that can read .reg files specifically, but from what it looks like it's just an INI file with an extra version information at the top.
Here's an example of how you could use the configparser module for that. Some notes:
.reg files are encoded in utf16
Before giving the file to the ConfigParser, a readline will skip the version info (something like Windows Registry Editor Version 5.00). Otherwise it will result in a MissingSectionHeaderError.
The names of values include quotation marks, which means you need to add them explicitly when looking up the value in the key.
import configparser
testloc = "C:\\Users\\test.reg"
regdata = configparser.ConfigParser()
with open(testloc, "r", encoding="utf-16") as f:
f.readline() # skip version info in first line
regdata.read_file(f)
key = regdata[r"HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Group Policy\{35378EAC-683F-11D2-A89A-00C04FBBCFA2}"]
value = key['"NoBackgroundPolicy"']
print(value)
There may be drawbacks of doing it this way though, for example in how the values you obtain are formatted.

How to make a config file that stores global variables but can also be easily edited from the level of the code?

I have a Python project that uses some global variables. The project has multiple .py files and all of them need to be able to read those variables easily. All of them should also be able to edit them.
Right now I am using a .py file that stores the variables and I simply import them in other files as a module. This solution is far from optimal, though, because I can't edit them easily. I naively tried to write setters for the variables, but obviously that doesn't work either.
I thought about using JSON for the variables, but then reading them wouldn't be as easy as 'import cfg'.
Is there a standard solution for this problem that I'm not aware of because I'm a noob?
Thanks for every answer.
Store information in a CSV and then use csv.reader to write the key, value pairs to something like a dictionary. Here's some example code to get you started:
csv file:
firstname Garrett
something_else blah
Python:
import csv
output = []
with open('csv.csv') as csvfile:
reader = csv.reader(csvfile, delimiter=' ')
for row in reader:
output.append(tuple(row))
return {key: value for (key, value) in output}

Resources