I am trying to simulate the reading of an hdf5 file in python3 for testing. The function controls, if the file contains all the required keywords. The structure of the file is:
Group: parent
Dataset: A
Dataset: B
My plan is to use MagicMock object for the file object and the group object.
Testfile:
def test_checkInput(self):
file = MagicMock()
my_values = MagicMock()
my_values.keys.return_value = ['A', 'B']
file.keys.return_value = {'parent': my_values}
self.assertTrue(reader_class.checkInput(file))
reader_class is the module where the function checkInput(file)
is defined
reader_class - file
def checkInput(file):
if not file.keys:
return False
if 'parent' in file.keys():
group = file['parent']
if 'A' in group.keys():
if 'B' in group.keys():
return True
else:
print(f"[ {__file__} ] : No Group: 'A'")
return False
else:
print(f"[ {__file__} ] : No Group: 'B'")
return False
else:
print(f"[ {__file__} ] : No Group: 'parent'")
return False
The problem is, that group.keys() does not return ['A', 'B'] as expected in the checkInput function. It returns a MagicMock-Object instead. How can a get the set values?
The problem was, that the MagicMock objects in the the reader_class file are different.
file.keys() returns another mock as file['parent']
I found a simple solution:
in the Testfile just use one MagicMock object and specify the return values
def test_checkInput(self):
file = MagicMock()
file.keys.return_value = 'parent'
file['parent'].keys.return_value = ['A', 'B']
self.assertTrue(reader_class.checkInput(file))
Related
How can I evaluate if a env variable is a boolean True, in Python? Is it correct to use:
if os.environ['ENV_VAR'] is True:
.......
Option 1
I think this works well:
my_env = os.getenv("ENV_VAR", 'False').lower() in ('true', '1', 't')
It allows: things like true, True, TRUE, 1, "1", TrUe, t, T, ...
Update: After I read the commentary of Klaas, I updated the original code my_env = bool(os.getenv(... to my_env = os.getenv(... because in will result in a bool type
Option 2
UPDATE:
After the #MattG commentary, I added a new solution that raises an error for entries like ttrue instead of returning False:
# ...
import os
# ...
def get_variable(name: str, default_value: bool | None = None) -> bool:
true_ = ('true', '1', 't') # Add more entries if you want, like: `y`, `yes`, `on`, ...
false_ = ('false', '0', 'f') # Add more entries if you want, like: `n`, `no`, `off`, ...
value: str | None = os.getenv(name, None)
if value is None:
if default_value is None:
raise ValueError(f'Variable `{name}` not set!')
else:
value = str(default_value)
if value.lower() not in true_ + false_:
raise ValueError(f'Invalid value `{value}` for variable `{name}`')
return value in true_
# ...
my_env1 = get_variable("ENV_VAR1")
my_env2 = get_variable(name="ENV_VAR2") # Raise error if variable was not set
my_env3 = get_variable(name="ENV_VAR3", default_value=False) # return False if variable was not set
All the same, but thats the most readable version for me:
DEBUG = (os.getenv('DEBUG', 'False') == 'True')
Here anything but True will evaluate to False. DEBUG is False unless explicitly set to True in ENV
I recommend using strtobool function
example:
DEBUG = strtobool(os.getenv("DEBUG", "false"))
You can check them in python documentation
https://docs.python.org/3/distutils/apiref.html#distutils.util.strtobool
Only one problem, they raise an error if you pass the wrong value
Code
from distutils.util import strtobool
print("Value: ", strtobool("false"))
print("Value: ", strtobool("Wrong value"))
Output
Value: 0
Traceback (most recent call last):
File "<string>", line 9, in <module>
File "/usr/lib/python3.8/distutils/util.py", line 319, in strtobool
raise ValueError("invalid truth value %r" % (val,))
ValueError: invalid truth value 'wrong value'
Neither of the ways you have will work. os.environ['ENV_VAR'] alone will cause a KeyError if the key doesn't exist, and will return the value associated with the 'ENV_VAR' if it does. In either case, you'll error out, or compare to True or "true" which will always result in False (unless the value associated with the environment variable happens to be "true"; but that isn't what you're after).
To check if a mapping contains a particular key, you would use in:
if 'ENV_VAR' in os.environ:
# It contains the key
else:
# It doesn't contain the key
Highly recommend environs:
from environs import Env
env = Env()
MY_BOOL_VALUE = env.bool("MY_BOOL_VALUE", False)
if MY_BOOL_VALUE:
print("MY_BOOL_VALUE was set to True.")
else:
print("MY_BOOL_VALUE was either set to False or was not defined.")
Another alternative that accepts either "false", "False", "true" or "True":
import os
import ast
def getenv_bool(name: str, default: str = "False"):
raw = os.getenv(name, default).title()
return ast.literal_eval(raw)
I use the following to have more strict typing and support wider boolean variations in inputs
import os
def getenv_bool(name: str, default: bool = False) -> bool:
return os.getenv(name, str(default)).lower() in ("yes", "y", "true", "1", "t")
Usage :
feature_1=getenv_bool('FEATURE_1', False)
If you don't want to use the environs library mentioned above then strtobool is perfect for this. The only problem is it is deprecated, there does not seem to be a replacement library anywhere, but luckily it is only a few lines of simple code with no dependencies. Just implement the code:
# Copied from distutils.util.strtobool, which is deprecated
def strtobool (val):
"""Convert a string representation of truth to true (1) or false (0).
True values are case insensitive 'y', 'yes', 't', 'true', 'on', and '1'.
false values are case insensitive 'n', 'no', 'f', 'false', 'off', and '0'.
Raises ValueError if 'val' is anything else.
"""
val = val.lower()
if val in ('y', 'yes', 't', 'true', 'on', '1'):
return 1
elif val in ('n', 'no', 'f', 'false', 'off', '0'):
return 0
else:
raise ValueError("invalid truth value %r" % (val,))
Use it like this:
my_env_var_value = strtobool(os.getenv("ENV_VAR", "False"))
And YES, this will throw an error if the environment variable has some value that is neither true nor false. In the great majority of cases that is probably the best course of action.
Another possible solution is parse values as JSON values:
import json
import os
def getenv(name, default="null"):
try:
return json.loads(os.getenv(name, default))
except json.JSONDecodeError:
return name
The try is for cases when is not possible a direct conversion.
assert getenv("0") == 0
assert getenv("1.1") = 1.1
assert getenv("true") == True
assert getenv("Hello") = "Hello"
assert getenv('["list", "of", "strings"]') == ["list", "of", "strings"]
Python Version:
3.9.13 (main, May 27 2022, 17:01:00)
I used solution below and it works perfect;
In your env file;
SOMEHING_ENABLED=True
and use case in your python file;
from distutils.util import strtobool
if bool(strtobool(os.getenv('SOMEHING_ENABLED'))):
# good to go.
Not: distutils will no longer work from version 3.12
I am ultimately trying to save variables that my browser finds to a file in order for it to be later recalled in order to compare if it has went through those values before. Before I reach that step, I am testing my code and have been running into issues:
First part of my code with no error:
import shelve
shelfFile = shelve.open('mydata')
cats = ['Zophie', 'Pooka', 'Simon']
shelfFile['cats'] = cats
shelfFile.close()
This does what it is intended to do, saves cats to a file.
import shelve
shelfFile = shelve.open('mydata')
cats = shelfFile['cats']
shelfFile.close()
new = 'Zophie', 'Pooka', 'Simon'
if new in cats:
print('Found it!')
else:
print("There is an error")
When I run the code it tells me there is an error rather than saying that it found it. Since the list variables are the same, why are they not matching?
I haven't seen a declaration of variables through the "comma-separated list" as you did: new = 'Zophie', 'Pooka', 'Simon'.
I'm pretty sure you just did typo and should use an array for names:
new = ['Zophie', 'Pooka', 'Simon']
for item in new:
if item in cats:
print('Found it!')
else:
print("Not found")
or:
new = ['Zophie', 'Pooka', 'Simon']
for item in new:
if item in cats:
print(f'Found {item}!')
else:
print(f'{item} is not found')
You are checking a tuple available in a list
new = 'Zophie', 'Pooka', 'Simon' is a tuple object
and you are expecting a cats = ['Zophie', 'Pooka', 'Simon'] as a list.
If cats = shelfFile['cats'] returns the list of
cats then you need to do is
for n in new:
if n in cats:
print('Found it!')
else:
print('Not found')
You can demonstrate the logic by running this script
>>> x = "a", "b", "c"
>>> print(x)
('a', 'b', 'c')
>>> y = ["a", "b", "c"]
>>> x in y
False
>>> for n in x:
... if n in y:
... print("found")
found
found
found
I have a python script which is watching a folder using inotify, and when a new json file is added I would like the file to be opened, and the keys of the json file to be renamed as indicated below (e.g. a - aa, b -- bb). The value in each key value pair should remain the same.
The below is an extract of the script in which I am trying to achieve this task.
When run I get the error
AttributeError: 'str' object has no attribute 'pop'
I am also pretty sure this is not the best or most efficient way to do this, so open to suggestions...
import glob
names_key = { 'a' : 'aa' ,
'b' : 'bb',
'c' : 'cc',
'd' : 'dd',
'e' : 'ee',
'f' : 'ff',
'g' : 'gg',
'h' : 'hh'
}
destinationFolder = "inspect"
all_files = glob.glob('inspect/*.json')
def rename_json():
for row in all_files:
for k, v in names_key.items():
for old_name in row:
if k == old_name:
row[v] = row.pop(old_name)
if __name__ == "__main__":
rename_json()
You can turn the JSON from your JSON file into a dict with json.loads(your_entire_json_file_as_a_string)
Then you can directly key into your json to change the appropriate keys.
For Example:
import json
def get_json_file_as_string(filepath):
with open(filepath) as json_file:
return json_file.read().replace('\n', '')
file_path = changed_json_file_path
names_key = { 'a' : 'aa' , 'b' : 'bb', }
json_dict = json.loads(get_json_file_as_string(changed_json_file_path))
json_dict = {'aa':1, 'bb':2}
for new_key, old_key in names_key.items():
if old_key in json_dict:
json_dict[new_key] = json_dict[old_key]
json_dict.pop(old_key)
print(json_dict)
Edit
This depends on the structure of the data in your JSON file. But if it's just a simple list of name value pairs this will work nicely
In python3, I have the following code:
path = '/path/to/file/containing/python/code'
source = open(path, 'r').read()
codeobject = compile(source, path, 'exec')
I have examined codeobject, but I don't see any way to get a list of all the functions that are defined within that object.
I know I can search the source string for lines that begin with def, but I want to get this info from the code object, if at all possible.
What am I missing?
A code object is a nested structure; functions are created when the code object is executed, with their bodies embedded as separate code objects that are part of the constants:
>>> example = '''\
... def foobar():
... print('Hello world!')
... '''
>>> codeobject = compile(example, '', 'exec')
>>> codeobject
<code object <module> at 0x11049ff60, file "", line 1>
>>> codeobject.co_consts
(<code object foobar at 0x11049fe40, file "", line 1>, 'foobar', None)
>>> codeobject.co_consts[0]
<code object foobar at 0x11049fe40, file "", line 1>
>>> codeobject.co_consts[0].co_name
'foobar'
When you disassemble the top-level code object you can see that the function objects are created from such code objects:
>>> import dis
>>> dis.dis(codeobject)
1 0 LOAD_CONST 0 (<code object foobar at 0x11049fe40, file "", line 1>)
2 LOAD_CONST 1 ('foobar')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (foobar)
8 LOAD_CONST 2 (None)
10 RETURN_VALUE
The MAKE_FUNCTION opcode takes the code object from the stack, as well as the function name and any default argument values from the stack; you can see the LOAD_CONST opcodes preceding it that put the code object and name there.
Not all code objects are functions however:
>>> compile('[i for i in range(10)]', '', 'exec').co_consts
(<code object <listcomp> at 0x1105cb030, file "", line 1>, '<listcomp>', 10, None)
>>> compile('class Foo: pass', '', 'exec').co_consts
(<code object Foo at 0x1105cb0c0, file "", line 1>, 'Foo', None)
If you wanted to list what functions are loaded in the bytecode, your best bet is to use the disassembly, not look for code objects:
import dis
from itertools import islice
# old itertools example to create a sliding window over a generator
def window(seq, n=2):
"""Returns a sliding window (of width n) over data from the iterable
s -> (s0,s1,...s[n-1]), (s1,s2,...,sn), ...
"""
it = iter(seq)
result = tuple(islice(it, n))
if len(result) == n:
yield result
for elem in it:
result = result[1:] + (elem,)
yield result
def extract_functions(codeobject):
codetype = type(codeobject)
signature = ('LOAD_CONST', 'LOAD_CONST', 'MAKE_FUNCTION', 'STORE_NAME')
for op1, op2, op3, op4 in window(dis.get_instructions(codeobject), 4):
if (op1.opname, op2.opname, op3.opname, op4.opname) == signature:
# Function loaded
fname = op2.argval
assert isinstance(op1.argval, codetype)
yield fname, op1.argval
This generates (name, codeobject) tuples for all functions that are loaded in a given code object.
Can someone explain this error? The contents of DictTest.py are below. If I copy (%paste) this code into an ipython terminal the test passes. If if call
>>> %run DictTest.py -m
The test fails with
name 'keys' is not defined
The 'keys' that it is complaining about is the "in keys" part of the dict comprehension. I am using 3.4.1 |Anaconda 2.1.0 (64-bit) on linux.
#!/usr/bin/python3.4
import unittest
class DictTest(unittest.TestCase):
def test_dict_comprehension(self):
code = """
d = {'a':1, 'b':2, 'c':3, 'd':4}
keys = ['a', 'd']
items = d.items()
nd = {k: v for k, v in items if k in keys}
print('>>>' + str(nd))
"""
try:
exec(code)
except Exception as e:
self.assertTrue(False, "Exec ERROR>>> %s" % e)
def main():
dt = DictTest()
dt.test_dict_comprehension()
if __name__ =='__main__':main()
The answer is (mostly) in the docs for exec, assignment statements, and comprehensions.\
Exec: exec(s) is equivalent to exec(s, globals(), locals()). At module scope (case1), locals is globals(). In function scope (case2), they are two different objects. "If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition." The following gives the same error about 'keys' not recognized.
class C:
d = {'a':1, 'b':2, 'c':3, 'd':4}
keys = ['a', 'd']
items = d.items()
nd = {k: v for k, v in items if k in keys}
print('>>>' + str(nd))
=: name = value binds name to value in the local namespace, which may or may not be the same as the global namespace.
{comprehension}: In 3.x, a comprehension is evaluated in a separate context (except for the source of the first for clause -- not documented very well). So items is immediately evaluated and while 'keys' is evaluated in the new context, where locals only has binding for 'k', and 'v' (which is why 'k' gets evaluated). For case 2, 'keys' in not in globals either, and an exception is raised.
A solution for this code:
d = {}
...
exec(code, d, d)
Other uses might require additional initialization for d.