Hi I am writing unittest using pytest. But I am not able to mock few db functions. We are using psycopg2 for db connections and executions. Response of query returned from psycopg2 is of the type DictRow which can be accessed either by key or by index.
Ex:
response = ['prajwal', '23', 'engineer'] #Response of a query "select name, age , job from users"
>>>response[0]
'prajwal'
>>>response['name']
'prajwal'
I want to know is there any way by which we can covert dict/list to above mentioned type.
Looking at the source for psycopg2, creating a DictRow requires passing in a DictCursor object. However the only thing it uses from DictCursor appears to be an index and description attribute.
# found in lib\site-packages\psycopg2.extras.py
class DictRow(list):
"""A row object that allow by-column-name access to data."""
__slots__ = ('_index',)
def __init__(self, cursor):
self._index = cursor.index
self[:] = [None] * len(cursor.description)
The index looks like a dict with a mapping a key to an index. e.g.response['name'] = 0
The description looks like your dict that you want to convert.
If you're feeling hacky you could take advantage of duck typing and pretend you're passing in a cursor when you're just satisfying the requirements.
The only caveat is after we instantiate the DictRow, we need to populate it. Our fake cursor hack will take care of the rest.
from psycopg2.extras import DictRow
class DictRowHack:
def __init__(self, my_dict):
# we need to set these 2 attributes so that
# it auto populates our indexes
self.index = {key: i for i, key in enumerate(my_dict)}
self.description = my_dict
def dictrow_from_dict(my_dict):
# this is just a little helper function
# so you don't always need to go through
# the steps to recreate a DictRow
fake_cursor = DictRowHack(my_dict)
my_dictrow = DictRow(fake_cursor)
for k, v in my_dict.items():
my_dictrow[k] = v
return my_dictrow
response = {'name': 'prajwal', 'age': '23', 'job': 'engineer'}
my_dictrow = dictrow_from_dict(response)
print(my_dictrow[1])
print(my_dictrow['name'])
print(type(my_dictrow))
Related
I am attempting to sort a dataframe by a column called 'GameId', which are currently of type string and when I attempt to sort the result is unexpected. I have tried the following but still return a type string.
TEST['GameId'] = TEST['GameId'].astype(int)
type('GameId')
One way to make the data life easier is using dataclasses!
from dataclasses import dataclass
# here will will be calling the dataclass decorator to send hints for data type!
#dataclass
class Columns:
channel_id : int
frequency_hz : int
power_dBmV : float
name : str
# this class will call the data class to organise the data as data.frequency data.power_dBmV etc
class RadioChannel:
radio_values = ['channel_id', 'frequency', 'power_dBmV']
def __init__(self, data): # self is 'this' but for python, it just means that you mean to reference 'this' or self instance
self.data = data # this instances data is called data here
data = Columns(channel_id=data[0], frequency=data[1], power_dBmv=data[4], name=data[3]) # now we give data var a val!
def present_data(self):
# this is optional class method btw
from rich.console import Console
from rich.table import Table
console = Console()
table = Table(title="My Radio Channels")
for item in self.radio_values:
table.add_column(item)
table.add_row(data.channel_id, data.frequency_hz, data.power_dBmv)
console.print(table)
# ignore this if its confusing
# now inside your functional part of your script
if __name__ == '__main__':
myData = []
# calling an imaginary file here to read
with open("my_radio_data_file", 'r') as myfile:
mylines = myfile.readlines()
for line in myline:
myData.append(line)
myfile.close()
#my data would look like a string ["value", value, 00, 0.0, "hello joe, from: world"]
ch1 = radioChannel(data=myData[0])
ch1.present_data()
This way you can just call the class object on each line of a data file. and print it to see if it lines up. once you get the hang of it, it starts to get fun.
I used rich console here, but it works well with pandas and normal dataframes!
dataclasses help the interpreter find its way with type hints and class structure.
Good Luck and have fun!
Note: I am using python 3.8.2, so dictionaries are considered ordered
I was creating a binary tree, and I used dictionaries to model the tree.
For example:
{1:[2, 3], 2:[4, 5], 3:[], 4:[], 5:[]}
In the example, the tree above would look like this:
1
/ \
2 3
/ \
4 5
I was trying to simulate the ‘rising’ of some nodes, and keep the order of the dict as well.
I know that using myDict[key1], myDict[key2] = myDict[key2], myDict[key1] won’t work, as the places of the values change, not the keys.
I was also thinking of using .popitem() to remove the last value until I’m either at key1 or key2, and the keep going until I get to the other key, but this seems kinda hard. Is there any other way to do this?
Dictionaries, although they now keep the insertion order, are not arbitrarily orderable otherwise. If you want really to use the dict order information to build your tree, I think the only reliable way would be to build a fresh dictionary, copying the contents of the original one, for each of these swap operations.
A more reasonable approach, if you want an arbitrarily ordered dictionary would be to inherit from collections.abc.MutableMappingand kep track of your data inside that object, using a dictionary and some other data structure, such as a list.
It may sound complicated, but it may be simpler than you think:
from collections.abc import MutableMapping
class SuperOrdered(MutableMapping):
def __init__(self):
self.data = {}
self.order = []
def __setitem__(self, key, value):
if key not in self.data:
self.order.append(key)
self.data[key] = value
def __getitem__(self, key):
return self.data[key]
def __delitem__(self, key):
del self.data[key]
self.order.remove(key)
def __len__(self):
return len(self.data)
def __iter__(self):
yield from iter(self.order)
def replace_key(self, oldkey, newkey, value):
if newkey in self.data:
del self[newkey]
position = self.order.index(oldkey)
self.order[position] = newkey
self.data[newkey] = value
def __repr__(self):
return f"{self.__class__.__name__}({{{', '.join(repr(key) + ':' + repr(self.data[key]) for key in self)}}})"
And voilá - the mapping + the "replace_key" method should be enough for you to build your tree as you are thinking about it.
This is the class above in the interactive prompt:
In [18]: aa = SuperOrdered()
In [19]: aa["a"] = 1;aa["b"] = 2;aa["c"] = 3
In [20]: aa
Out[20]: SuperOrdered({'a':1, 'b':2, 'c':3})
In [21]: aa.replace_key("a", "d", 4)
In [22]: aa
Out[22]: SuperOrdered({'d':4, 'b':2, 'c':3})
Apart from this answer, and out of topic: if you want to check a tree implementation that I hope is "production ready", I've published one as part of my extradict package (pip installable).
update: One might also inherit from collections.OrderedDict and add a replace_key method there. That code would have to deal with OrderedDict internals, but it would not be hard.
External links:
Github modification
Since you only want to swap two keys, we could start off with the following code:
>>> changedDict = {}
>>> for key, value in myDict.items():
... if key not in (key1, key2):
... changedDict[key] = value
And continue as follows:
... elif key == key1:
... changedDict[key] = myDict[key2]
... else:
... changedDict[key] = myDict[key1]
This question may be opinion based, but I figured I'd give it shot.
I am attempting to create a variety of classes which gets its values from JSON data. The JSON data is not under my control so I have to parse the data and select the values I want. My current implementation subclasses UserDict from python3's collection module. However, I have had iterations where I have directly created attributes and set the values to the parsed data.
The reason I changed to using the UserDict is the ease of using the update function.
However, I feel odd calling the object and using MyClass['attribute'] rather than MyClass.attribute
Is there a more pythonic way to model this data?
I am not 100% convinced that this makes sense, but you could try this:
class MyClass (object):
def __init__(self, **kwargs):
for key in kwargs.keys():
setattr(self, key, kwargs[key])
my_json = {"a":1, "b":2, "c":3}
my_instance = MyClass(**my_json)
print (my_instance.a)
# 1
print (my_instance.b)
# 2
print (my_instance.c)
# 3
--- edit
in case you have nested data you could also try this:
class MyClass (object):
def __init__(self, **kwargs):
for key in kwargs.keys():
if isinstance(kwargs[key],dict):
setattr(self, key, MyClass(**kwargs[key]))
else:
setattr(self, key, kwargs[key])
my_json = {"a":1, "b":2, "c":{"d":3}}
my_instance = MyClass(**my_json)
print (my_instance.a)
# 1
print (my_instance.b)
# 2
print (my_instance.c.d)
# 3
I have the following code structure written in Python3.6, which I need to test using sqlite3 (because of standards defined in my project):
class BigSecretService:
""" Class designed to make calculations based on data stored in MySQL. """
def load_data(self):
# load some data using sqlalchemy ORM
def get_values_from_fields(self, fields):
# here's getting values via sqlalchemy execute with raw query:
self.sql_service.execute(SOME_QUERY)
def process_data(self, data, values):
# again execute some raw query
# process data and put into result list
return reuslt_list
def make_calculations(self, params):
data = self.load_data()
values = self.get_values_from_fields(fields)
result_vector = process_data(data, values)
SOME_QUERY is in separate module and it's format looks like this:
"SELECT SUM(some_field) FROM some_table WHERE col1 = :col1 AND col2 = :col2"
To cover make_calculations in my component test I designed awful patches:
class PatchedConnection:
""" Class is used to transform queries to sqlite format before executing. """
def __init__(self, connection, engine):
self.connection = connection
self.engine = engine
def __call__(self):
conn = self.connection()
conn.execute = self.patched_execute(conn.execute)
return conn
def transform_date(self, date):
try:
# quick check just for testing
if '+00:00' in date:
date = date.replace('T', ' ').replace('+00:00', '.000000')
finally:
return date
def patched_execute(self, f_execute):
def prepare_args_for_sqlite(query, *args):
# check if query is in sqlite format
if args:
if '?' in str(query):
args = list(map(self.transform_date, list(args[0].values())))
return self.engine.execute(str(query), args)
return f_execute(query, args[0])
else:
return f_execute(query)
return prepare_args_for_sqlite
Then in test it looks like this:
QUERY_TEMPLATE_SQLITE = 'SELECT SUM(some_field) FROM some_table WHERE col1 = ? AND col2 = ?'
with mock.patch('path_to_my_service.SOME_QUERY', QUERY_TEMPLATE_SQLITE):
self.sql_service.get_connection = PatchedConnection(self.sql_service.get_connection, self.engine)
response = self.client.simulate_post("/v1/secret_service/make_calculations",
headers=self.auth_header,
body=json.dumps(payload))
self.assertEqual(response.status_code, 200)
# then check response.text
It works so far, but I believe there must be much better solution. Moreover, in patched_execute args from dict are being converted to list, and who knows if order of dict values will be the same all the time.
So, my question is how to perform such testing in a correct way with given tools?
If you need to intercept and manipulate the SQL being sent to the database then using core events https://docs.sqlalchemy.org/en/13/core/events.html would be the most straightforward way of doing this. The before_cursor_execute event would suit your purposes as outlined in the following example from the SQLAlchemy documentation.
#event.listens_for(engine, "before_cursor_execute", retval=True)
def before_cursor_execute(conn, cursor, statement, parameters, context, executemany):
# do something with statement, parameters
return statement, parameters
From the example you have given however, I'm not sure that this is necessary. The MySQL query you have listed is also a valid SQLite query and needs no manipulation. Also if you pass your parameters as python objects, rather than as strings, then again no manipulation should be needed as SQLAlchemy will map these correctly to the backend.
Please correct my code
PS - i'm fairly new to python
class Contact:
def __init__(self,cid, email):
self.cid=cid
self.email=email
def ind(contacts):
index={}
#Code here
return index
contacts = [Contact(1,'a'),
Contact(2,'b'),
Contact(3,'c'),
Contact(4,'a')]
print(ind(contacts))
Need the output to be like -
{'a':[1,4], 'b':2, 'c':3}
The following methods create list values like:
{'a':[1,4], 'b':[2], 'c':[3]}
I can't imagine why this wouldn't be fine, but I've added a method at the end that gets your specific output.
This doesn't maintain order of the emails:
def ind(contracts):
index={}
for contract in contracts:
index.setdefault(contract.email, []).append(contract.cid)
return index
To maintain order (e.g. start with 'a'), add from collects import OrderedDict to the top of your file and then the method is:
def ind(contracts):
index = OrderedDict()
for contract in contracts:
index.setdefault(contract.email, []).append(contract.cid)
return index
The printout of index will look different, but it acts the same as a normal dict object (just with ordering).
Exact output (with ordering):
def ind(contracts):
index = OrderedDict()
for contract in contracts:
if contract.email in index:
value = index[contract.email]
if not isinstance(value, list):
index[contract.email] = [value]
index[contract.email].append(contract.cid)
else:
index[contract.email] = contract.cid
return index