Find values from the Nested Dict - python-3.x

Something I'm doing incorrect, but I got data with list of nested dict, I need to pull the value and store in the output dict.
Data:
[{
"encoding":"json",
"result": {
"aclList":[
{
"chipName":"",
"countersEnabled":true,
"countersIncomplete":false,
"dynamic":false,
"name":"FW-RULE-IN",
"readonly":false,
"sequence":[
{
"convertSymbols":true,
"sequenceNumber":1,
"text":"TEST: RULE 1"
},
{
"convertSymbols":true,
"sequenceNumber":2,
"text":"TEST: RULE 2"
},
{
"convertSymbols":true,
"sequenceNumber":3,
"text":"TEST: RULE 3"
},]}]}}]
Like to put sequenceNumber, text values in to output result in Dict.
here is the code I got, but not working, how to fix it?
def parse_data(data):
rules = {}
acllist = []
for k, v in data[0].items():
for h, j in k['result'].items():
for i in h['aclList'][0]:
for l in i[0]['sequence']:
rules['NUM'] = int(j['sequenceNumber'])
rules['TEXT_RULES'] = j['text']
acllist.append(rules.copy())
print (j['sequenceNumber'], j['text'])
return acllist

I'm assuming there can be multiple elements to these lists. If there aren't you could simplify to.
for dct in data[0]['result']['aclList'][0]['sequence']:
rules['NUM'] = int(dct['sequenceNumber'])
rules['TEXT_RULES'] = dct['text']
acllist.append(rules.copy())
print (inner['sequenceNumber'], dct['text'])
However if you want to iterate through every list and gather the elements for each version, it would look more like this:
def parse_data(data):
acllist = []
# for each dict in the initial list
for dct in data:
# for each list in the ['aclList'] key of the ['result'] key
for lst in dct['result']['aclList']:
# for each inner dict in the ['sequence'] key
for inner in lst['sequence']:
rules = {}
rules['NUM'] = int(inner['sequenceNumber'])
rules['TEXT_RULES'] = inner['text']
acllist.append(rules.copy())
print (inner['sequenceNumber'], inner['text'])
return acllist
data =[{
"encoding":"json",
"result": {
"aclList":[
{
"chipName":"",
"countersEnabled":True,
"countersIncomplete":False,
"dynamic":False,
"name":"FW-RULE-IN",
"readonly":False,
"sequence":[
{
"convertSymbols":True,
"sequenceNumber":1,
"text":"TEST: RULE 1"
},
{
"convertSymbols":True,
"sequenceNumber":2,
"text":"TEST: RULE 2"
},
{
"convertSymbols":True,
"sequenceNumber":3,
"text":"TEST: RULE 3"
},]}]}}]
print(parse_data(data))
Where you went wrong is that you are iterating through the key/values of you dicts.
for k, v in data[0].items():
k here would contain "encoding" and "result".
And then you are trying to iterate through those keys.
for h, j in k['result'].items():
But what you are essentially doing here is saying:
k = "result"
for h, j in k['result'].items():
Or:
for h, j in "result"['result'].items():
You can see why this would throw an error.

Related

is there a method to collect data intelligently from website?

i want to get data from this link https://meshb.nlm.nih.gov/treeView
the problem is to get all the tree, we should click on + each time and for each line to get the children node of the tree,
but I want to display all the tree just on one click then i want to copy all the content.
Any ideas, please?
Well, it all depends what you mean by "intelligently". Not sure if that meets the criteria, but you might want to try this.
import json
import string
import requests
abc = string.ascii_uppercase
base_url = "https://meshb.nlm.nih.gov/api/tree/children/"
follow_url = "https://meshb.nlm.nih.gov/record/ui?ui="
tree = {}
for letter in abc[:1]:
res = requests.get(f"{base_url}{letter}").json()
tree[letter] = {
"Records": [i["RecordName"] for i in res],
"FollowURLS": [f"{follow_url}{i['RecordUI']}" for i in res],
}
print(json.dumps(tree, indent=2))
This prints:
{
"A": {
"Records": [
"Body Regions",
"Musculoskeletal System",
"Digestive System",
"Respiratory System",
"Urogenital System",
"Endocrine System",
"Cardiovascular System",
"Nervous System",
"Sense Organs",
"Tissues",
"Cells",
"Fluids and Secretions",
"Animal Structures",
"Stomatognathic System",
"Hemic and Immune Systems",
"Embryonic Structures",
"Integumentary System",
"Plant Structures",
"Fungal Structures",
"Bacterial Structures",
"Viral Structures"
],
"FollowURLS": [
"https://meshb.nlm.nih.gov/record/ui?ui=D001829",
"https://meshb.nlm.nih.gov/record/ui?ui=D009141",
"https://meshb.nlm.nih.gov/record/ui?ui=D004064",
"https://meshb.nlm.nih.gov/record/ui?ui=D012137",
"https://meshb.nlm.nih.gov/record/ui?ui=D014566",
"https://meshb.nlm.nih.gov/record/ui?ui=D004703",
"https://meshb.nlm.nih.gov/record/ui?ui=D002319",
"https://meshb.nlm.nih.gov/record/ui?ui=D009420",
"https://meshb.nlm.nih.gov/record/ui?ui=D012679",
"https://meshb.nlm.nih.gov/record/ui?ui=D014024",
"https://meshb.nlm.nih.gov/record/ui?ui=D002477",
"https://meshb.nlm.nih.gov/record/ui?ui=D005441",
"https://meshb.nlm.nih.gov/record/ui?ui=D000825",
"https://meshb.nlm.nih.gov/record/ui?ui=D013284",
"https://meshb.nlm.nih.gov/record/ui?ui=D006424",
"https://meshb.nlm.nih.gov/record/ui?ui=D004628",
"https://meshb.nlm.nih.gov/record/ui?ui=D034582",
"https://meshb.nlm.nih.gov/record/ui?ui=D018514",
"https://meshb.nlm.nih.gov/record/ui?ui=D056229",
"https://meshb.nlm.nih.gov/record/ui?ui=D056226",
"https://meshb.nlm.nih.gov/record/ui?ui=D056224"
]
}
}
If you want all of it, just remove [:1] from the loop. If there's no entry for a given letter on the page you'll get, well, an empty entry in the dictionary.
Obviously, you can dump the entire response, but that's just a proof of concept.
Try this, some parts are a bit tricky but it manages to give you the tree:
import requests as r
import operator
import string
link = 'https://meshb.nlm.nih.gov/api/tree/children/{}'
all_data = []
for i in string.ascii_uppercase:
all_data.append({'RecordName': i, 'RecordUI': '', 'TreeNumber': i, 'HasChildren': True})
res = r.get(link.format(i))
data_json = res.json()
all_data += data_json
# This request will get all the rest of the data at once, other than A-Z or A..-Z..
# This request takes time to load, depending on your network, it got like 3 million+ characters
res = r.get(link.format('.*'))
data_json = res.json()
all_data += data_json
# Sorting the data depending on the TreeNumber
all_data.sort(key=operator.itemgetter('TreeNumber'))
# Printing the tree using tabulations
for row in all_data:
l = len(row['TreeNumber'])
if l == 3:
print('\t', end='')
elif l > 3:
print('\t'*(len(row['TreeNumber'].split('.'))+1), end='')
print(row['RecordName'])

Python list add variables in rows

im trying to add variables to a list that i created. Got a result from a session.execute.
i´ve done this:
def machine_id(session, machine_serial):
stmt_raw = '''
SELECT
id
FROM
machine
WHERE
machine.serial = :machine_serial_arg
'''
utc_now = datetime.datetime.utcnow()
utc_now_iso = pytz.utc.localize(utc_now).isoformat()
utc_start = datetime.datetime.utcnow() - datetime.timedelta(days = 30)
utc_start_iso = pytz.utc.localize(utc_start).isoformat()
stmt_args = {
'machine_serial_arg': machine_serial,
}
stmt = text(stmt_raw).columns(
#ts_insert = ISODateTime
)
result = session.execute(stmt, stmt_args)
ts = utc_now_iso
ts_start = utc_start_iso
ID = []
for row in result:
ID.append({
'id': row[0],
'ts': ts,
'ts_start': ts_start,
})
return ID
In trying to get the result over api like this:
def form_response(response, session):
result_machine_id = machine_id(session, machine_serial)
if not result_machine_id:
response['Error'] = 'Seriennummer nicht vorhanden/gefunden'
return
response['id_timerange'] = result_machine_id
Output looks fine.
{
"id_timerange": [
{
"id": 1,
"ts": "2020-08-13T08:32:25.835055+00:00",
"ts_start": "2020-07-14T08:32:25.835089+00:00"
}
]
}
Now i only want the id from it as a parameter for another function. Problem is i think its not a list. I cant select the first element. result_machine_id[0] result is like the posted Output. I think in my first function i only add ts & ts_start to the first row? Is it possible to add emtpy rows and then add 'ts':ts as value?
Help would be nice
If I have understood your question correctly ...
Your output looks like dict. so access its id_timerange key which gives you a list. Access the first element which gives you another dict. On this dict you have an id key:
result_machine_id["id_timerange"][0]["id"]

Accessing Elements in a nested list in python

I am a newbie in python and I have data in a nested list in this format. I want to do something if the value of age in the dictionaries are equal.
[
[
{
"name":"AAAAA",
"age":"12",
"class":"box"
}
],
[
{
"name":"DDDD",
"age":"10",
"class":"space"
}
],
[
{"name":"BBBB"},
{"age":"16"},
{"class":"cage"}
],
[
{
"name":"EEEE",
"age":"20",
"class":"sage"
}
],
]
The main idea is to store all the ages in a list, get list of duplicate ages and access the dictionary using that list
age_list = []
for l in dict_list:
temp_dict = l[0]
age_list.append(temp_dict["age"])
# get set of duplicates in age list
dup_ages = set([x for x in l if l.count(x) > 1])
for index, age in enumerate(age_list):
for dup_age in dup_ages: # do something for a given duplicate age
if dup_age == age:
dict_list[index] # do something in the dict

List comprehension to dynamically populate attribute of object

What’s the correct way to dynamically populate the list given to the attribute Value? At the moment its its fixed to 3 (0-2), but would be more useful if it was based on INSTANCE_PARAMS[i]["instances"].
I was thinking along the lines of list comprehension but unsure how to write it into the code.
for i in INSTANCE_PARAMS:
output = template.add_output([
Output(
str(INSTANCE_PARAMS[i]["name"]).capitalize() + "Ips",
Description="IPs for " + str(INSTANCE_PARAMS[i]["name"]),
Value=Join(",", [
GetAtt(ec2.Instance(INSTANCE_PARAMS[i]["name"] + "0"), "PrivateIp"),
GetAtt(ec2.Instance(INSTANCE_PARAMS[i]["name"] + "1"), "PrivateIp"),
GetAtt(ec2.Instance(INSTANCE_PARAMS[i]["name"] + "2"), "PrivateIp"),
],
),
)
],
)
INSTANCE_PARAMS = {
"masters": {
"name": "master",
"instances": 3,
"image_id": "ami-xxxxxxxxxx",
"instance_type": "t1.micro",
"security_group_id": [
"MasterSG",
"ClusterSG"
],
},
}
Achieved it with the following:
template = Template()
for i in INSTANCE_PARAMS:
# declared a new list
tag_value = []
# got the counter
for r in range(INSTANCE_PARAMS[i]["instances"]):
# added each one to the new list
tag_value.append(GetAtt(ec2.Instance(INSTANCE_PARAMS[i]["name"] + str(r)), "PrivateIP"))
output = template.add_output([
Output(
str(INSTANCE_PARAMS[i]["name"]).capitalize() + "Ips",
Description="IPs for " + str(INSTANCE_PARAMS[i]["name"]),
# passed in the list
Value=Join(",", tag_value,
),
)
],
)
print(template.to_yaml())

Changing the values of a map nested in another map

Using HQL queries I've been able to generate the following map, where the keys represent the month number constant defined in java.util.Calendar, and every value is a map:
[
0:[ client_a:[order1, order2, order3]],
1:[ client_b:[order4], client_c:[order5, order6], client_d:[order7]],
2:[ client_e:[order8, order9], client_f:[order10]]
]
order1, order2, ... are instances of a domain class called Order:
class Order {
String description
Date d
int quantity
}
Now I've got that structure containing orders that belong to some specific year, but I don't really care about the Order object itself. I just want the sum of the quantities of all the orders of each month. So the structure should look something like this:
[
0:[ client_a:[321]],
1:[ client_b:[9808], client_c:[516], client_d:[20]],
2:[ client_e:[22], client_f:[10065]]
]
I don't mind if the values are lists of one element or not lists at all. If this is possible, it would be fine anyway:
[
0:[ client_a:321 ],
1:[ client_b:9808, client_c:516, client_d:20 ],
2:[ client_e:22, client_f:10065 ]
]
I know I have to apply something like .sum{it.quantity} to every list of orders to get the result I want, but I don't know how to iterate over them as they are nested within another map.
Thank you.
Here You go:
class Order {
String description
Date d
int quantity
}
def orders = [
0:[ client_a:[new Order(quantity:1), new Order(quantity:2), new Order(quantity:3)]],
1:[ client_b:[new Order(quantity:4)], client_c:[new Order(quantity:5), new Order(quantity:6)], client_d:[new Order(quantity:7)]],
2:[ client_e:[new Order(quantity:8), new Order(quantity:9)], client_f:[new Order(quantity:10)]]
]
def count = orders.collectEntries { k, v ->
def nv = v.collectEntries { nk, nv ->
[(nk): nv*.quantity.sum()]
}
[(k):(nv)]
}
assert count == [0:[client_a:6], 1:[client_b:4, client_c:11, client_d:7],2:[client_e:17, client_f:10]]
def map = [
0:[ client_a:[[q: 23], [q: 28], [q: 27]]],
1:[ client_b:[[q: 50]], client_c:[[q: 100], [q: 58]], client_d:[[q: 90]]],
2:[ client_e:[[q: 48], [q: 60]], client_f:[[q: 72]]]
]
map.collectEntries { k, v ->
[ k, v.collectEntries { key, val ->
[ key, val*.q.sum() ]
} ]
}
you can also use val.sum { it.q } instead of val*.q.sum()

Resources