{
"success": true,
"time": 1660441201,
"currency": "RUB",
"items": {
"186150629_143865972": {
"price": "300.00",
"buy_order": 220,
"avg_price": "279.405789",
"popularity_7d": "19",
"market_hash_name": "CS:GO Case Key",
"ru_name": "\u041a\u043b\u044e\u0447 \u043e\u0442 \u043a\u0435\u0439\u0441\u0430 CS:GO",
"ru_rarity": "\u0431\u0430\u0437\u043e\u0432\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0441\u0430",
"ru_quality": "",
"text_color": "D2D2D2",
"bg_color": ""
},
"186150630_143865972": {
"price": "993.06",
"buy_order": 200.02,
"avg_price": "573.320000",
"popularity_7d": "1",
"market_hash_name": "eSports Key",
"ru_name": "\u041a\u043b\u044e\u0447 eSports",
"ru_rarity": "\u0431\u0430\u0437\u043e\u0432\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0441\u0430",
"ru_quality": "",
"text_color": "D2D2D2",
"bg_color": ""
}
}
}
So that is my dictionary.For example I want to get all values from "price" and "market_hash_name"
How should I iterate to get them?
Saving the input dictionary you have defined into the variable input_dict, we can extract this information using simple list comprehenstions, either to extract it as tuples or as dictionaries :)
A step by step extraction can be seen here-
>>> input_dict['items']
{'186150630_143865972': {'popularity_7d': '1', 'bg_color': '', 'text_color': 'D2D2D2', 'ru_name': '\\u041a\\u043b\\u044e\\u0447 eSports', 'avg_price': '573.320000', 'price': '993.06', 'market_hash_name': 'eSports Key', 'buy_order': 200.02, 'ru_rarity': '\\u0431\\u0430\\u0437\\u043e\\u0432\\u043e\\u0433\\u043e \\u043a\\u043b\\u0430\\u0441\\u0441\\u0430', 'ru_quality': ''}, '186150629_143865972': {'popularity_7d': '19', 'bg_color': '', 'text_color': 'D2D2D2', 'ru_name': '\\u041a\\u043b\\u044e\\u0447 \\u043e\\u0442 \\u043a\\u0435\\u0439\\u0441\\u0430 CS:GO', 'avg_price': '279.405789', 'price': '300.00', 'market_hash_name': 'CS:GO Case Key', 'buy_order': 220, 'ru_rarity': '\\u0431\\u0430\\u0437\\u043e\\u0432\\u043e\\u0433\\u043e \\u043a\\u043b\\u0430\\u0441\\u0441\\u0430', 'ru_quality': ''}}
>>> list(input_dict['items'].values())
[{'popularity_7d': '1', 'bg_color': '', 'text_color': 'D2D2D2', 'ru_name': '\\u041a\\u043b\\u044e\\u0447 eSports', 'avg_price': '573.320000', 'price': '993.06', 'market_hash_name': 'eSports Key', 'buy_order': 200.02, 'ru_rarity': '\\u0431\\u0430\\u0437\\u043e\\u0432\\u043e\\u0433\\u043e \\u043a\\u043b\\u0430\\u0441\\u0441\\u0430', 'ru_quality': ''}, {'popularity_7d': '19', 'bg_color': '', 'text_color': 'D2D2D2', 'ru_name': '\\u041a\\u043b\\u044e\\u0447 \\u043e\\u0442 \\u043a\\u0435\\u0439\\u0441\\u0430 CS:GO', 'avg_price': '279.405789', 'price': '300.00', 'market_hash_name': 'CS:GO Case Key', 'buy_order': 220, 'ru_rarity': '\\u0431\\u0430\\u0437\\u043e\\u0432\\u043e\\u0433\\u043e \\u043a\\u043b\\u0430\\u0441\\u0441\\u0430', 'ru_quality': ''}]
>>> [(i['price'], i['market_hash_name']) for i in input_dict['items'].values()]
[('993.06', 'eSports Key'), ('300.00', 'CS:GO Case Key')]
>>> [{'price': i['price'], 'market_hash_name': i['market_hash_name']} for i in input_dict['items'].values()]
[{'price': '993.06', 'market_hash_name': 'eSports Key'}, {'price': '300.00', 'market_hash_name': 'CS:GO Case Key'}]
As you can see, the final two extractions give a tuple and a dictionary result with the information you need.
Convert to dataframe, then just call those 2 columns.
import pandas as pd
import json
data = '''{
"success": true,
"time": 1660441201,
"currency": "RUB",
"items": {
"186150629_143865972": {
"price": "300.00",
"buy_order": 220,
"avg_price": "279.405789",
"popularity_7d": "19",
"market_hash_name": "CS:GO Case Key",
"ru_name": "\u041a\u043b\u044e\u0447 \u043e\u0442 \u043a\u0435\u0439\u0441\u0430 CS:GO",
"ru_rarity": "\u0431\u0430\u0437\u043e\u0432\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0441\u0430",
"ru_quality": "",
"text_color": "D2D2D2",
"bg_color": ""
},
"186150630_143865972": {
"price": "993.06",
"buy_order": 200.02,
"avg_price": "573.320000",
"popularity_7d": "1",
"market_hash_name": "eSports Key",
"ru_name": "\u041a\u043b\u044e\u0447 eSports",
"ru_rarity": "\u0431\u0430\u0437\u043e\u0432\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0441\u0430",
"ru_quality": "",
"text_color": "D2D2D2",
"bg_color": ""
}
}
}'''
jsonData = json.loads(data)
df = pd.DataFrame(jsonData['items']).T
df = df[['price', 'market_hash_name']]
Output:
print(df)
price market_hash_name
186150629_143865972 300.00 CS:GO Case Key
186150630_143865972 993.06 eSports Key
Related
Is this the best way to parse a Json output from Google Ads Stream. I am parsing the json with pandas & it is taking too much time
record counts is around 700K
[{
"results": [
{
"customer": {
"resourceName": "customers/12345678900",
"id": "12345678900",
"descriptiveName": "ABC"
},
"campaign": {
"resourceName": "customers/12345678900/campaigns/12345",
"name": "Search_Google_Generic",
"id": "12345"
},
"adGroup": {
"resourceName": "customers/12345678900/adGroups/789789",
"id": "789789",
"name": "adgroup_details"
},
"metrics": {
"clicks": "500",
"conversions": 200,
"costMicros": "90000000",
"allConversionsValue": 5000.6936,
"impressions": "50000"
},
"segments": {
"device": "DESKTOP",
"date": "2022-10-28"
}
}
],
"fieldMask": "segments.date,customer.id,customer.descriptiveName,campaign.id,campaign.name,adGroup.id,adGroup.name,segments.device,metrics.costMicros,metrics.impressions,metrics.clicks,metrics.conversions,metrics.allConversionsValue",
"requestId": "fdhfgdhfgjf"
}
]
This is the sample json.I am saving the stream in json file and then reading using pandas and trying to dump in csv file
I want to convert it to CSV format, Like
with open('Adgroups.json', encoding='utf-8') as inputfile:
df = pd.read_json(inputfile)
df_new = pd.DataFrame(columns= ['Date', 'Account_ID', 'Account', 'Campaign_ID','Campaign',
'Ad_Group_ID', 'Ad_Group','Device',
'Cost', 'Impressions', 'Clicks', 'Conversions', 'Conv_Value'])
for i in range(len(df['results'])):
results = df['results'][i]
for result in results:
new_row = pd.Series({ 'Date': result['segments']['date'],
'Account_ID': result['customer']['id'],
'Account': result['customer']['descriptiveName'],
'Campaign_ID': result['campaign']['id'],
'Campaign': result['campaign']['name'],
'Ad_Group_ID': result['adGroup']['id'],
'Ad_Group': result['adGroup']['name'],
'Device': result['segments']['device'],
'Cost': result['metrics']['costMicros'],
'Impressions': result['metrics']['impressions'],
'Clicks': result['metrics']['clicks'],
'Conversions': result['metrics']['conversions'],
'Conv_Value': result['metrics']['allConversionsValue']
})
df_new = df_new.append(new_row, ignore_index = True)
df_new.to_csv('Adgroups.csv', encoding='utf-8', index=False)
Don't use df.append. It's very slow because it has to copy the dataframe over and over again. I think it's being deprecated for this reason.
You can build the rows using list comprehension before constructing the data frame:
import json
with open("Adgroups.json") as fp:
data = json.load(fp)
columns = [
"Date",
"Account_ID",
"Account",
"Campaign_ID",
"Campaign",
"Ad_Group_ID",
"Ad_Group",
"Device",
"Cost",
"Impressions",
"Clicks",
"Conversions",
"Conv_Value",
]
records = [
(
r["segments"]["date"],
r["customer"]["id"],
r["customer"]["descriptiveName"],
r["campaign"]["id"],
r["campaign"]["name"],
r["adGroup"]["id"],
r["adGroup"]["name"],
r["segments"]["device"],
r["metrics"]["costMicros"],
r["metrics"]["impressions"],
r["metrics"]["clicks"],
r["metrics"]["conversions"],
r["metrics"]["allConversionsValue"],
)
for d in data
for r in d["results"]
]
df = pd.DataFrame(records, columns=columns)
I have the following test list:
testing = [
{'score': [('a', 90)],'text': 'abc'},
{'score': [('a', 80)], 'text': 'kuku'},
{'score': [('a', 70)], 'text': 'lulu'},
{'score': [('b', 90)], 'text': 'dalu'},
{'score': [('b', 86)], 'text': 'pupu'},
{'score': [('b', 80)], 'text': 'mumu'},
{'score': [('c', 46)], 'text': 'foo'},
{'score': [('c', 26)], 'text': 'too'}
]
I would like to go through each dict, group by the score's tuple first element (a, b or c) and average the second element + collect the texts for each first element of score's tuple to get the following:
{"a": {"avg_score": 80, "texts_unique": ['abc', 'kuku', 'lulu']}, "b": the same logic... }
I have seen a pandas approach, any best practice to do this?
Try:
from statistics import mean
testing = [
{"score": [("a", 90)], "text": "abc"},
{"score": [("a", 80)], "text": "kuku"},
{"score": [("a", 70)], "text": "lulu"},
{"score": [("b", 90)], "text": "dalu"},
{"score": [("b", 86)], "text": "pupu"},
{"score": [("b", 80)], "text": "mumu"},
{"score": [("c", 46)], "text": "foo"},
{"score": [("c", 26)], "text": "too"},
]
out = {}
for d in testing:
out.setdefault(d["score"][0][0], []).append((d["score"][0][1], d["text"]))
out = {
k: {
"avg_score": mean(i for i, _ in v),
"texts_unique": list(set(i for _, i in v)),
}
for k, v in out.items()
}
print(out)
Prints:
{
"a": {"avg_score": 80, "texts_unique": ["abc", "kuku", "lulu"]},
"b": {
"avg_score": 85.33333333333333,
"texts_unique": ["mumu", "dalu", "pupu"],
},
"c": {"avg_score": 36, "texts_unique": ["foo", "too"]},
}
You can use itertools.groupby to group your data around the letter key and then use a helper function to return the desired object for each letter:
import itertools
def grouper(g):
return { 'avg_score' : sum(t['score'][0][1] for t in g)/len(g), 'texts_unique' : list(set(t['text'] for t in g)) }
res = { k : grouper(list(g)) for k, g in itertools.groupby(testing, key=lambda t:t['score'][0][0]) }
Output:
{
"a": {
"avg_score": 80.0,
"texts_unique": [
"abc",
"lulu",
"kuku"
]
},
"b": {
"avg_score": 85.33333333333333,
"texts_unique": [
"mumu",
"dalu",
"pupu"
]
},
"c": {
"avg_score": 36.0,
"texts_unique": [
"foo",
"too"
]
}
}
I have this dict:
data_flights = {
"prices": [
{ "city": "Paris", "iataCode": "AAA", "lowestPrice": 54, "id": 2 },
{ "city": "Berlin", "iataCode": "BBB", "lowestPrice": 42, "id": 3 },
{ "city": "Tokyo", "iataCode": "CCC", "lowestPrice": 485, "id": 4 },
{ "city": "Sydney", "iataCode": "DDD", "lowestPrice": 551, "id": 5 },
],
"date": "31/03/2022"
}
Can I acess a dict using a key value from one of the dics, without using for loop?
something like this:
data_flights["prices"]["city" == "Berlin"]
You can achieve this by either using a comprehension or the filter built in.
comprehension:
[e for e in d['prices'] if e['city'] == 'Berlin']
filter:
list(filter(lambda e: e['city'] == 'Berlin', d['prices']))
Both would result in:
[{'city': 'Berlin', 'iataCode': 'BBB', 'lowestPrice': 42, 'id': 3}]
You can use list comprehension
x = [a for a in data_flights["prices"] if a["city"] == "Berlin"]
>>> x
[{'city': 'Berlin', 'iataCode': 'BBB', 'lowestPrice': 42, 'id': 3}]
I am trying to write nested type of dictionary in python. I am providing my input and expected output and my tried code.
This is my input:
input = [['10', 'PS_S1U_X2_LP', 'permit', 'origin', 'igp', 'RM_S1U_X2_LP'],
['20', '', 'permit', '', '', 'RM_S1U_X2_LP'],
['10', 'MPLS-LOOPBACK', 'permit', '', '', 'MPLS-LOOPBACK-RLFA'],
]
And my desired output is:
output =
"route_policy_list": [
{
"policy_terms": [],
"route_policy_statement": [
{
"entry": "10",
"prefix_list": "PS_S1U_X2_LP",
"action_statements": [
{
"action_value": "igp",
"action": "permit",
"action_statement": "origin"
}
]
},
{
"entry": "20",
"prefix_list": "",
"action_statements": [
{
"action_value": "",
"action": "permit",
"action_statement": ""
}
]
}
],
"name": "RM_S1U_X2_LP"
},
{
"policy_terms": [],
"route_policy_statement": [
{
"entry": "10",
"prefix_list": "MPLS-LOOPBACK",
"action_statements": [
{
"action_value": "",
"action": "permit",
"action_statement": ""
}
]
}
],
"name": "MPLS-LOOPBACK-RLFA"
}
]
And I have tried this code:
from collections import defaultdict
res1 = defaultdict(list)
for fsm1 in input:
name1 = fsm1.pop()
action = fsm1[2]
action_statement = fsm1[3]
action_value = fsm1[4]
item1 = dict(zip(['entry','prefix_list'],fsm1))
res1['action'] = action
res1['action_statement'] = action_statement
res1['action_value'] = action_value
res1[name].append(item1)
print(res1)
Please help me to get desired output as mentioned above as i am new to coding and struggling to write.
Here is the final code. I used setdefault method to group the data first then used simple for loop to represent the data in requested way.
# Input
input = [['10', 'PS_S1U_X2_LP', 'permit', 'origin', 'igp', 'RM_S1U_X2_LP'],
['20', '', 'permit', '', '', 'RM_S1U_X2_LP'],
['10', 'MPLS-LOOPBACK', 'permit', '', '', 'MPLS-LOOPBACK-RLFA'],
]
# Main code
d = {}
final = []
for i in input:
d.setdefault(i[-1], []).append(i[:-1])
for i, v in d.items():
a = {}
a["policy_terms"] = []
a["route_policy_statement"] = [{"entry": j[0], "prefix_list":j[1], "action_statements":[{"action_value":j[-2], "action": j[-4], "action_statement": j[-3]}]} for j in v]
a["name"] = i
final.append(a)
final_dict = {"route_policy_list": final}
print (final_dict)
# Output
# {'route_policy_list': [{'policy_terms': [], 'route_policy_statement': [{'entry': '10', 'prefix_list': 'PS_S1U_X2_LP', 'action_statements': [{'action_value': 'origin', 'action': 'PS_S1U_X2_LP', 'action_statement': 'permit'}]}, {'entry': '20', 'prefix_list': '', 'action_statements': [{'action_value': '', 'action': '', 'action_statement': 'permit'}]}], 'name': 'RM_S1U_X2_LP'}, {'policy_terms': [], 'route_policy_statement': [{'entry': '10', 'prefix_list': 'MPLS-LOOPBACK', 'action_statements': [{'action_value': '', 'action': 'MPLS-LOOPBACK', 'action_statement': 'permit'}]}], 'name': 'MPLS-LOOPBACK-RLFA'}]}
I hope this helps and count!
It seems like every sublist in input consists of the same order of data, so I would create another list of indices such as
indices = ['entry', 'prefix_list', 'action', 'action_statement', 'action_value', 'name']
and then just hard code the values, because it seems you want specific values in specific places.
dic_list = []
for lst in input:
dic = {'policy terms' : [],
'route_policy_statements' : {
indices[0] : lst[0],
indices[1] : lst[1],
'action_statements' : {
indices[2] : lst[2],
indices[3] : lst[3],
indices[4] : lst[4]
},
indices[5] : lst[5]
}
}
dic_list.append(dic)
I have a file as follows:
using python i want to convert these multiple line to single list as below output. Please help me on this
x = {
"name": "Ken",
"age": 45,
"married": True,
"children": ("Alice", "Bob"),
"pets": [ 'Dog' ],
"cars": [
{"model": "Audi A1", "mpg": 15.1},
{"model": "Zeep Compass", "mpg": 18.1}
],
}
pdf = FPDF()
pdf.add_page()
pdf.set_font("Arial", size=12)
x.keys()
for key in x.keys():
print(key)
keys = key.strip().split("\n")
keys = list(key)
print(keys)
For this I'm getting output as below
['name']
['age']
['married']
['children']
['pets']
['cars']
expected output:
['name','age','married','children','pets','cars']
from fpdf import FPDF
x = {
"name": "Ken",
"age": 45,
"married": True,
"children": ("Alice", "Bob"),
"pets": [ 'Dog' ],
"cars": [
{"model": "Audi A1", "mpg": 15.1},
{"model": "Zeep Compass", "mpg": 18.1}
],
}
pdf = FPDF()
pdf.add_page()
pdf.set_font("Arial", size=12)
list_of_keys = [i for i in x.keys()]
print(list_of_keys)
result:
['name', ' age', 'married', 'children', 'pets', 'cars']