Unique nested dictionary from the 'for' loop Python3 - python-3.x

I've got the host which executes commands via 'subprocess' and gets output list of several parameters. The problem is that output can not be correctly modified to be translated to the dictionary, whether it's yaml or json. After list is received Regexp takes part to match valuable information and to perform grouping. I am interested in getting a unique dictionary, where crossing keys are put into nested dictionary.
Here is the code and the example of output list:
from re import compile,match
# Output can differ from request to request, the "keys" from the #
# list_of_values can dublicate or appear more than two times. The values #
# mapped to the keys can differ too. #
list_of_values = [
"paramId: '11'", "valueId*: '11'",
"elementId: '010_541'", 'mappingType: Both',
"startRng: ''", "finishRng: ''",
'DbType: sql', "activeSt: 'false'",
'profile: TestPr1', "specificHost: ''",
'hostGroup: tstGroup10', 'balance: all',
"paramId: '194'", "valueId*: '194'",
"elementId: '010_541'", 'mappingType: Both',
"startRng: '1020304050'", "finishRng: '1020304050'",
'DbType: sql', "activeSt: 'true'",
'profile: TestPr1', "specificHost: ''",
'hostGroup: tstGroup10', 'balance: all']
re_compile_valueId = compile(
"valueId\*:\s.(?P<valueId>\d{1,5})"
"|elementId:\s.(?P<elementId>\d{3}_\d{2,3})"
"|startRng:\s.(?P<startRng>\d{1,10})"
"|finishRng:\s.(?P<finishRng>\d{1,10})"
"|DbType:\s(?P<DbType>nosql|sql)"
"|activeSt:\s.(?P<activeSt>true|false)"
"|profile:\s(?P<profile>[A-z0-9]+)"
"|hostGroup:\s(?P<hostGroup>[A-z0-9]+)"
"|balance:\s(?P<balance>none|all|priority group)"
)
iterator_loop = 0
uniq_dict = dict()
next_dict = dict()
for element in list_of_values:
match_result = match(re_compile_valueId,element)
if match_result:
temp_dict = match_result.groupdict()
for key, value in temp_dict.items():
if value:
if key == 'valueId':
uniq_dict['valueId'+str(iterator_loop)] = ''
iterator_loop +=1
next_dict.update({key: value})
else:
next_dict.update({key: value})
uniq_dict['valueId'+str(iterator_loop-1)] = next_dict
print(uniq_dict)
This code right here responses with:
{
'valueId0':
{
'valueId': '194',
'elementId': '010_541',
'DbType': 'sql',
'activeSt': 'true',
'profile': 'TestPr1',
'hostGroup': 'tstGroup10',
'balance': 'all',
'startRng': '1020304050',
'finishRng': '1020304050'
},
'valueId1':
{
'valueId': '194',
'elementId': '010_541',
'DbType': 'sql',
'activeSt': 'true',
'profile': 'TestPr1',
'hostGroup': 'tstGroup10',
'balance': 'all',
'startRng': '1020304050',
'finishRng': '1020304050'
}
}
And I was waiting for something like:
{
'valueId0':
{
'valueId': '11',
'elementId': '010_541',
'DbType': 'sql',
'activeSt': 'false',
'profile': 'TestPr1',
'hostGroup': 'tstGroup10',
'balance': 'all',
'startRng': '',
'finishRng': ''
},
'valueId1':
{
'valueId': '194',
'elementId': '010_541',
'DbType': 'sql',
'activeSt': 'true',
'profile': 'TestPr1',
'hostGroup': 'tstGroup10',
'balance': 'all',
'startRng': '1020304050',
'finishRng': '1020304050'
}
}
I've also got another code below, which runs and puts values as expected. But the structure breaks the idea of having this all looped around, because each dictionary result key has its own order number mapped. The example below. The list_of_values and re_compile_valueId can be used from previous example.
for element in list_of_values:
match_result = match(re_compile_valueId,element)
if match_result:
temp_dict = match_result.groupdict()
for key, value in temp_dict.items():
if value:
if key == 'balance':
key = key+str(iterator_loop)
uniq_dict.update({key: value})
iterator_loop +=1
else:
key = key+str(iterator_loop)
uniq_dict.update({key: value})
print(uniq_dict)
The output will look like:
{
'valueId1': '11', 'elementId1': '010_541',
'DbType1': 'sql', 'activeSt1': 'false',
'profile1': 'TestPr1', 'hostGroup1': 'tstGroup10',
'balance1': 'all', 'valueId2': '194',
'elementId2': '010_541', 'startRng2': '1020304050',
'finishRng2': '1020304050', 'DbType2': 'sql',
'activeSt2': 'true', 'profile2': 'TestPr1',
'hostGroup2': 'tstGroup10', 'balance2': 'all'
}
Would appreciate any help! Thanks!

It appeared that some documentation reading needs to be performed :D
The copy() of the next_dict under else statement needs to be applied. Thanks to thread:
Why does updating one dictionary object affect other?
Many thanks to answer's author #thefourtheye (https://stackoverflow.com/users/1903116/thefourtheye)
The final code:
for element in list_of_values:
match_result = match(re_compile_valueId,element)
if match_result:
temp_dict = match_result.groupdict()
for key, value in temp_dict.items():
if value:
if key == 'valueId':
uniq_dict['valueId'+str(iterator_loop)] = ''
iterator_loop +=1
next_dict.update({key: value})
else:
next_dict.update({key: value})
uniq_dict['valueId'+str(iterator_loop-1)] = next_dict.copy()
Thanks for the involvement to everyone.

Related

I am trying to get the output in below nested dictionary format for the given input

I am trying to get the output in the form of nested dictionary. I am practicing this for python automation for Network simulator. I tried but not able to achieve. Kindly someone help me on this.
Interface IP-Address OK? Method Status Protocol
Vlan1 unassigned YES NVRAM up up
Vlan30 30.1.1.2 YES NVRAM up up
Vlan306 192.168.25.3 YES NVRAM up down
GigabitEthernet0/0 11.19.17.19 YES NVRAM up up
Te1/0/3 unassigned YES unset up up
Te1/0/21 unassigned YES unset up up
Te1/0/35 unassigned YES unset up up
output should be in below format.
{'interface': {
'vlan1': {
'name': 'vlan1',
'ip': 'unassigned',
'ok_status': 'YES',
'method': 'NVRAM',
'status': 'up',
'protocol': 'up',
},
'vlan30': {
'name': 'vlan30',
'ip': '30.1.1.2',
'ok_status': 'YES',
'method': 'NVRAM',
'status': 'up',
'protocol': 'up',
}
}
}
If you place your input data in a text file (example D:/test.txt), following code will format it according to your specification:
comma_sep = ""
with open("D:/test.txt") as f:
for line in f:
parts = [part for part in line.split(' ') if part != '']
comma_sep += ','.join(parts)
lines = [item for item in comma_sep.split('\n')]
output_dict = {"interface": {}}
for index, line in enumerate(lines):
if index == 0:
continue
# Compose each interface
interface = {}
for item_index, item in enumerate(line.split(',')):
if item_index == 0:
item = item.lower()
interface = {item: {}}
interface[item].update({'name': item})
main_key = item
elif item_index == 1:
interface[main_key].update({'ip': item})
elif item_index == 2:
interface[main_key].update({'ok_status': item})
elif item_index == 3:
interface[main_key].update({'method': item})
elif item_index == 4:
interface[main_key].update({'status': item})
elif item_index == 5:
interface[main_key].update({'protocol': item})
output_dict["interface"].update(interface)
print(output_dict)

API to pull the list of supported languages in AWS Translate

I am currently working on a project where I need to translate Customer comments into English from the source language on AWS. It is easy to do so using AWS Translate but before I call translate API to translate text into English, I want to check whether the source language is supported by AWS or not?
One solution is to put all the language codes supported by AWS Translate into a list and then check source language against this list. This is easy but it is going to be messy and I want to make it more dynamic.
So, I am thinking to code like this
import boto3
def translateUserComment(source_language):
translate = boto3.client(service_name='translate', region_name='region', use_ssl=True)
languages_supported = tanslate.<SomeMethod>()
if source_language in languages_supported:
result = translate.translate_text(Text="Hello, World",
SourceLanguageCode=source_language, TargetLanguageCode="en")
print('TranslatedText: ' + result.get('TranslatedText'))
print('SourceLanguageCode: ' + result.get('SourceLanguageCode'))
print('TargetLanguageCode: ' + result.get('TargetLanguageCode'))
else:
print("The source language is not supported by AWS Translate")
Problem is that I am not able to find out any API call to get the list of languages/language codes supported by AWS Translate for place.
Before I posted this question,
I have tried to search for similar questions on stackoverflow
I have gone through the AWS Translate Developer guide but still no luck
Any suggestion/ redirection to the right approach is highly appreciated.
Currently there is no API for this service, although this code would work, in this code a class is created Translate_lang with all the language codes and country wise
from here-> https://docs.aws.amazon.com/translate/latest/dg/what-is.html
, you can call this class into your program and use it by creating an instance of the class:
translate_lang_check.py
class Translate_lang:
def __init__(self):
self.t_lang = {'Afrikaans': 'af', 'Albanian': 'sq', 'Amharic': 'am',
'Arabic': 'ar', 'Armenian': 'hy', 'Azerbaijani': 'az',
'Bengali': 'bn', 'Bosnian': 'bs', 'Bulgarian': 'bg',
'Catalan': 'ca', 'Chinese (Simplified)': 'zh',
'Chinese (Traditional)': 'zh-TW', 'Croatian': 'hr',
'Czech': 'cs', 'Danish': 'da ', 'Dari': 'fa-AF',
'Dutch': 'nl ', 'English': 'en', 'Estonian': 'et',
'Farsi (Persian)': 'fa', 'Filipino Tagalog': 'tl',
'Finnish': 'fi', 'French': 'fr', 'French (Canada)': 'fr-CA',
'Georgian': 'ka', 'German': 'de', 'Greek': 'el', 'Gujarati': 'gu',
'Haitian Creole': 'ht', 'Hausa': 'ha', 'Hebrew': 'he ', 'Hindi': 'hi',
'Hungarian': 'hu', 'Icelandic': 'is', 'Indonesian': 'id ', 'Italian': 'it',
'Japanese': 'ja', 'Kannada': 'kn', 'Kazakh': 'kk', 'Korean': 'ko',
'Latvian': 'lv', 'Lithuanian': 'lt', 'Macedonian': 'mk', 'Malay': 'ms',
'Malayalam': 'ml', 'Maltese': 'mt', 'Mongolian': 'mn', 'Norwegian': 'no',
'Persian': 'fa', 'Pashto': 'ps', 'Polish': 'pl', 'Portuguese': 'pt',
'Romanian': 'ro', 'Russian': 'ru', 'Serbian': 'sr', 'Sinhala': 'si',
'Slovak': 'sk', 'Slovenian': 'sl', 'Somali': 'so', 'Spanish': 'es',
'Spanish (Mexico)': 'es-MX', 'Swahili': 'sw', 'Swedish': 'sv',
'Tagalog': 'tl', 'Tamil': 'ta', 'Telugu': 'te', 'Thai': 'th',
'Turkish': 'tr', 'Ukrainian': 'uk', 'Urdu': 'ur', 'Uzbek': 'uz',
'Vietnamese': 'vi', 'Welsh': 'cy'}
def check_lang_in_translate(self, given_country):
if given_country in self.t_lang:
return self.t_lang[given_country]
else:
return None
def check_lang_code_in_translate(self, given_lang):
if given_lang in list(self.t_lang.values()):
return True
else:
return False
You can call and check your lang codes using the methods of this class:
from translate_lang_check import Translate_lang
tl = Translate_lang()
print(tl.check_lang_code_in_translate('en'))

pythonic way deal with DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.?

redis veersion 3.4.1
must be use hash, can't use str or other data type
data:
{'_anno': {
'ctp': 'list',
'dt': [],
'ml': 0,
'na': 'apple',
'pos': -1,
'rel': '',
'st_var': '',
'tp': 'object'},
'_att': {'_cuser': 'apple card',
'_last_editor': 'apple card',
'_protext': 'authorize',
'_status': 'normal',
'_theme_id': 'apple card',
'_view': '12'},
}
my code
pool = redis.ConnectionPool(host=host, port=port)
conn = redis.StrictRedis(connection_pool=pool)
conn.hmset("aaaaaa",data)
raise error
DataError: Invalid input of type: 'dict'. Convert to a bytes, string,
int or float first.
now code
pool = redis.ConnectionPool(host=host, port=port)
conn = redis.StrictRedis(connection_pool=pool)
new_data={}
for key,value in data.items():
new_data[key]=json.dumps(value)
conn.hmset("aaaaaa",new_data)
Is there a more pythonic way?
The solution for you problem is to use hexdigest() or digest() to convert your dictionary, you can use that:
hashlib.sha256(mdp.encode()).hexdigest()

Getting tags from python 3 dictionary object. AWS Boto3 Python 3

I have a dictionary object that is being returned to me from AWS. I need to pull the tag "based_on_ami" out of this dictionary. I have tried converting to a list, but I am new to programming and have not been able to figure out how to access Tags since they are a few levels down in the dictionary.
What is the best way for me to pull that tag out of the dictionary and put it into a variable i can use?
{
'Images':[
{
'Architecture':'x86_64',
'CreationDate':'2017-11-27T14:41:30.000Z',
'ImageId':'ami-8e73e0f4',
'ImageLocation':'23452345234545/java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5',
'ImageType':'machine',
'Public':False,
'OwnerId':'23452345234545',
'State':'available',
'BlockDeviceMappings':[
{
'DeviceName':'/dev/sda1',
'Ebs':{
'Encrypted':False,
'DeleteOnTermination':True,
'SnapshotId':'snap-0c10e8f5ced5b5240',
'VolumeSize':8,
'VolumeType':'gp2'
}
},
{
'DeviceName':'/dev/sdb',
'VirtualName':'ephemeral0'
},
{
'DeviceName':'/dev/sdc',
'VirtualName':'ephemeral1'
}
],
'EnaSupport':True,
'Hypervisor':'xen',
'Name':'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5',
'RootDeviceName':'/dev/sda1',
'RootDeviceType':'ebs',
'SriovNetSupport':'simple',
'Tags':[
{
'Key':'service',
'Value':'baseami'
},
{
'Key':'cloudservice',
'Value':'ami'
},
{
'Key':'Name',
'Value':'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5'
},
{
'Key':'os',
'Value':'ubuntu 16.04 lts'
},
{
'Key':'based_on_ami',
'Value':'ami-aa2ea8d0'
}
],
'VirtualizationType':'hvm'
}
],
'ResponseMetadata':{
'RequestId':'2c376c75-c31f-4aba-a058-173f3b125a00',
'HTTPStatusCode':200,
'HTTPHeaders':{
'content-type':'text/xml;charset=UTF-8',
'transfer-encoding':'chunked',
'vary':'Accept-Encoding',
'date':'Fri, 01 Dec 2017 18:17:53 GMT',
'server':'AmazonEC2'
},
'RetryAttempts':0
}
}
The best way to approach this type of problem is to find the value you're looking for, and then work outwards until you find a solution. You need to look at what is at each of those levels.
So, what are you looking for? You're looking for the Value for based_on_ami's Key. So your final step is going to be:
if obj['Key'] == 'based_on_ami':
# do something with obj['Value'].
But how do you get there? Well, the object is inside of a list, so you'll need to iterate the list:
for tag in <some list>:
if tag['Key'] == 'based_on_ami':
# do something with tag['Value'].
What is that list? It's the list of tags:
for tag in image['Tags']:
if tag['Key'] == 'based_on_ami':
# do something with tag['Value'].
And where are those tags? In an image object that you find in a list:
for image in image_list:
for tag in image['Tags']:
if tag['Key'] == 'based_on_ami':
# do something with tag['Value'].
The image list is the value found at the Images key in your initial dict.
image_list = my_data['Images']
for image in image_list:
for tag in image['Tags']:
if tag['Key'] == 'based_on_ami':
# do something with tag['Value'].
And now you're collecting all of those values, so you'll need a list and you'll need to append to it:
result = []
image_list = my_data['Images']
for image in image_list:
for tag in image['Tags']:
if tag['Key'] == 'based_on_ami':
result.append(tag['Value'])
So, I took your example above, and added another based_on_ami node with the value quack:
{'ResponseMetadata': {'RequestId': '2c376c75-c31f-4aba-a058-173f3b125a00', 'RetryAttempts': 0, 'HTTPHeaders': {'vary': 'Accept-Encoding', 'transfer-encoding': 'chunked', 'server': 'AmazonEC2', 'content-type': 'text/xml;charset=UTF-8', 'date': 'Fri, 01 Dec 2017 18:17:53 GMT'}, 'HTTPStatusCode': 200}, 'Images': [{'Public': False, 'CreationDate': '2017-11-27T14:41:30.000Z', 'BlockDeviceMappings': [{'Ebs': {'SnapshotId': 'snap-0c10e8f5ced5b5240', 'VolumeSize': 8, 'Encrypted': False, 'VolumeType': 'gp2', 'DeleteOnTermination': True}, 'DeviceName': '/dev/sda1'}, {'VirtualName': 'ephemeral0', 'DeviceName': '/dev/sdb'}, {'VirtualName': 'ephemeral1', 'DeviceName': '/dev/sdc'}], 'OwnerId': '23452345234545', 'ImageLocation': '23452345234545/java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'RootDeviceName': '/dev/sda1', 'ImageType': 'machine', 'Hypervisor': 'xen', 'RootDeviceType': 'ebs', 'State': 'available', 'Architecture': 'x86_64', 'Name': 'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'Tags': [{'Value': 'baseami', 'Key': 'service'}, {'Value': 'ami', 'Key': 'cloudservice'}, {'Value': 'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'Key': 'Name'}, {'Value': 'ubuntu 16.04 lts', 'Key': 'os'}, {'Value': 'ami-aa2ea8d0', 'Key': 'based_on_ami'}], 'EnaSupport': True, 'SriovNetSupport': 'simple', 'ImageId': 'ami-8e73e0f4'}, {'Public': False, 'CreationDate': '2017-11-27T14:41:30.000Z', 'BlockDeviceMappings': [{'Ebs': {'SnapshotId': 'snap-0c10e8f5ced5b5240', 'VolumeSize': 8, 'Encrypted': False, 'VolumeType': 'gp2', 'DeleteOnTermination': True}, 'DeviceName': '/dev/sda1'}, {'VirtualName': 'ephemeral0', 'DeviceName': '/dev/sdb'}, {'VirtualName': 'ephemeral1', 'DeviceName': '/dev/sdc'}], 'VirtualizationType': 'hvm', 'OwnerId': '23452345234545', 'ImageLocation': '23452345234545/java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'RootDeviceName': '/dev/sda1', 'ImageType': 'machine', 'Hypervisor': 'xen', 'RootDeviceType': 'ebs', 'State': 'available', 'Architecture': 'x86_64', 'Name': 'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'Tags': [{'Value': 'baseami', 'Key': 'service'}, {'Value': 'ami', 'Key': 'cloudservice'}, {'Value': 'java8server_ubuntu16-2b71edd1-f95e-4ee5-8fd6-d8a46975fdb5', 'Key': 'Name'}, {'Value': 'ubuntu 16.04 lts', 'Key': 'os'}, {'Value': 'quack', 'Key': 'based_on_ami'}], 'EnaSupport': True, 'SriovNetSupport': 'simple', 'ImageId': 'ami-8e73e0f4'}]}
My result:
['ami-aa2ea8d0', 'quack']
info = {...}
tags = []
for image in info['Images']:
for tag in image['Tags']:
if tag['Key'] == 'based_on_ami':
tags.append(tag['Value'])
print(tags)

Export nested dictionary to csv

I need to export a nested dictionary to CSV. Here's what each entry looks like (that needs to be one line in the csv later):
{'createdTime': '2017-10-30T12:33:02.000Z',
'fields': {'Date': '2017-10-30T12:32:56.000Z',
'field1': 'example#gmail.com',
'field2': 1474538185964188,
'field3': 6337,
....},
'id': 'reca7LBr64XM1ClWy'}
I think I need to iterate through the dictionary and create a list of lists(?) to create the csv from using the csv module.
['Date', 'field1', 'field2', 'field3', ...],
['2017-10-30T12:32:56.000Z', 'example#gmail.com', 1474538185964188, 6337 ...]
My problem is to find a smart way to iterate through the dict to get to a list like this.
You can get the values in the below way:
def process_data():
csv_data = [{'createdTime': '2017-10-30T12:33:02.000Z',
'fields': {'Date': '2017-10-30T12:32:56.000Z',
'field1': 'example#gmail.com',
'field2': 1474538185964188,
'field3': 6337},
'id': 'reca7LBr64XM1ClWy'},
{'createdTime': '2017-10-30T12:33:02.000Z',
'fields': {'Date': '2017-10-30T12:32:56.000Z',
'field1': 'example#gmail.com',
'field2': 1474538185964188,
'field3': 6337},
'id': 'reca7LBr64XM1ClWy'}]
headers = [key for key in csv_data[0]['fields'].keys()]
body = []
for row in csv_data:
body_row = []
for colomn_header in headers:
body_row.append(row['fields'][colomn_header])
body.append(body_row)
process_data()
#header -- ['Date', 'field1', 'field2', 'field3']
#body -- [['2017-10-30T12:32:56.000Z', 'example#gmail.com',
# 1474538185964188, 6337], ['2017-10-30T12:32:56.000Z',
# 'example#gmail.com', 1474538185964188, 6337]]

Resources