Is there a more efficient way to parse the following response from cloudformation
I tried for loop but I am wondering if there is a better way
response = {
'Stacks': [
{
'Outputs': [
{
'Description': 'ARN of the load balancer',
'ExportName': 'xx',
'OutputKey': 'LoadBalancerARN',
'OutputValue': 'arn:aws:elasticloadbalancing:ss'
},
{
'Description': 'ARN of the ecs service',
'ExportName': 'xxx',
'OutputKey': 'ServiceARN',
'OutputValue': 'arn:aws:ecs:5O'
},
{
'Description': 'ARN of the ecs task definition',
'ExportName': 'xxx',
'OutputKey': 'TaskDefinitionARN',
'OutputValue': 'arn:aws:ecs:xxx9'
}
]
}
]
}
def main(response):
outputs = response['Stacks'][0]['Outputs']
for output in outputs:
for k, v in output.items():
if 'OutputKey' in k and 'ServiceARN' in v:
print('ServiceARN:{}'.format(output['OutputValue']))
if 'OutputKey' in k and 'TaskDefinitionARN' in v:
print('TaskDefinitionARN:
{}'.format(output['OutputValue']))
main(response)
I trying to get ServiceARN and TaskDefinitionARN values
I found another option
import json
response = {}
def main(response):
json_string = json.dumps(response)
x = json.loads(json_string)
for i in x['Stacks'][0]['Outputs']:
if i['OutputKey'] == "ServiceARN":
print(i['OutputValue'])
main(response)
Related
I am trying to retrieve some information about my ads in google. I have done the same for campaigns, adgroups and keywords and I haven't got a single issue, however, when it comes to ads and their metrics, I cannot manage to extract something other than adCost.
I have the following code to retrieve the info I need from google ads:
def ad_data(service, view_id, start_date, end_date):
# Initialize request with first page of results
request = {
'reportRequests': [
{
'viewId': view_id,
'dateRanges': [
{
'startDate': start_date,
'endDate': end_date
}
],
'metrics': [
{
'expression': 'ga:adCost'
},
{
'expression': 'ga:adClicks'
},
{
'expression': 'ga:adImpressions'
}
],
'dimensions': [
{
'name': 'ga:date'
},
{
'name': 'ga:campaign'
},
{
'name': 'ga:adContent'
}
],
'pageSize': 1000
}
]
}
all_data = []
while True:
# Make API call and get response
response = service.reports().batchGet(body=request).execute()
report = response.get('reports', [])[0]
rows = report.get('data', {}).get('rows', [])
all_data.extend(rows)
# Check if there are more pages of results
page_token = report.get('nextPageToken', None)
if not page_token:
break
# Update request with next page of results
request['reportRequests'][0]['pageToken'] = page_token
return all_data
The issue that I have is that for some reason I'm not being ale to retrieve metrics other than adCost, hence it shows me the following error:
HttpError: <HttpError 400 when requesting https://analyticsreporting.googleapis.com/v4/reports:batchGet?alt=json returned "Unknown metric(s): ga:adImpressions
For details see https://developers.google.com/analytics/devguides/reporting/core/dimsmets.". Details: "Unknown metric(s): ga:adImpressions
For details see https://developers.google.com/analytics/devguides/reporting/core/dimsmets.">
Could anyone help me out?
Below is my jenkins groovy script to read data and write as json file.
import groovy.json.JsonOutput
def arn = ""
def name = ""
def os = ""
pipeline {
agent any
stages {
stage('Hello') {
steps {
script{
def ret = sh(script: 'aws devicefarm list-devices', returnStdout: true)
def jsonObj = readJSON text: ret
def currentVersion = "13.0"
def values = currentVersion.split('\\.')
def json_str = ""
for(String item: jsonObj.devices) {
os = item.os
if(!os.contains("\\.")) {
osV = os + ".0"
} else{
osV = os
}
def osValues = osV.split('\\.')
if(values[0].toInteger() <= osValues[0].toInteger()) {
name = item.name
def data = [
name: "$name",
os: "$os",
]
json_str += JsonOutput.toJson(data)
}
}
def json_beauty = JsonOutput.prettyPrint(json_str)
writeJSON(file: 'message124.json', json: json_beauty)
}
}
}
}
}
But here, it only saves the first value. Not all the values. Could you tell me where I am wrong here
It's not 100% clear what you actually want to end up with, but I think you want a JSON file containing the items where OS is greater than a magic number.
It's helpful to provide enough data to duplicate the problem, and eliminate everything that isn't directly related. I think what you want is something like this:
jsonObj = [
devices: [
[ os: '3', name: 'Name 1'],
[ os: '10.2', name: 'Name 10.2'],
[ os: '7', name: 'Name 7'],
[ os: '3', name: 'Name 3'],
],
]
values = ['5']
def normalizeOs(os) {
os.contains(".") ? os : "$os.0"
}
def shouldSkip(normalizedOs) {
osValues = os.split('\\.')
values[0].toInteger() > osValues[0].toInteger()
}
selected = []
for (item: jsonObj.devices) {
os = normalizeOs(item.os)
if (shouldSkip(os)) continue
selected.push([name: item.name, os: os])
}
json = new groovy.json.JsonBuilder(selected)
println(json)
Outputs:
[{"name":"Name 7","os":"7.0"},{"name":"Name 10.2","os":"10.2"}]
Have JSON received from some REST API:
{
"advertiser_id": {
"8253":{
"name":"Signify",
"id":8253
},
"2920":{
"name":"Hyundai",
"id":2920
}
}
}
I wan't to extract maps inside numbers like 8253, 2920 but without hard mapping these numbers, they can be different at any time. Anyway these numbers just duplicates id inside.
Expected output after transformation:
[
{
"name":"Signify",
"id":8253
},
{
"name":"Hyundai",
"id":2920
}
]
Tried with:
import groovy.json.*
def json = '''
'''
def p = new JsonSlurper().parseText(json)
def result = p["advertiser_id"].collectEntries{ k, v ->
[
id: v.id,
name: v.name
]
}
But it returns only one "object":
{
"id": 8905,
"name": "Spotify"
}
Also achieved wrong result with next code:
def a = p["advertiser_id"].collectMany {
it.value.collect{ k, v ->
[
id: k,
name: v
]
}
}
you want to build a list - so you need collect instead of collectEntries
def p = new JsonSlurper().parseText(json)
def result = p["advertiser_id"].collect{ k, v ->
[
id: v.id,
name: v.name
]
}
and you are not doing any transformation to nested objects - so, you could simplify the code to this:
def result = p.advertiser_id.collect{ k, v -> v }
I have a JSON file and the structure of the file is as below
[json file with the structure][1]
I am trying to get all the details into dataframe or tabular form, Tried using denormalize and could not get the actual result.
{
"body": [{
"_id": {
"s": 0,
"i": "5ea6c8ee24826b48cc560e1c"
},
"fdfdsfdsf": "V2_1_0",
"dsd": "INDIA-",
"sdsd": "df-as-3e-ds",
"dsd": 123,
"dsds": [{
"dsd": "s_10",
"dsds": [{
"dsdsd": "OFFICIAL",
"dssd": {
"dsds": {
"sdsd": "IND",
"dsads": 0.0
}
},
"sadsad": [{
"fdsd": "ABC",
"dds": {
"dsd": "INR",
"dfdsfd": -1825.717444
},
"dsss": [{
"id": "A:B",
"dsdsd": "A.B"
}
]
}, {
"name": "dssadsa",
"sadds": {
"sdsads": "INR",
"dsadsad": 180.831415
},
"xcs": "L:M",
"sds": "L.M"
}
]
}
]
}
]
}
]
}
This structure is far too nested to put directly into a dataframe. First, you'll need to use the ol' flatten_json function. This function isn't in a library (to my knowledge), but you see it around a lot. Save it somewhere.
def flatten_json(nested_json):
"""
Flatten json object with nested keys into a single level.
Args:
nested_json: A nested json object.
Returns:
The flattened json object if successful, None otherwise.
"""
out = {}
def flatten(x, name=''):
if type(x) is dict:
for a in x:
flatten(x[a], name + a + '_')
elif type(x) is list:
i = 0
for a in x:
flatten(a, name + str(i) + '_')
i += 1
else:
out[name[:-1]] = x
flatten(nested_json)
return out
Applying it to your data:
import json
with open('deeply_nested.json', r) as f:
flattened_json = flatten_json(json.load(f))
df = pd.json_normalize(flattened_json)
df.columns
Index(['body_0__id_s', 'body_0__id_i', 'body_0_schemaVersion',
'body_0_snapUUID', 'body_0_jobUUID', 'body_0_riskSourceID',
'body_0_scenarioSets_0_scenario',
'body_0_scenarioSets_0_modelSet_0_modelPolicyLabel',
'body_0_scenarioSets_0_modelSet_0_valuation_pv_unit',
'body_0_scenarioSets_0_modelSet_0_valuation_pv_value',
'body_0_scenarioSets_0_modelSet_0_measures_0_name',
'body_0_scenarioSets_0_modelSet_0_measures_0_value_unit',
'body_0_scenarioSets_0_modelSet_0_measures_0_value_value',
'body_0_scenarioSets_0_modelSet_0_measures_0_riskFactors_0_id',
'body_0_scenarioSets_0_modelSet_0_measures_0_riskFactors_0_underlyingRef',
'body_0_scenarioSets_0_modelSet_0_measures_1_name',
'body_0_scenarioSets_0_modelSet_0_measures_1_value_unit',
'body_0_scenarioSets_0_modelSet_0_measures_1_value_value',
'body_0_scenarioSets_0_modelSet_0_measures_1_riskFactors',
'body_0_scenarioSets_0_modelSet_0_measures_1_underlyingRef'],
dtype='object')
I would like to tag the host that I am spining up using boto3 python api
response = client.allocate_hosts(
AutoPlacement='on'|'off',
AvailabilityZone='string',
ClientToken='string',
InstanceType='string',
Quantity=123,
TagSpecifications=[
{
'ResourceType': 'dedicated-host',
'Tags': [
{
'Key': 'string',
'Value': 'string'
},
]
},
])
Here is what I am doing
Availability Zone,Instance Type , Quantity are parameterized and I use dictionary to input data
count = 10
input_dict = {}
input_dict['AvailabilityZone'] = 'us-east-1a'
input_dict['InstanceType'] = 'c5.large'
input_dict['Quantity'] = int(count)
instance = client.allocate_hosts(**input_dict,)
print(str(instance))
This code works for me but i need to tag the resource too
TagSpecifications=[
{
'ResourceType': 'customer-gateway'|'dedicated-host'|'dhcp-options'|'elastic-ip'|'fleet'|'fpga-image'|'image'|'instance'|'internet-gateway'|'launch-template'|'natgateway'|'network-acl'|'network-interface'|'reserved-instances'|'route-table'|'security-group'|'snapshot'|'spot-instances-request'|'subnet'|'transit-gateway'|'transit-gateway-attachment'|'transit-gateway-route-table'|'volume'|'vpc'|'vpc-peering-connection'|'vpn-connection'|'vpn-gateway',
'Tags': [
{
'Key': 'string',
'Value': 'string'
},
]
},
]
how can I input that into the dictionary .. It seems like tag specification has dictionary inside dict .. I am making syntax errors. I tried the below code without success.
input_dict['TagSpecifications'] = [{'ResourceType':'dedicated-host','Tags':[{'key':'Name','Value':'demo'},]},]
The easiest way is to simply pass values directly:
response = client.allocate_hosts(
AvailabilityZone='us-east-1a',
InstanceType='c5.large',
Quantity=10,
TagSpecifications=[
{
'ResourceType': 'dedicated-host',
'Tags': [
{
'Key': 'Name',
'Value': 'Demo'
}
]
}
])