I have a JSON file which looks like below. I am getting the parameters for name and product_version. Using both of them I need to get the relese_version and latest boolean value.
eg:- If var1 = section1 var2 = 2.6.0 then the release_version should be taken as 2.6.0.9 and latest as false in groovy.
file.json
{
"platforms": [
{
"name": "section1",
"versions": [
{
"product_version": "2.6.0",
"release_version": "2.6.0.9",
"latest": false
},
{
"product_version": "3.0.0",
"release_version": "3.0.0.3",
"latest": false
}
]
},
{
"name": "section2",
"versions": [
{
"product_version": "2.6.0",
"release_version": "2.6.0.9",
"latest": false
},
{
"product_version": "3.0.0",
"release_version": "3.0.0.3",
"latest": false
}
]
}
]
}
This is the code snippet I tried out.
filename = "file.json"
def jsonSlurper = new JsonSlurper()
parsed_json = jsonSlurper.parse(new File(filename))
release_tag = json.parsed_json.find {platforms.name == "section1".version[].product_version == "2.6.0".release_version}
println release_tag
But this didn't work. Please help me with this
You first have to find the platform by name (which could fail); next
find in the versions the product version. E.g.
def data = new groovy.json.JsonSlurper().parse("data.json" as File)
def name='section1'
def productVersion = '2.6.0'
// XXX
def result = data.platforms.find{ it.name == name }?.versions?.find{ it.product_version == productVersion }
assert result.release_version == '2.6.0.9'
assert result.latest == false
Note the use of the "elvis operator" after the first find to
short-circuit.
If you have to do many such lookups on the same data file, it might make
sense to shape the data into a better form for the lookup you are doing
(e.g. turn that into maps of maps for your two lookup keys)
Related
I am in no way an expert with groovy so please don't hold that against me.
I have JSON that looks like this:
{
"metrics": [
{
"name": "metric_a",
"help": "This tracks your A stuff.",
"type": "GAUGE",
"labels": [
"pool"
],
"unit": "",
"aggregates": [],
"meta": [
{
"category": "CAT A",
"deployment": "environment-a"
}
],
"additional_notes": "Some stuff (potentially)"
},
...
]
...
}
I'm using it as a source for automated documentation of all the metrics. So, I'm iterating through it in various ways to get the information I need. So far so good, I'm most of the way there. The problem is this all needs to be organized per the deployment environment. Meaning, multiple metrics will share the same value for deployment.
My thought was I could create a map with deployment as the key and the metric name for any metric that has a matching deployment as the value. Once I have that map, it should be easy for me to organize things the way they should be. I can't figure out how to do that. The result is all the metric names are added which is expected since I'm not doing anything to filter them out. I was thinking that groupBy would make sense here but I can't figure out how to use it effectively and frankly I'm not sure it will solve my problem by itself. Here is my code so far:
parentChild = [:]
children = []
metrics.each { metric ->
def metricName = metric.name
def depName = metric.meta.findResult{ it.deployment }
children.add(metricName)
parentChild.put(depName, children)
}
What is the best way to create a new map where the values for each key are based off a specific condition?
EDIT: The desired result would be each key in the resulting map would be a unique deployment value from all the metrics (as a string). Each value would be name of each metric that contains that deployment (as an array).
[environment-a:
[metric_a,metric_b,metric_c,...],
environment-b:
[metric_d,metric_e,metric_f,...]
...]
I would use a combo of withDefault() to pre-fill each map-entry value with a fresh TreeSet-instance (sorted no-duplicates set) and standard inject().
I reduced your sample data to the bare minimum and added some new nodes:
import groovy.json.*
String input = '''\
{
"metrics": [
{
"name": "metric_a",
"meta": [
{
"deployment": "environment-a"
}
]
},
{
"name": "metric_b",
"meta": [
{
"deployment": "environment-a"
}
]
},
{
"name": "metric_c",
"meta": [
{
"deployment": "environment-a"
},
{
"deployment": "environment-b"
}
]
},
{
"name": "metric_d",
"meta": [
{
"deployment": "environment-b"
}
]
}
]
}'''
def json = new JsonSlurper().parseText input
def groupedByDeployment = json.metrics.inject( [:].withDefault{ new TreeSet() } ){ res, metric ->
metric.meta.each{ res[ it.deployment ] << metric.name }
res
}
assert groupedByDeployment.toString() == '[environment-a:[metric_a, metric_b, metric_c], environment-b:[metric_c, metric_d]]'
If your metrics.meta array is supposed to have a single value, you can simplify the code by replacing the line:
metric.meta.each{ res[ it.deployment ] << metric.name }
with
res[ metric.meta.first().deployment ] << metric.name
I have a json as below and I am trying to get the list of all the "dev" hosts using Groovy.
How can I do that ? I have attached a sample code that I am using currently, but that does not work for obvious reasons. I am new to Groovy.
{
"app1": {
"dev": [
"host1",
"host2"
],
"qa": null,
"uat": [
"host11"
]
},
"app2": {
"qa": null,
"stable": null,
"dev": [
"host3",
"host4"
]
}
CODE:
apiResponse = <Code which returns the json as mentioned above>
def parser = new JsonSlurper()
def host_list = parser.parseText(apiResponse)
dev_hosts = host_list[]['dev']
print dev_hosts
Expected Result:
['host1','host2','host3','host4']
There are any number of ways to do it and the best way may depend on knowing some more about the common usage patterns with respect to how many objects are in the top level JSON document etc, but the following does function:
String jsonInput = """
{
"app1": {
"dev": [
"host1",
"host2"
],
"qa": null,
"uat": [
"host11"
]
},
"app2": {
"qa": null,
"stable": null,
"dev": [
"host3",
"host4"
]
}
}"""
def object = new JsonSlurper().parseText(jsonInput)
def results = object.collect { it.value.dev }.flatten()
assert results == ['host1', 'host2', 'host3', 'host4']
def dev_hosts = host_list.values().collectMany{it['dev']}
edit: OP adjusted the specs: dev should be a variable and the lists
need to be flattened (at first level presumably).
I have a JSON file and the structure of the file is as below
[json file with the structure][1]
I am trying to get all the details into dataframe or tabular form, Tried using denormalize and could not get the actual result.
{
"body": [{
"_id": {
"s": 0,
"i": "5ea6c8ee24826b48cc560e1c"
},
"fdfdsfdsf": "V2_1_0",
"dsd": "INDIA-",
"sdsd": "df-as-3e-ds",
"dsd": 123,
"dsds": [{
"dsd": "s_10",
"dsds": [{
"dsdsd": "OFFICIAL",
"dssd": {
"dsds": {
"sdsd": "IND",
"dsads": 0.0
}
},
"sadsad": [{
"fdsd": "ABC",
"dds": {
"dsd": "INR",
"dfdsfd": -1825.717444
},
"dsss": [{
"id": "A:B",
"dsdsd": "A.B"
}
]
}, {
"name": "dssadsa",
"sadds": {
"sdsads": "INR",
"dsadsad": 180.831415
},
"xcs": "L:M",
"sds": "L.M"
}
]
}
]
}
]
}
]
}
This structure is far too nested to put directly into a dataframe. First, you'll need to use the ol' flatten_json function. This function isn't in a library (to my knowledge), but you see it around a lot. Save it somewhere.
def flatten_json(nested_json):
"""
Flatten json object with nested keys into a single level.
Args:
nested_json: A nested json object.
Returns:
The flattened json object if successful, None otherwise.
"""
out = {}
def flatten(x, name=''):
if type(x) is dict:
for a in x:
flatten(x[a], name + a + '_')
elif type(x) is list:
i = 0
for a in x:
flatten(a, name + str(i) + '_')
i += 1
else:
out[name[:-1]] = x
flatten(nested_json)
return out
Applying it to your data:
import json
with open('deeply_nested.json', r) as f:
flattened_json = flatten_json(json.load(f))
df = pd.json_normalize(flattened_json)
df.columns
Index(['body_0__id_s', 'body_0__id_i', 'body_0_schemaVersion',
'body_0_snapUUID', 'body_0_jobUUID', 'body_0_riskSourceID',
'body_0_scenarioSets_0_scenario',
'body_0_scenarioSets_0_modelSet_0_modelPolicyLabel',
'body_0_scenarioSets_0_modelSet_0_valuation_pv_unit',
'body_0_scenarioSets_0_modelSet_0_valuation_pv_value',
'body_0_scenarioSets_0_modelSet_0_measures_0_name',
'body_0_scenarioSets_0_modelSet_0_measures_0_value_unit',
'body_0_scenarioSets_0_modelSet_0_measures_0_value_value',
'body_0_scenarioSets_0_modelSet_0_measures_0_riskFactors_0_id',
'body_0_scenarioSets_0_modelSet_0_measures_0_riskFactors_0_underlyingRef',
'body_0_scenarioSets_0_modelSet_0_measures_1_name',
'body_0_scenarioSets_0_modelSet_0_measures_1_value_unit',
'body_0_scenarioSets_0_modelSet_0_measures_1_value_value',
'body_0_scenarioSets_0_modelSet_0_measures_1_riskFactors',
'body_0_scenarioSets_0_modelSet_0_measures_1_underlyingRef'],
dtype='object')
I'm trying to generate JSON body dynamically using values in csv file. For this i'm making use of JSR223 PreProcessor with groovy script.
I'm expecting the below format to be generate when i run the groovy script
{
"transactionId": "100",
"lineItems": [{
"lineItemNo": "1",
"cardInfo": {
"cardNumber": "3456"
}
},
{
"lineItemNo": "2",
"cardInfo": {
"cardNumber": "45698"
}
}
]
}
but when i execute script i'm getting below format
POST data:
{
"transactionId": "100",
"lineItems": [
{
"lineItemNo": "1",
"Cardinfo": [
9255000012794606,
9255000012794645
]
},
{
"lineItemNo": "1",
"Cardinfo": [
9255000012794606,
9255000012794645
]
}
]
}
Script to generate json body
File csvFile = new File("D:\\Project Related Docs\\Jmeter\\apache-jmeter-5.0\\bin\\Map_Performance\\Map_New_Auto_jmx\\2Cards.csv")
def cards = csvFile.readLines()
List<String> cardnumbmer = new ArrayList<>()
def counter = 1
cards.eachWithIndex{line,idx->cardnumbmer.add(line)}
log.info("size of csv = "+cardnumbmer.size())
log.info("File conents = "+cardnumbmer[0])
//build the json body
def ids = new groovy.json.JsonSlurper().parseText(cardnumbmer.toString())
log.info("cardnumbmer to string = "+cardnumbmer.toString())
def builder = new groovy.json.JsonBuilder()
builder([
transactionId:"100",
lineItems:ids.collect{[lineItemNo:"1",Cardinfo: ids.collect{carnumber: it}]}
])
//sampler.getArguments().removeAllArguments()
sampler.addNonEncodedArgument('',builder.toPrettyString(),'')
sampler.setPostBodyRaw(true);
--CSV FILE have cardnumbers listed in row-wise look like below
9255000012794606
9255000012794645
Request to help me to know how to fix this issue.
ids.collect{carnumber: it} is basically ids. Be explicit about returning the map: ids.collect{ [carnumber: it] }
Below code resolved the problem
//build the json body
def ids = new groovy.json.JsonSlurper().parseText(cardnumbmer.toString())
log.info("cardnumbmer to string = "+cardnumbmer.toString())
def builder = new groovy.json.JsonBuilder()
def count = 1
builder([
transactionId:"100",
//lineItems:ids.collect{[lineItemNo:"1",Cardinfo: count.collect{[carnumber: it]}]}
lineItems:ids.collect{[lineItemNo:count++,Cardinfo: [Cardnumber:it]]}
])
//sampler.getArguments().removeAllArguments()
sampler.addNonEncodedArgument('',builder.toPrettyString(),'')
sampler.setPostBodyRaw(true);
I am new to Python. I am writhing a code to generate a excel file having the data sourced by calling API and correlate those to get desired result.
basically taking input from one database and search that in others and fetch related information.
The 4 databases have below data :
EEp
---------------------
{u'data': [{u'_id': u'5c30702c8ca9f51da8178df4',
u'encap': u'vlan-24',
u'ip': u'7.12.12.16',
u'mac': u'5B:P9:01:9E:42:08'}]}
PathEp
-----------
{u'data': [{u'_id': u'5c54a81a8ca9f51da84ae08e',
u'paths': u'paths-1507',
u'endpoint': u'eth1/10',
u'cep': u'5B:P9:01:9E:42:08',
u'tenant': u'ESX'}]}
ip4_address
-----------------------
{u'data': [{u'Allocation': u'Build_Reserved',
u'address': u'7.12.12.16',
u'name': u'fecitrix-1',
u'state': u'RESERVED'}]}
asset
---------------
{u'data': [{u'_id': u'57ccce8110dd54f02881fedc',
u'client': u'CES',
u'hostname': u'fecitrix-1'
u'os_team': u'Window'}]}
Logic:
If "mac" of EEp and "cep" of PathEp is same than take "encap","ip" ,"mac"
"paths" ,'endpoint","cep" and "tenant" (these values need to be exported
to excel)
Take ip of EEp and search in "ip4_address"
and get the "name" from ip4_address ( name need to be exported to excel).
If "name" of ip4_address is equal to "hostname" of database "asset" then take
"client" and "os_team" ( export that to excel)
I have written the script but not getting the desired result.
def get_host_details(self):
data = {
"find": {
"hostname": self.controller
},
"projection":{
"tenant": 1,
"paths": 1,
"endpoint":1
}
}
host_details = self.post("https://database.app.com/api/data/devices/PathEp/find", data)
#print host_details
hosts = []
for record in host_details:
if "mig" not in record["endpoint"]:
hosts.append(record)
return hosts
def get_ipaddress(self, controller):
host_record = {"tenant": "UNKNOWN",
"paths": "UNKNOWN",
"endpoint": "UNKNOWN",
"ip": "UNKNOWN",
"mac": "UNKNOWN",
"encap": "UNKNOWN"}
data = {
"find": {
"hostname": controller,
"ip": {
"$ne": "0.0.0.0"
}
},
"projection": {
"ip": 1,
"mac":1,
"encap":1,
}
}
endpoints = self.post("https://database.app.com/api/data/devices/EEp/find", data)
IPAM = self.get_dns()
print endpoints
host_details = self.get_host_details()
host_details_record = []
for record in endpoints:
for host in host_details:
if record["mac"] == host["cep"]:
host_record = {"tenant": host["tenant"],
"paths": host["paths"],
"endpoint": host["endpoint"],
"ip": record["ip"],
"mac": record["mac"],
"encap": record["encap"]}
host_details_record.append(host_record)
self.get_excel(host_details_record)
def get_dns(self, endpoints):
ip_dns_record = []
for each_endpoint in endpoints:
data = {
"find":
{
"address": {
"$eq": each_endpoint["ip"]
},
},
"projection":
{
"name": 1
}
}
dns_record = {"client":"UNKNOWN",
"os_team":"UNKNOWN",
ipam_record = self.post("https://database.app.com/api/data/"
"internal/ip4_address/find", data)
if ipam_record:
dns_record["ip_address"] = each_endpoint["ip"]
dns_record["hostname"] = ipam_record[0]["name"]
dns_record = self.get_remedy_details(ipam_record[0]["name"],
dns_record)
ip_dns_record.append(dns_record)
else:
dns_record["ip_address"] = each_endpoint["ip"]
dns_record["hostname"] = "UNKNOWN"
ip_dns_record.append(dns_record)
self.get_excel(ip_dns_record)
def get_remedy_details(self, hostname, dns_record):
data = {
"find":
{
"hostname": hostname.upper(),
}
}
remedy_data = self.post("https://database.app.com/api/data/internal/asset/find", data)
print(remedy_data)
#remedy_data = remedy_data["data"]
if remedy_data:
dns_record["client"] = remedy_data[0].get("client","UNKNOWN")
dns_record["os_team"] = remedy_data[0].get("os_team", "UNKNOWN")
else:
dns_record["client"] = "UNKNOWN"
dns_record["os_team"] = "UNKNOWN"
return dns_record
def get_excel(self, ip_dns_record):
filename = self.controller + ".xls"
excel_file = xlwt.Workbook()
sheet = excel_file.add_sheet('HOSTLIST')
sheet.write(0, 0, "IP Address")
sheet.write(0, 1, "HostName")
sheet.write(0, 2, "Client")
sheet.write(0, 3, "OS Team")
for count in xrange(1, len(ip_dns_record)+1):
sheet.write(count, 0,ip_dns_record[count - 1]["ip_address"])
sheet.write(count, 1,ip_dns_record[count - 1]["hostname"])
sheet.write(count, 2,ip_dns_record[count - 1]["client"])
sheet.write(count, 3,ip_dns_record[count - 1]["os_team"])
excel_file.save(filename)
if __name__ == "__main__":
controller = sys.argv[1]
OBJ = ACIHostList(controller)
print "SCRIPT COMPLETED"
No idea where I am going wrong and what needs to be done .
Your question leaves too much out. You should include all errors that you get. You should also comment your code as well so we can understand what you are trying to achieve in each step.
This is not an answer but something to try:
Rather than trying to wrap your head around a module like excel, wright your data to a simple CSV file. A CSV file can be opened up in excel and it formats correctly but is a lot easier to create.
import csv
data = [["a", "b"], ["c", "d"]]
with open("file.csv", "w+") as csv_file:
create_csv = csv.writer(csv_file)
create_csv .writerows(data)
simply grab all your data into a 2D list and using the above code dump it into a file so you can easily read it.
check the output of the file and see if you are getting the data you expect.
If you are not getting the desired data into this CSV file then there is an issue with your database queries.