Transpose JSON in Groovy - groovy

I have a JSON:
{
"range": "'All traffic",
"majorDimension": "ROWS",
"values": [
[
"Date",
"Sessions",
"Leeds",
"Сontact_query",
"wo_Сontact_query"
],
[
"20200107",
"3461",
"15",
"0",
"15"
],
[
"20200115",
"7824",
"43",
"0",
"43"
]
]
}
Elements in values array with index[0] are names for JSON properties. I want to map these names with values starting with index[0+]. Number of values can be different for each index. Mapping should be dynamical, without hard indexing.
I want to get following JSON:
[
{
"Date": "20200107",
"Sessions": "3461",
"Leeds": "15",
"Contact_query": "0",
"wo_Contact_query": "15"
},
{
"Date": "20200115",
"Sessions": "7824",
"Leeds": "43",
"Contact_query": "0",
"wo_Contact_query": "43"
}
]
I successfully did it with kinda weird JOLT config:
[
{
"operation": "shift",
"spec": {
"values": {
"0": null,
"*": {
"*": "[&1].#(2,[0].[&])"
}
}
}
},
{
"operation": "shift",
"spec": {
"0": null,
"*": "[]"
}
}
]
But i want to replace it with Groovy script.
I reached following solution:
def slurped = new JsonSlurper().parseText(text)
def val = slurped.values[0]
def json = slurped.values.collect{
[val, it].transpose().collectEntries()
}
But it also returns values for [0] element which one i should skip:
{
"Date":"Date",
"Sessions":"Sessions",
"Leeds":"Leeds",
"Contact_query":"Contact_query",
"wo_Contact_query":"wo_Contact_query"
}
How can I improve solution?

You can use pop to remove the first value of a collection:
def values = new groovy.json.JsonSlurper().parseText(text).values;
def val = values.pop()
def json = values.collect { [val, it].transpose().collectEntries() }

Another option using head and tail:
def json = new JsonSlurper().parseText(data)
def head = json.values.head() // first element, the "head"
def tail = json.values.tail() // the rest, excluding the head, i.e. "tail"
def result = tail.collect { [head, it].transpose().collectEntries() }

Related

Python search for all nested values in list of dictionaries that matches a nested key

I have the following list of dictionaries:
data = [
{
"connections": None
},
{
"connections": {
"endpoints": [
{
"id": "ee0d38d3-89fb-4e5b-b9bd-83ee554949ab",
"vlan": 200,
},
{
"id": "192ee48f-ad20-48ad-acae-9c8d33e4687b",
"vlan": 100,
},
]
}
},
{
"connections": {
"endpoints": [
{
"id": "4e6b8844-9a91-4098-aa92-ef97ce89cbed",
"vlan": 200,
},
{
"id": "577fcb45-57ab-4903-be60-5b8ac84b8a09",
"vlan": 100,
},
]
}
},
]
I want to search for 'id' "ee0d38d3-89fb-4e5b-b9bd-83ee554949ab" in data and extract all the vlans for any matching 'id'. So in this example, my resulting variable should only contain a list of matches as such:
[100]
I say a list because it is possible that 'id' "ee0d38d3-89fb-4e5b-b9bd-83ee554949ab" will also have another entry in data but with a different vlan. So you could end up with in my result:
[200, 300]
I tried the following but I get a "TypeError: 'NoneType' object is not subscriptable":
vlans = [d["connections"]["endpoints"]["vlan"] for d in data if d["connections"]["endpoints"]["id"] == id]
You do something like below:
res = [endpoint['vlan'] for item in data if item["connections"] for endpoint in item["connections"]["endpoints"] if endpoint['id']== "ee0d38d3-89fb-4e5b-b9bd-83ee554949ab"]
and if you need unique vlan values you can use
list(set(res))

why mongodb converting python dictionary to object

#dataclass_json
#dataclass
class Metadata:
key = None
data = None
def some_method():
raw_data =
{
"key": "ZEl",
"data": [
{
"type": "rich_text",
"elements": [
{
"type": "text",
"text": "Some text"
"elements" : [
{ "type":"text_section",
"text":"inner text"
}
]
}
]
}
]
}
metadata = Metadata.from_dict(raw_data)
print(metdata.to_dict()) # print data as expected
mongo_connector.collection.insert_one(metadata.to_dict())
Here i am using pymongo to interact with the MongoDB. This dictionary is getting written into mongodb correctly but it is converting inner elements key to Object. I tried to use asdict(), to_dict() of dataclasses but nothing worked. While i print the data via print it data looks consistant but writing into mongo, it is holding some another shape.
Output of Mongo
{
_id: ObjectId("3b8a25c03f011d64a77f015d"),
key: 'ZEl',
data: [
{
type: 'rich_text',
elements: [
{
type: 'text',
text: 'Some text',
elements: [ [Object] ]
}
]
}
]
}
I could not understand this behaviour of converting inner elements to [ [Object] ]`

NodeJs How to convert parent-child json array to json-object with key-id and value array of all childes

I have a parent-child array that lookes like this:
const myTestArray = [
{
"id": "1",
"parent": "root"
},
{
"id": "2",
"parent": "1"
},
{
"id": "3",
"parent": "1"
},
{
"id": "4",
"parent": "2"
},
{
"id": "5",
"parent": "4"
}
];
From this I have been able to create an nested parent-child json-tree, but now I need another data-structure that looks like this:
const childesArray = {
"1": ["1", "2", "3", "4", "5"],
"2": ["2", "4", "5"],
"3": ["3"],
"4": ["4", "5"],
"5": ["5"]
};
For each element in myTestArray I need an array that contains all childes (deep) for that element.
I guess I also here can use reduce on this array, but I'm stucked right now how I can achive this. Anyone that can point me in the right direction? Or have any solution on this?
Br. Rune
There are several ways to achieve this, but I tried to keep it simple. Take a look at this example.
const nesting = {};
myTestArray.forEach((item) => {
const { parent, id } = item;
if (!nesting[id]) {
nesting[id] = [id];
}
if (!nesting[parent] && parent !== "root") {
nesting[parent] = [parent];
}
if (parent !== "root") {
nesting[parent].push(id);
}
});
console.log(nesting);
//
// {
// "1":["1","2","3"],
// "2":["2","4"],
// "3":["3"],
// "4":["4","5"],
// "5":["5"]
// }
//
You could take the JSON object and iterate through it with a ForEach loop. You can then append an element to the index of its parent, and you would end up with a 2-dimensional array such as the one you specified.
var newArr = {};
myTestArray.forEach(e => {
newArr[e.id] = [e.id];
if(e.parent != "root"){
newArr[e.parent].push(e.id);
}
});
would produce
newArr = { '1': [ '1', '2', '3' ],
'2': [ '2', '4' ],
'3': [ '3' ],
'4': [ '4', '5' ],
'5': [ '5' ] }
Have been using my day on this :)
Think I have a solution now.
Suggestions on improvements are welcome.
function getNestedChildrenArray(inputArray: Array<Record<string,unknown>>, inputId: string) {
let myA: Array<string> = inputArray.reduce((previousValue: Array<string>, currentValue: Record<string,unknown>) => {
if(typeof currentValue.id === 'string' && inputId === currentValue.parent) previousValue.push(currentValue.id);
return previousValue;
}, []);
if(myA.length > 0) {
myA = myA.concat(myA.reduce((previousValue: Array<string>, currentValue: string) => {
previousValue = previousValue.concat(getNestedChildrenArray(inputArray, currentValue));
return previousValue;
},[]));
}
return myA;
}
function getNestedChildrenArrayJsonObject(inputArray: Array<Record<string,unknown>>) {
return inputArray.reduce((previousValue: Record<string,unknown>, currentValue: Record<string,unknown>, index, array) => {
if(typeof currentValue.id === 'string') {
previousValue[currentValue.id] = [currentValue.id].concat(getNestedChildrenArray(array, currentValue.id));
}
return previousValue;
}, {});
}
console.log(getNestedChildrenArrayJsonObject(myTestArray));

Find and convert values in nested dict and update db collection

I have converted many xml files to json using xmltodict and inserted those in arangodb.
Now I well loop over the collection and change some values in the database. LIke day, mount and year from string to int. The documents can be very nested and the values that I well change can be in different places.
This is what I have of the code.
# Get the API wrapper for "FORM16" collection.
FORM16 = db.collection('FORM16')
def recursive_items(dictionary):
for key, value in dictionary.items():
if type(value) is dict:
yield from recursive_items(value)
else:
yield (key, value)
search_key = 'LOW_VALUE'
for item in FORM16:
for key, value in recursive_items(item):
if search_key in list(key):
item[search_key] = int(item[search_key])
else:
pass
FORM16.update(item)
{'_id': 'FORM16/2098312',
'_key': '2098312',
'_rev': '_blGxlRi---',
'_old_rev': '_blGvpVO---'}
The code runs but It won’t update the database and the document that I receive that has changed is only the last document in the collection.
What do I have to change in the code to convert values in keys like day, mount and year to int?
EDIT:
This is one of the nested json's doc. that I well update
{
"DOFFIN_ESENDERS": {
"DOFFIN_APPENDIX": {
"AUTHORITY_ORGANISATION_NR": "986 105 174",
"DOFFIN_FORM_TYPE": {
"NATIONAL": {
"EXPRESSION_OF_INTEREST_URL": "https://kgv.doffin.no/ctm/Supplier/Notice/260549",
"EXTERNAL_DOCUMENT_URL": "https://kgv.doffin.no/ctm/Supplier/Documents/Folder/124452",
"LOCATION": {
"NATIONWIDE": null
},
"PUBLISH_TO_TED": null
}
}
},
"FORM_SECTION": {
"PRIOR_INFORMATION_DEFENCE": {
"CATEGORY": "ORIGINAL",
"FD_PRIOR_INFORMATION_DEFENCE": {
"AUTHORITY_PRIOR_INFORMATION_DEFENCE": {
"NAME_ADDRESSES_CONTACT_PRIOR_INFORMATION": {
"CA_CE_CONCESSIONAIRE_PROFILE": {
"ADDRESS": "Postboks 800, Postmottak",
"ATTENTION": "Ole Jan Skoglund",
"CONTACT_POINT": "Forsvarets logistikkorganisasjon",
"COUNTRY": {
"VALUE": "NO"
},
"E_MAILS": {
"E_MAIL": "olskoglund#mil.no"
},
"FAX": "+47 67863799",
"ORGANISATION": {
"NATIONALID": "986105174",
"OFFICIALNAME": "Forsvarets logistikkorganisasjon"
},
"PHONE": "+47 67863787",
"POSTAL_CODE": "LILLEHAMMER",
"TOWN": "N-2617"
},
"FURTHER_INFORMATION": {
"IDEM": null
},
"INTERNET_ADDRESSES_PRIOR_INFORMATION": {
"URL_BUYER": "https://kgv.doffin.no/ctm/Supplier/CompanyInformation/Index/1127",
"URL_GENERAL": "http://www.forsvaret.no"
}
},
"TYPE_AND_ACTIVITIES_OR_CONTRACTING_ENTITY_AND_PURCHASING_ON_BEHALF": {
"PURCHASING_ON_BEHALF": {
"PURCHASING_ON_BEHALF_NO": null
},
"TYPE_AND_ACTIVITIES": {
"TYPE_OF_ACTIVITY": {
"VALUE": "DEFENCE"
},
"TYPE_OF_CONTRACTING_AUTHORITY": {
"VALUE": "MINISTRY"
}
}
}
},
"CTYPE": "SUPPLIES",
"LEFTI_PRIOR_INFORMATION": null,
"OBJECT_WORKS_SUPPLIES_SERVICES_PRIOR_INFORMATION": {
"ADDITIONAL_INFORMATION": {
"P": "Konkurransen vil bli utført som en forhandlet prosedyre etter en planlagt kunngjøring ultimo 2015 i henhold til “Forskrift 4. oktober 2013 nr. 1185 om forsvars og sikkerhetsanskaffelser“ basert på Eu direktiv 2009/81/EC fra Europa Parlamentet."
},
"CPV": {
"CPV_ADDITIONAL": [
{
"CPV_CODE": {
"CODE": "18900000"
}
},
{
"CPV_CODE": {
"CODE": "18930000"
}
},
{
"CPV_CODE": {
"CODE": "18937000"
}
},
{
"CPV_CODE": {
"CODE": "33000000"
}
},
{
"CPV_CODE": {
"CODE": "33120000"
}
},
{
"CPV_CODE": {
"CODE": "33124000"
}
},
{
"CPV_CODE": {
"CODE": "33140000"
}
},
{
"CPV_CODE": {
"CODE": "33141000"
}
},
{
"CPV_CODE": {
"CODE": "33141100"
}
},
{
"CPV_CODE": {
"CODE": "33141200"
}
},
{
"CPV_CODE": {
"CODE": "33141300"
}
},
{
"CPV_CODE": {
"CODE": "50400000"
}
}
],
"CPV_MAIN": {
"CPV_CODE": {
"CODE": "33100000"
}
}
},
"FRAMEWORK_AGREEMENT": {
"VALUE": "YES"
},
"QUANTITY_SCOPE_WORKS_DEFENCE": {
"COSTS_RANGE_AND_CURRENCY": {
"CURRENCY": "NOK",
"RANGE_VALUE_COST": {
"HIGH_VALUE": "200000000",
"LOW_VALUE": "150000000"
}
},
"F16_DIVISION_INTO_LOTS": {
"DIV_INTO_LOT_NO": null
},
"TOTAL_QUANTITY_OR_SCOPE": {
"P": "Forsvarets logistikkorganisasjon planlegger å skifte ut Forsvarets prehospitale sanitetssystem. Vi ser derfor etter en systemleverandør som kan levere test moduler, store initielle systemleveranser og ta ansvar for effektiv etterforsyning til Forsvaret på rammeavtaler med inntil syv års varighet."
}
},
"SCHEDULED_DATE_PERIOD": {
"PERIOD_WORK_DATE_STARTING": {
"MONTHS": "84"
}
},
"TITLE_CONTRACT": {
"P": "RFI P9346 -Nytt Prehospital Sanitetssystem til Forsvaret"
},
"TYPE_CONTRACT_PLACE_DELIVERY_DEFENCE": {
"SITE_OR_LOCATION": {
"LABEL": "N-2055 Nordkisa",
"NUTS": {
"CODE": "NO"
}
},
"TYPE_CONTRACT_PI_DEFENCE": {
"TYPE_CONTRACT": {
"VALUE": "SUPPLIES"
}
}
}
},
"OTH_INFO_PRIOR_INFORMATION": {
"ADDITIONAL_INFORMATION": {
"P": "Vi ønsker svar både fra Systemleverandører og Underleverandører på denne RFI."
},
"INFORMATION_REGULATORY_FRAMEWORK": {
"TAX_LEGISLATION": {
"TAX_LEGISLATION_VALUE": "www.lovdata.no"
}
},
"NOTICE_DISPATCH_DATE": {
"DAY": "28",
"MONTH": '11',
"YEAR": "2014"
},
"RELATES_TO_EU_PROJECT_NO": null
}
},
"FORM": "16",
"LG": "NB",
"VERSION": "R2.0.8.S02"
}
},
"VERSION": "V2.0.0",
"http://www.w3.org/2001/XMLSchema-instance:noNamespaceSchemaLocation": "DOFFIN_ESENDERS.xd",
"xmlns": {
"xsi": "http://www.w3.org/2001/XMLSchema-instance"
}
}
}
It looks like your code is correct, assuming the JSON blob at the bottom is a representation of item. Just make sure the data you're passing to .update() includes a valid _key and/or _id attribute.
However, it looks like your update statement is not indented properly and/or out of order. I would put the update inline, when you make the change:
FORM16 = db.collection('FORM16')
for item in FORM16:
for key, value in recursive_items(item):
if search_key in list(key):
item[search_key] = int(item[search_key])
FORM16.update(item)
else:
pass
or in the top-level for loop:
FORM16 = db.collection('FORM16')
for item in FORM16:
for key, value in recursive_items(item):
if search_key in list(key):
item[search_key] = int(item[search_key])
else:
pass
FORM16.update(item)
I did find a functin for convert sting to int and float in JSON files.
def _decode(o):
# Note the "unicode" part is only for python2
if isinstance(o, str):
try:
return int(o)
except ValueError:
try:
return float(o)
except ValueError:
return o
elif isinstance(o, dict):
return {k: _decode(v) for k, v in o.items()}
elif isinstance(o, list):
return [_decode(v) for v in o]
else:
return o
path = 'C:/doffin/test/'
for filename in os.listdir(path):
if not filename.endswith('.json'):
continue
#26  
#fullname = os.path.join(path, filename)
fullname = os.path.join(path, filename)
with open(fullname, 'rb') as f:
jsonstr = f.read()
json_sting = json.loads(jsonstr, object_hook=_decode)
json_str2 = json.dumps(json_sting)
with open(fullname[:-4] + ".json", 'w') as f:
f.write(json_str2)
and afther that I use arango import form the shell. It works better then the API.

Groovy: Convert Json to Text

Would like to convert the below json record into a text using Groovy
import groovy.json.*
def js = """{
"title": {
"titleid": "222",
"titlename": "ABCD",
"titledesc": null
},
"customer": {
"customerDetail": {
"customerid": 878378743,
"customerstatus": "ACTIVE",
"customersystems": {
"customersystem1": "SYS01",
"customersystem2": null
},
"sysid": null
},
"store": {
"storeid": "LOS002",
"storename": "LAStore",
"areacode": "JDHJ8K988"
},
"persons": {
"person1": {
"personid": "123",
"personname": "IIISKDJKJSD"
},
"person2": {
"personid": "456",
"personname": "IUDFIDIKJK"
}
},
"order": {
"orderdetail": {
"orderid": "4291026",
"ordername": "ORD93999"
}
},
"product": {
"orderdate": "20190101",
"currency": "USD",
"amount": 1000.23
}
}
}
"""
def data = new JsonSlurper().parseText(js)
Expected output should as below with proper header names:
customerId,customerstatus,customersystem1,sysid,storeid,storename,person1.personid,person1.personname,orderid,orderdate,currency,amount,titlename
878378743,ACTIVE,SYS01,null,LOS002,LAStore,123,IIISKDJKJSD,4291026,20190101,USD,1000.23
This is just a single json record so how would I convert all my json records using Groovy?
The following code:
import groovy.json.*
def js = """
[
{
"title": {
"titleid": "222",
"titlename": "ABCD",
"titledesc": null
},
"customer": {
"customerDetail": {
"customerid": 878378743,
"customerstatus": "ACTIVE",
"customersystems": {
"customersystem1": "SYS01",
"customersystem2": null
},
"sysid": null
},
"store": {
"storeid": "LOS002",
"storename": "LAStore",
"areacode": "JDHJ8K988"
},
"persons": {
"person1": {
"personid": "123",
"personname": "IIISKDJKJSD"
},
"person2": {
"personid": "456",
"personname": "IUDFIDIKJK"
}
},
"order": {
"orderdetail": {
"orderid": "4291026",
"ordername": "ORD93999"
}
},
"product": {
"orderdate": "20190101",
"currency": "USD",
"amount": 1000.23
}
}
}
]
"""
/*
customerId,customerstatus,customersystem1,sysid,storeid,storename,person1.personid,person1.personname,orderid,orderdate,currency,amount,titlename
878378743,ACTIVE,SYS01,null,LOS002,LAStore,123,IIISKDJKJSD,4291026,20190101,USD,1000.23
*/
def data = new JsonSlurper().parseText(js)
def mappings = [
customerId: { n -> n.customer.customerDetail.customerid },
customerstatus: { n -> n.customer.customerDetail.customerstatus },
customersystem1: { n -> n.customer.customerDetail.customersystems.customersystem1 },
sysid: { n -> n.customer.customerDetail.sysid },
storeid: { n -> n.customer.store.storeid },
storename: { n -> n.customer.store.storename },
'person1.personid': { n -> n.customer.persons.person1.personid },
'person1.personname': { n -> n.customer.persons.person1.personname },
orderid: { n -> n.customer.order.orderdetail.orderid },
orderdate: { n -> n.customer.product.orderdate },
currency: { n -> n.customer.product.currency },
amount: { n -> n.customer.product.amount },
titlename: { n -> n.title.titlename }
]
def headers = mappings.keySet().join(',') //edited thanks to comment
println headers
data.each { item ->
def row = mappings.collect { k, v -> v(item) }.join(',')
println row
}
does what you ask for. Note that I made the json be a list of items instead of the single item since it seemed from your text that that is what you were after.
Running the above code produces:
~> groovy solution.groovy
customerId,customerstatus,customersystem1,sysid,storeid,storename,person1.personid,person1.personname,orderid,orderdate,currency,amount,titlename
878378743,ACTIVE,SYS01,null,LOS002,LAStore,123,IIISKDJKJSD,4291026,20190101,USD,1000.23,ABCD
~>
note that if this is going into some critical system and is not just a one-off ad-hoc piece of code, you should probably do things like check the return value of v(item) and log some error of otherwise handle when there is no value for a certain path in the json etc.
Should also be noted that the above code relies on the fact that a map literal in groovy (i.e. def mappings = [:]) creates an instance of java's LinkedHashMap which has predictable iteration order for things like keySet() and collect { }.
<< edit >>
For a single item json blob, you would change the code as follows:
def js = """
{
...
}
"""
def item = new JsonSlurper().parseText(js)
def mappings = ...
def headers = mappings.keySet().join(',') //edited thanks to comment
println headers
def row = mappings.collect { k, v -> v(item) }.join(',')
println row
where ... denotes that the block is unchanged from the example above.

Resources