I need to do a json transformation in apache nifi. The json keys in the payload would be dynamically generated.
For example in the input given below, the 'customer' has attributes 'fname' and 'lname'. I need to change this 'fname' -> 'firstname' and 'lname' -> 'lastname' as provided in the 'mappingvalues'.
Since I am newbie to nifi. I dont know where to start. I have tried some json transformers like jolt. But couldn't achieve the expected result.
The jolt transform that i have used is given below :
[
{
"operation": "shift",
"spec": {
"customer": {
"*": {
"#": "&"
}
}
}
}
]
which produced an output
{
"fname" : "akhil",
"lname" : "kumar"
}
The input and expected output of what I need to achieve is given below :
{
"customer": {
"fname": "akhil",
"lname": "kumar",
.
.
.
},
"mappingvalues": {
"fname": "firstname",
"lname": "lastname",
.
.
.
}
}
##OUTPUT
{
"customer": {
"firstname": "akhil",
"lastname": "kumar",
.
.
.
}
}
*Is there any way to achieve the same in nifi with or without using jolt transform? Is it possible to do the same with groovy script? *
Please help me on the same.
the code in groovy with recursive mapping:
import groovy.json.JsonSlurper
def ff = session.get()
if(!ff)return
def json = ff.read().withReader("UTF-8"){r-> new JsonSlurper().parse(r) }
def mappings = json.remove('mappingvalues')
def mapper(o, mappings){
if(o instanceof Map){
//json object. let's iterate it and do mapping
o = o.collectEntries{k,v-> [ (mappings[k] ?: k), mapper(v,mappings) ] }
}else if(o instanceof List){
//map elements in array
o = o.collect{v-> mapper(v,mappings) }
}
return o
}
json = mapper(json,mappings)
ff.write("UTF-8"){w-> new JsonBuilder(json).writeTo(w) }
REL_SUCCESS << ff
Related
I would like to remove the array SEO from the json when the keys "Description" and "Title" in the has no value.
json:
[
{
"SEO": [
{
"Description": "",
"Title": ""
}
],
"accesoires": [
"1167296"
],
"shortCode": "S-576",
"spareParts": [
"800236"
]
}]
I tried the below code but i'm not able to remove the array.
def Message processData(Message message) {
def body = message.getBody(String);
def json = new JsonSlurper().parseText(body)
json.each{
it.SEO.each{
if(!(it.findResults{k, v -> v?.size() > 0 && v[0]?.length() > 0 ? v[0] : null })){
json.remove("SEO")
} } }
def out= JsonOutput.toJson(json)
message.setBody(out)
return message}
To remove the array "SEO" from the JSON when the keys "Description" and "Title" have no value, you can use the following Groovy code:
def jsonString = '[{"SEO": [{"Description": "", "Title": ""}], "accesoires": ["1167296"], "shortCode": "S-576", "spareParts": ["800236"]}]'
def json = new JsonSlurper().parseText(jsonString)
for (item in json) {
if (!item.SEO[0].Description && !item.SEO[0].Title) {
item.remove('SEO')
}
}
println(JsonOutput.toJson(json))
This will first parse the JSON string into a list of maps using JsonSlurper. Then it iterates through each map in the list and checks if the "Description" and "Title" keys in the "SEO" array are empty. If they are, it removes the "SEO" array from the map using the remove() method. Finally, it prints the modified JSON using the JsonOutput.toJson() method.
I am in no way an expert with groovy so please don't hold that against me.
I have JSON that looks like this:
{
"metrics": [
{
"name": "metric_a",
"help": "This tracks your A stuff.",
"type": "GAUGE",
"labels": [
"pool"
],
"unit": "",
"aggregates": [],
"meta": [
{
"category": "CAT A",
"deployment": "environment-a"
}
],
"additional_notes": "Some stuff (potentially)"
},
...
]
...
}
I'm using it as a source for automated documentation of all the metrics. So, I'm iterating through it in various ways to get the information I need. So far so good, I'm most of the way there. The problem is this all needs to be organized per the deployment environment. Meaning, multiple metrics will share the same value for deployment.
My thought was I could create a map with deployment as the key and the metric name for any metric that has a matching deployment as the value. Once I have that map, it should be easy for me to organize things the way they should be. I can't figure out how to do that. The result is all the metric names are added which is expected since I'm not doing anything to filter them out. I was thinking that groupBy would make sense here but I can't figure out how to use it effectively and frankly I'm not sure it will solve my problem by itself. Here is my code so far:
parentChild = [:]
children = []
metrics.each { metric ->
def metricName = metric.name
def depName = metric.meta.findResult{ it.deployment }
children.add(metricName)
parentChild.put(depName, children)
}
What is the best way to create a new map where the values for each key are based off a specific condition?
EDIT: The desired result would be each key in the resulting map would be a unique deployment value from all the metrics (as a string). Each value would be name of each metric that contains that deployment (as an array).
[environment-a:
[metric_a,metric_b,metric_c,...],
environment-b:
[metric_d,metric_e,metric_f,...]
...]
I would use a combo of withDefault() to pre-fill each map-entry value with a fresh TreeSet-instance (sorted no-duplicates set) and standard inject().
I reduced your sample data to the bare minimum and added some new nodes:
import groovy.json.*
String input = '''\
{
"metrics": [
{
"name": "metric_a",
"meta": [
{
"deployment": "environment-a"
}
]
},
{
"name": "metric_b",
"meta": [
{
"deployment": "environment-a"
}
]
},
{
"name": "metric_c",
"meta": [
{
"deployment": "environment-a"
},
{
"deployment": "environment-b"
}
]
},
{
"name": "metric_d",
"meta": [
{
"deployment": "environment-b"
}
]
}
]
}'''
def json = new JsonSlurper().parseText input
def groupedByDeployment = json.metrics.inject( [:].withDefault{ new TreeSet() } ){ res, metric ->
metric.meta.each{ res[ it.deployment ] << metric.name }
res
}
assert groupedByDeployment.toString() == '[environment-a:[metric_a, metric_b, metric_c], environment-b:[metric_c, metric_d]]'
If your metrics.meta array is supposed to have a single value, you can simplify the code by replacing the line:
metric.meta.each{ res[ it.deployment ] << metric.name }
with
res[ metric.meta.first().deployment ] << metric.name
I have following JSON response anonymous body and I need to parse nested arrays dynamically to retrieve a key's value based on a condition by using find or findAll in the groovy's closures
[
{
"children": [
{
"attr": {
"reportId": "1",
"reportShortName": "ABC",
"description": "test,
}
},
{
"attr": {
"reportId": "2",
"reportShortName": "XYZ",
"description": "test",
}
}
}
]
I've tried the following ways and had no luck to retrieve the reportId key's value from the JSON response
package com.src.test.api;
import static io.restassured.RestAssured.given;
import io.restassured.path.json.JsonPath;
import io.restassured.response.Response;
public class GetReportId {
public void getReportId(String reportName) throws Exception {
String searchReports = "http://localhost:8080/reports";
Response resp=given().request().when().get(searchReports).then().extract().response();
JsonPath jsonPath = new JsonPath(resp.asString());
String reportId1 =jsonPath.get("$.find{it.children.contains(restAssuredJsonRootObject.$.children.find{it.attr.reportShortName == 'ABC'})}.attr.reportId");
String reportId2 = jsonPath.get("$.find{it.children.attr.reportShortName.contains(restAssuredJsonRootObject.$.children.find{it.attr.reportShortName.equals('XYZ')}.attr.reportShortName)}.attr.reportId");
System.out.println("ReportId: " + reportId1);
}
}
There could be multiple JSON objects in the parent anonymous array and need to make use of find or findAll within the groovy closures to get the reportId
Need to get the reportId, but seems that something is wrong. Any help would be appreciated.
Assuming you want all the reportIds
List<String> reportIds = jsonPath.get("children.flatten().attr.reportId");
will give you what you want, even it the parent anonymous array has multiple entries.
I tested with the following JSON
[
{
"children": [
{
"attr": {
"reportId": "1",
"reportShortName": "ABC",
"description": "test"
}
},
{
"attr": {
"reportId": "2",
"reportShortName": "XYZ",
"description": "test"
}
}
]
},
{
"children": [
{
"attr": {
"reportId": "3",
"reportShortName": "DEF",
"description": "test"
}
},
{
"attr": {
"reportId": "4",
"reportShortName": "IJK",
"description": "test"
}
}
]
}
]
and it gives me ["1", "2", "3", "4"] i.e. reportIds from all the children
If you know the index of the reportId you're looking for then you can use it like so:
String reportId = jsonPath.get("children.flatten().attr.reportId[0]");
If you're looking for the reportId of a particular report you can do that too:
String reportId = jsonPath.get("children.flatten().attr.find{it.reportShortName == 'ABC'}.reportId")
will give you "1".
Note: The type of the variable you assign the results to are important for type inference and casting. For example, you CANNOT do:
String [] reportIds = jsonPath.get("children.flatten().attr.reportId");
or
int reportId = jsonPath.get("children.flatten().attr.reportId[0]");
Both those things will throw a ClassCastException.
1.I am having a list which contains multiple Maps that looks like below Map.
Each map contains many keys I want to get value of "name"
{
"question":{
"com.forms.tree":{
"requiredByDefault":true,
"questionDetails":{
"com.forms.Details":{
"preferredFormComponent":"TEXT"
}
},
"locale":{
"language":"en"
},
"formField":{
"name":"CUSTOM_347",
"tag":"input",
"url":"Demo"
}
}
},
"Field":"true"
},{
"question":{
"com.forms.tree":{
"questionDetails":{
"com.forms.Details":{
"preferredFormComponent":"TEXT"
}
},
"locale":{
"language":"en"
},
"formField":{
"name":"CUSTOM_348",
"url":"Demo"
}
}
},
"Field":"true"}
I want to get the value of "name" which falls in every Map but don't want to iterate like question?."com.forms.tree"?.formField?.name.
Is there any other approach in groovy?
So given the json:
def jsonTxt = '''{
"question":{
"com.forms.tree":{
"requiredByDefault":true,
"questionDetails":{
"com.forms.Details":{
"preferredFormComponent":"TEXT"
}
},
"locale":{
"name": "test",
"language":"en"
},
"formField":{
"name":"CUSTOM_347",
"tag":"input",
"url":"Demo"
}
}
},
"Field":"true"
}'''
We can parse it with:
import groovy.json.*
def json = new JsonSlurper().parseText(jsonTxt)
You want to find the "formField" entry in that object, so lets write a recursive finder that will walk through out map of maps looking for the first entry with the given key:
static findFirstByKey(Map map, key) {
map.get(key) ?: map.findResult { k, v -> if(v in Map) findFirstByKey(v, key) }
}
And you can then check it works:
assert findFirstByKey(json, 'formField')?.name == "CUSTOM_347"
I have required to remove object from array that satisfies the condition, I am able to update the object of array on the basis of condition, which is as follow:
PUT twitter/twit/1
{"list":
[
{
"tweet_id": "1",
"a": "b"
},
{
"tweet_id": "123",
"a": "f"
}
]
}
POST /twitter/twit/1/_update
{"script":"foreach (item :ctx._source.list) {
if item['tweet_id'] == tweet_id) {
item['new_field'] = 'ghi';
}
}",
"params": {tweet_id": 123"}
}
this is working
for remove i am doing this
POST /twitter/twit/1/_update
{ "script": "foreach (item : ctx._source.list) {
if item['tweet_id'] == tweet_id) {
ctx._source.list.remove(item);
}
}",
"params": { tweet_id": "123" }
}
but this is not working and giving this error,
ElasticsearchIllegalArgumentException[failed to execute script];
nested: ConcurrentModificationException; Error:
ElasticsearchIllegalArgumentException[failed to execute script];
nested: ConcurrentModificationException
I am able to remove whole array or whole field using
"script": "ctx._source.remove('list')"
I am also able to remove object from array by specifying all the keys of an object using
"script":"ctx._source.list.remove(tag)",
"params" : {
"tag" : {"tweet_id": "123","a": "f"}
my node module elastic search version is 2.4.2 elastic search server is 1.3.2
You get that because you are trying to modify a list while iterating through it, meaning you want to change a list of object and, at the same time, listing those objects.
You instead need to do this:
POST /twitter/twit/1/_update
{
"script": "item_to_remove = nil; foreach (item : ctx._source.list) { if (item['tweet_id'] == tweet_id) { item_to_remove=item; } } if (item_to_remove != nil) ctx._source.list.remove(item_to_remove);",
"params": {"tweet_id": "123"}
}
If you have more than one item that matches the criteria, use a list instead:
POST /twitter/twit/1/_update
{
"script": "items_to_remove = []; foreach (item : ctx._source.list) { if (item['tweet_id'] == tweet_id) { items_to_remove.add(item); } } foreach (item : items_to_remove) {ctx._source.list.remove(item);}",
"params": {"tweet_id": "123"}
}
For people that need this working in elasticsearch 2.0 and up, the nil and foreach don't get recognized by groovy.
So here's a updated version, including a option to replace a item with the same id by a new object.
and also passing it the upsert will make sure the item gets added even if the document doesn't exist yet
{
"script": "item_to_remove = null; ctx._source.delivery.each { elem -> if (elem.id == item_to_add.id) { item_to_remove=elem; } }; if (item_to_remove != null) ctx._source.delivery.remove(item_to_remove); if (item_to_add.size() > 1) ctx._source.delivery += item_to_add;",
"params": {"item_to_add": {"id": "5", "title": "New item"}},
"upsert": [{"id": "5", "title": "New item"}]
}