Failed to add element to new ArrayList in Groovy - groovy

I am trying to simulate a multimap, each value of langVarMap is a list. When I add a new String to the list, I get the following error:
No signature of method: java.lang.Boolean.add() is applicable for argument types: (java.lang.String) values: [mm]
Here is the code snippet:
def langs = engine.languages as Set
def langVarMap = [:]
engine.models.each { model ->
def lang = (model.#language.text()) // String
def variant = (model.#variant.text()) // String
langs.add(lang)
if (langVarMap.get(lang)) {
def a = langVarMap.get(lang) //ArrayList
langVarMap.put(lang, a.add(variant))
}
else {
langVarMap.put(lang, [variant])
}
Thanks in advance.

Problem is with this line:
langVarMap.put(lang, a.add(variant))
ArrayList.add(E e) returns boolean not the list. Adding result of add action adds a boolean value of TRUE to the map after which cannot call add method on it. Need to rewrite as following:
if (langVarMap.get(lang)) {
def a = langVarMap.get(lang) //ArrayList
// a is already in langVarMap so don't need to put into ma again
a.add(variant)
} else {
langVarMap.put(lang, [variant])
}
And can further refine with this to remove the redundant lookup.
def a = langVarMap.get(lang) //ArrayList
if (a) {
a.add(variant)
} else {
langVarMap.put(lang, [variant])
}

Related

Nested JSON with duplicate keys

I will have to process 10 billion Nested JSON records per day using NiFi (version 1.9). As part of the job, am trying to convert the nested JSON to csv using Groovy script. I referred the below Stack Overflow questions related to the same topic and came up with the below code.
Groovy collect from map and submap
how to convert json into key value pair completely using groovy
But am not sure how to retrieve the value of duplicate keys. Sample json is defined in the variable "json" in the below code. key "Flag1" will be coming in multiple sections (i.e., "OF" & "SF"). I want to get the output as csv.
Below is the output if I execute the below groovy code 2019-10-08 22:33:29.244000,v12,-,36178,0,0/0,10.65.5.56,sf,sf (flag1 key value is replaced by that key column's last occurrence value)
I am not an expert in Groovy. Also please suggest if there is any other better approach, so that I will give a try.
import groovy.json.*
def json = '{"transaction":{"TS":"2019-10-08 22:33:29.244000","CIPG":{"CIP":"10.65.5.56","CP":"0"},"OF":{"Flag1":"of","Flag2":"-"},"SF":{"Flag1":"sf","Flag2":"-"}}'
def jsonReplace = json.replace('{"transaction":{','{"transaction":[{').replace('}}}','}}]}')
def jsonRecord = new JsonSlurper().parseText(jsonReplace)
def columns = ["TS","V","PID","RS","SR","CnID","CIP","Flag1","Flag1"]
def flatten
flatten = { row ->
def flattened = [:]
row.each { k, v ->
if (v instanceof Map) {
flattened << flatten(v)
} else if (v instanceof Collection && v.every {it instanceof Map}) {
v.each { flattened << flatten(it) }
} else {
flattened[k] = v
}
}
flattened
}
print "output: " + jsonRecord.transaction.collect {row -> columns.collect {colName -> flatten(row)[colName]}.join(',')}.join('\n')
Edit: Based on the reply from #cfrick and #stck, I have tried the option and have follow up question below.
#cfrick and #stck- Thanks for your response.
Original source JSON record will have more than 100 columns and I am using "InvokeScriptedProcessor" in NiFi to trigger the Groovy script.
Below is the original Groovy script am using in "InvokeScriptedProcessor" in which I have used Streams(inputstream, outputstream). Is this what you are referring.
Am I doing anything wrong?
import groovy.json.JsonSlurper
class customJSONtoCSV implements Processor {
def REL_SUCCESS = new Relationship.Builder().name("success").description("FlowFiles that were successfully processed").build();
def log
static def flatten(row, prefix="") {
def flattened = new HashMap<String, String>()
row.each { String k, Object v ->
def key = prefix ? prefix + "_" + k : k;
if (v instanceof Map) {
flattened.putAll(flatten(v, k))
} else {
flattened.put(key, v.toString())
}
}
return flattened
}
static def toCSVRow(HashMap row) {
def columns = ["CIPG_CIP","CIPG_CP","CIPG_SLP","CIPG_SLEP","CIPG_CVID","SIPG_SIP","SIPG_SP","SIPG_InP","SIPG_SVID","TG_T","TG_R","TG_C","TG_SDL","DL","I_R","UAP","EDBL","Ca","A","RQM","RSM","FIT","CSR","OF_Flag1","OF_Flag2","OF_Flag3","OF_Flag4","OF_Flag5","OF_Flag6","OF_Flag7","OF_Flag8","OF_Flag9","OF_Flag10","OF_Flag11","OF_Flag12","OF_Flag13","OF_Flag14","OF_Flag15","OF_Flag16","OF_Flag17","OF_Flag18","OF_Flag19","OF_Flag20","OF_Flag21","OF_Flag22","OF_Flag23","SF_Flag1","SF_Flag2","SF_Flag3","SF_Flag4","SF_Flag5","SF_Flag6","SF_Flag7","SF_Flag8","SF_Flag9","SF_Flag10","SF_Flag11","SF_Flag12","SF_Flag13","SF_Flag14","SF_Flag15","SF_Flag16","SF_Flag17","SF_Flag18","SF_Flag19","SF_Flag20","SF_Flag21","SF_Flag22","SF_Flag23","SF_Flag24","GF_Flag1","GF_Flag2","GF_Flag3","GF_Flag4","GF_Flag5","GF_Flag6","GF_Flag7","GF_Flag8","GF_Flag9","GF_Flag10","GF_Flag11","GF_Flag12","GF_Flag13","GF_Flag14","GF_Flag15","GF_Flag16","GF_Flag17","GF_Flag18","GF_Flag19","GF_Flag20","GF_Flag21","GF_Flag22","GF_Flag23","GF_Flag24","GF_Flag25","GF_Flag26","GF_Flag27","GF_Flag28","GF_Flag29","GF_Flag30","GF_Flag31","GF_Flag32","GF_Flag33","GF_Flag34","GF_Flag35","VSL_VSID","VSL_TC","VSL_MTC","VSL_NRTC","VSL_ET","VSL_HRES","VSL_VRES","VSL_FS","VSL_FR","VSL_VSD","VSL_ACB","VSL_ASB","VSL_VPR","VSL_VSST","HRU_HM","HRU_HD","HRU_HP","HRU_HQ","URLF_CID","URLF_CGID","URLF_CR","URLF_RA","URLF_USM","URLF_USP","URLF_MUS","TCPSt_WS","TCPSt_SE","TCPSt_WSFNS","TCPSt_WSF","TCPSt_EM","TCPSt_RSTE","TCPSt_MSS","NS_OPID","NS_ODID","NS_EPID","NS_TrID","NS_VSN","NS_LSUT","NS_STTS","NS_TCPPR","CQA_NL","CQA_CL","CQA_CLC","CQA_SQ","CQA_SQC","TS","V","PID","RS","SR","CnID","A_S","OS","CPr","CVB","CS","HS","SUNR","SUNS","ML","MT","TCPSL","CT","MS","MSH","SID","SuID","UA","DID","UAG","CID","HR","CRG","CP1","CP2","AIDF","UCB","CLID","CLCL","OPTS","PUAG","SSLIL"]
return columns.collect { column ->
return row.containsKey(column) ? row.get(column) : ""
}.join(',')
}
#Override
void initialize(ProcessorInitializationContext context) {
log = context.getLogger()
}
#Override
Set<Relationship> getRelationships() {
return [REL_SUCCESS] as Set
}
#Override
void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException {
try {
def session = sessionFactory.createSession()
def flowFile = session.get()
if (!flowFile) return
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def bufferedReader = new BufferedReader(new InputStreamReader(inputStream, 'UTF-8'))
def jsonSlurper = new JsonSlurper()
def line
def header = "CIPG_CIP,CIPG_CP,CIPG_SLP,CIPG_SLEP,CIPG_CVID,SIPG_SIP,SIPG_SP,SIPG_InP,SIPG_SVID,TG_T,TG_R,TG_C,TG_SDL,DL,I_R,UAP,EDBL,Ca,A,RQM,RSM,FIT,CSR,OF_Flag1,OF_Flag2,OF_Flag3,OF_Flag4,OF_Flag5,OF_Flag6,OF_Flag7,OF_Flag8,OF_Flag9,OF_Flag10,OF_Flag11,OF_Flag12,OF_Flag13,OF_Flag14,OF_Flag15,OF_Flag16,OF_Flag17,OF_Flag18,OF_Flag19,OF_Flag20,OF_Flag21,OF_Flag22,OF_Flag23,SF_Flag1,SF_Flag2,SF_Flag3,SF_Flag4,SF_Flag5,SF_Flag6,SF_Flag7,SF_Flag8,SF_Flag9,SF_Flag10,SF_Flag11,SF_Flag12,SF_Flag13,SF_Flag14,SF_Flag15,SF_Flag16,SF_Flag17,SF_Flag18,SF_Flag19,SF_Flag20,SF_Flag21,SF_Flag22,SF_Flag23,SF_Flag24,GF_Flag1,GF_Flag2,GF_Flag3,GF_Flag4,GF_Flag5,GF_Flag6,GF_Flag7,GF_Flag8,GF_Flag9,GF_Flag10,GF_Flag11,GF_Flag12,GF_Flag13,GF_Flag14,GF_Flag15,GF_Flag16,GF_Flag17,GF_Flag18,GF_Flag19,GF_Flag20,GF_Flag21,GF_Flag22,GF_Flag23,GF_Flag24,GF_Flag25,GF_Flag26,GF_Flag27,GF_Flag28,GF_Flag29,GF_Flag30,GF_Flag31,GF_Flag32,GF_Flag33,GF_Flag34,GF_Flag35,VSL_VSID,VSL_TC,VSL_MTC,VSL_NRTC,VSL_ET,VSL_HRES,VSL_VRES,VSL_FS,VSL_FR,VSL_VSD,VSL_ACB,VSL_ASB,VSL_VPR,VSL_VSST,HRU_HM,HRU_HD,HRU_HP,HRU_HQ,URLF_CID,URLF_CGID,URLF_CR,URLF_RA,URLF_USM,URLF_USP,URLF_MUS,TCPSt_WS,TCPSt_SE,TCPSt_WSFNS,TCPSt_WSF,TCPSt_EM,TCPSt_RSTE,TCPSt_MSS,NS_OPID,NS_ODID,NS_EPID,NS_TrID,NS_VSN,NS_LSUT,NS_STTS,NS_TCPPR,CQA_NL,CQA_CL,CQA_CLC,CQA_SQ,CQA_SQC,TS,V,PID,RS,SR,CnID,A_S,OS,CPr,CVB,CS,HS,SUNR,SUNS,ML,MT,TCPSL,CT,MS,MSH,SID,SuID,UA,DID,UAG,CID,HR,CRG,CP1,CP2,AIDF,UCB,CLID,CLCL,OPTS,PUAG,SSLIL"
outputStream.write("${header}\n".getBytes('UTF-8'))
while (line = bufferedReader.readLine()) {
def jsonReplace = line.replace('{"transaction":{','{"transaction":[{').replace('}}}','}}]}')
def jsonRecord = new JsonSlurper().parseText(jsonReplace)
def a = jsonRecord.transaction.collect { row ->
return flatten(row)
}.collect { row ->
return toCSVRow(row)
}
outputStream.write("${a}\n".getBytes('UTF-8'))
}
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
session.commit()
}
catch (e) {
throw new ProcessException(e)
}
}
#Override
Collection<ValidationResult> validate(ValidationContext context) { return null }
#Override
PropertyDescriptor getPropertyDescriptor(String name) { return null }
#Override
void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) { }
#Override
List<PropertyDescriptor> getPropertyDescriptors() {
return [] as List
}
#Override
String getIdentifier() { return null }
}
processor = new customJSONtoCSV()
If I should not use "collect" then what else I need to use to create the rows.
In the output flow file, the record output is coming inside []. I tried the below but it is not working. Not sure whether am doing the right thing. I want csv output without []
return toCSVRow(row).toString()
If you know what you want to extract exactly (and given you want to
generate a CSV from it) IMHO you are way better off to just shape the
data in the way you later want to consume it. E.g.
def data = new groovy.json.JsonSlurper().parseText('[{"TS":"2019-10-08 22:33:29.244000","CIPG":{"CIP":"10.65.5.56","CP":"0"},"OF":{"Flag1":"of","Flag2":"-"},"SF":{"Flag1":"sf","Flag2":"-"}}]')
extractors = [
{ it.TS },
{ it.V },
{ it.PID },
{ it.RS },
{ it.SR },
{ it.CIPG.CIP },
{ it.CIPG.CP },
{ it.OF.Flag1 },
{ it.SF.Flag1 },]
def extract(row) {
extractors.collect{ it(row) }
}
println(data.collect{extract it})
// ⇒ [[2019-10-08 22:33:29.244000, null, null, null, null, 10.65.5.56, 0, of, sf]]
As stated in the other answer, due to the sheer amount of data you are trying to
convert::
Make sure to use a library to generate the CSV file from that, or else
you will hit problems with the content, you try to write (e.g. line
breaks or the data containing the separator char).
Don't use collect (it is eager) to create the rows.
The idea is to modify "flatten" method - it should differentiate between same nested keys by providing parent key as a prefix.
I've simplified code a bit:
import groovy.json.*
def json = '{"transaction":{"TS":"2019-10-08 22:33:29.244000","CIPG":{"CIP":"10.65.5.56","CP":"0"},"OF":{"Flag1":"of","Flag2":"-"},"SF":{"Flag1":"sf","Flag2":"-"}}'
def jsonReplace = json.replace('{"transaction":{','{"transaction":[{').replace('}}','}}]')
def jsonRecord = new JsonSlurper().parseText(jsonReplace)
static def flatten(row, prefix="") {
def flattened = new HashMap<String, String>()
row.each { String k, Object v ->
def key = prefix ? prefix + "." + k : k;
if (v instanceof Map) {
flattened.putAll(flatten(v, k))
} else {
flattened.put(key, v.toString())
}
}
return flattened
}
static def toCSVRow(HashMap row) {
def columns = ["TS","V","PID","RS","SR","CnID","CIP","OF.Flag1","SF.Flag1"] // Last 2 keys have changed!
return columns.collect { column ->
return row.containsKey(column) ? row.get(column) : ""
}.join(', ')
}
def a = jsonRecord.transaction.collect { row ->
return flatten(row)
}.collect { row ->
return toCSVRow(row)
}.join('\n')
println a
Output would be:
2019-10-08 22:33:29.244000, , , , , , , of, sf

Clone of list still correct the original list

In groovy the original value get overwritten when I change values in a clone list. Does anyone know if I am doing it wrong or it is a bug older groovy?
I am doing something like this:
List<Foo> myFooList = fooList.newFoos.findAll { it.type == "Types}
List<Foo> newFoo = fooList.oldFoos.findAll { it.type == "Types}.clone()
newFoo.each {
it.value = "neeeew value"
}
Foo fooOne = newFoo.each { foooo ->
fooTwo = fooList.oldFoos.find { it.id == foooo.id}
if(fooTwo.value != foooo.value) {
//Here it should go... but it turns out that fooTwo.value == foooo.value
}
}
the clone method called on list produces a new list but with the same objects in it.
you want to build new list with new objects. here is an example:
#groovy.transform.ToString
class Foo{
String type
String value
}
def fooList = [
new Foo(type:"Types", value:'old value1'),
new Foo(type:"Not", value:'old value2'),
new Foo(type:"Types", value:'old value3'),
new Foo(type:"Not", value:'old value4'),
]
def newFooList = fooList.
findAll{it.type=='Types'}.
collect{ new Foo(type:it.type, value:"new value") } //build new array with transformed elements
//check the original list
fooList.each{assert it.value!='new value'}
//check new list
newFooList.each{assert it.value=='new value'}
assert newFooList.size()==2
println fooList
println newFooList
I solved the issue by adding clone of the element as well, any way it became to much of cowboy fix:
List<Foo> myFooList = fooList.newFoos.findAll { it.type == "Types}
List<Foo> newFoo = fooList.oldFoos.findAll { it.type == "Types}.collect {it.clone()}
newFoo.each {
it.value = "neeeew value"
}
Foo fooOne = newFoo.each { foooo ->
fooTwo = fooList.oldFoos.find { it.id == foooo.id}
if(fooTwo.value != foooo.value) {
//Here it should go... but it turns out that fooTwo.value == foooo.value
}
}

Creation of custom comparator for map in groovy

I have class in groovy
class WhsDBFile {
String name
String path
String svnUrl
String lastRevision
String lastMessage
String lastAuthor
}
and map object
def installFiles = [:]
that filled in loop by
WhsDBFile dbFile = new WhsDBFile()
installFiles[svnDiffStatus.getPath()] = dbFile
now i try to sort this with custom Comparator
Comparator<WhsDBFile> whsDBFileComparator = new Comparator<WhsDBFile>() {
#Override
int compare(WhsDBFile o1, WhsDBFile o2) {
if (FilenameUtils.getBaseName(o1.name) > FilenameUtils.getBaseName(o2.name)) {
return 1
} else if (FilenameUtils.getBaseName(o1.name) > FilenameUtils.getBaseName(o2.name)) {
return -1
}
return 0
}
}
installFiles.sort(whsDBFileComparator);
but get this error java.lang.String cannot be cast to WhsDBFile
Any idea how to fix this? I need to use custom comparator, cause it will be much more complex in the future.
p.s. full source of sample gradle task (description of WhsDBFile class is above):
project.task('sample') << {
def installFiles = [:]
WhsDBFile dbFile = new WhsDBFile()
installFiles['sample_path'] = dbFile
Comparator<WhsDBFile> whsDBFileComparator = new Comparator<WhsDBFile>() {
#Override
int compare(WhsDBFile o1, WhsDBFile o2) {
if (o1.name > o2.name) {
return 1
} else if (o1.name > o2.name) {
return -1
}
return 0
}
}
installFiles.sort(whsDBFileComparator);
}
You can try to sort the entrySet() :
def sortedEntries = installFiles.entrySet().sort { entry1, entry2 ->
entry1.value <=> entry2.value
}
you will have a collection of Map.Entry with this invocation. In order to have a map, you can then collectEntries() the result :
def sortedMap = installFiles.entrySet().sort { entry1, entry2 ->
...
}.collectEntries()
sort can also take a closure as parameter which coerces to a Comparator's compare() method as below. Usage of toUpper() method just mimics the implementation of FilenameUtils.getBaseName().
installFiles.sort { a, b ->
toUpper(a.value.name) <=> toUpper(b.value.name)
}
// Replicating implementation of FilenameUtils.getBaseName()
// This can be customized according to requirement
String toUpper(String a) {
a.toUpperCase()
}

Call method multiple times with each member of a collection as parameter

Let's say that I have a collection of parameters
def params = ['a','b','c']
Is there a short way to run a method that accepts a single parameter once for every element of a collection to replace this:
params.each {
foo(it)
}
with something more declarative (like a "reverse" spread operator)?
You can use collect:
def params = ['a','b','c']
def foo(param) {
'foo-' + param
}
assert ['foo-a', 'foo-b', 'foo-c'] == params.collect { foo(it) }
Or just a closure
def foo = { a -> a + 2 }
def modified = list.collect foo
You can use method pointer:
def l = [1,2,3]
l.each(new A().&lol)
class A {
def lol(l) {
println l
}
}
Or add a method that will do the task you need:
def l = [1,2,3]
List.metaClass.all = { c ->
delegate.collect(c)
}
l.all(new A().&lol)
class A {
def lol(l) {
println l
return l+2
}
}

groovy.lang.MissingMethodException: No signature of method

I am getting the following error -
groovy.lang.MissingMethodException: No signature of method: Script64$_run_closure5_closure7_closure8_closure9_closure10_closure11.doCall() is applicable for argument types: (java.lang.String) values: Possible solutions: doCall(java.lang.Object, java.lang.Object), isCase(java.lang.Object), isCase(java.lang.Object) error at line:
Code - EDIT
import groovy.xml.*
List tempList = []
List listgenerated = []
def count = 0
for (a in 0..totalCount-1)
{
//nameList and valueList lists will have all the contents added as below commented pseudo code
/*for (b in 0..50)
{
nameList.add(b,number) // number is some calculated value
valueList.add(b,number)
e.g. nameList=[name1, name2, name3,name4, name5]
valueList =[val1, val2, val3, , val5]
listgenerated should be = [[name1:val1, name2:val2], [name3:val3, name4: , name5:val5]]
} */
tempList = []
for (j in count..nameList.size())
{
count = j
def nameKey = nameList[j]
def value
if (nameKey != null)
{
value = valueList[j]
tempList << [(nameKey) : value]
}
}
count = count
listgenerated.putAt(a,tempList)
number = number +1
}
def process = { binding, element, name ->
if( element[ name ] instanceof Collection ) {
element[ name ].each { n ->
binding."$name"( n )
}
}
else if( element[ name ] ) {
binding."$name"( element[ name ] )
}
}
class Form {
List fields
}
def list = [[ name:'a', val:'1' ], [ name:'b', val :'2', name2:4, xyz:'abc', pqr:'']] //Edited list
f = new Form( fields: list ) //Works fine
f = new Form( fields: listgenerated ) //Gives the above error
String xml = XmlUtil.serialize( new StreamingMarkupBuilder().with { builder ->
builder.bind { binding ->
data {
f.fields.each { fields ->
item {
fields.each { name, value ->
process( binding, fields, name )
}
}
}
}
}
} )
If while creating the "listgenerated" single quotes are added around values it takes it as character and while printing both lists seem different.
I am unable to figure it out what exactly is going wrong. Any help is appreciated. Thanks.
Ref - Groovy: dynamically create XML for collection of objects with collections of properties
I believe, where you do:
//some loop to add multiple values to the list
listgenerated << name+":"+value
You need to do:
//some loop to add multiple values to the list
listgenerated << [ (name): value ]
And add a map to the list rather than a String. It's hard to say though as your code example doesn't run without alteration, and I don't know if it's the alterations that are solving the problem

Resources