Execute Groovy Script to transform date in Nifi - groovy

I'm trying to transform my JSON dates using Nifi. They are imported in this format:
import groovy.json.JsonSlurper
import groovy.json.JsonBuilder
def ff = session.get()
if(!ff)return
ff = session.write(ff, {rawIn, rawOut->
// transform streams into reader and writer
rawIn.withReader("UTF-8"){reader->
rawOut.withWriter("UTF-8"){writer->
//parse reader into Map
def json = new JsonSlurper().parse(reader)
// set my variable and define what format it is in
json.date = new Date(json.date as Long).format('HH:mm yyyy-MM-dd')
// Reformat it
json.date = DateFormat.parse("yyyy-MM-dd HH:mm", json.date)
//write changed object to writer
new JsonBuilder(json).writeTo(writer)
}
}
} as StreamCallback)
session.transfer(ff, REL_SUCCESS)
The incoming flowfile has this body:
[{"date":"09:00 2019-05-29","data":460.0,"name":"login"},{"date":"10:00 2019-05-29","data":548.0,"name":"login"},{"date":"11:00 2019-05-14","data":0.0,"name":"login"},{"date":"00:00 2019-06-15","data":0.0,"name":"login"}]
I want this output:
[{"date":"2019-05-29 09:00","data":460.0,"name":"login"},{"date":"2019-05-29 10:00","data":548.0,"name":"login"},{"date":"2019-05-14 11:00","data":0.0,"name":"login"},{"date":"2019-06-15 00:00","data":0.0,"name":"login"}]
The error I get is this:
Can anyone please help me understand where I am going wrong?

The input is a list of the objects in question. The incoming date is a
String -- not a Long.
So the first error is to use json.date as it implies json*.date
(which gives a list of all date).
Next casting the date to Long, create a new Date and then format it is
the wrong way around.
So to change the format of all the date something like this is needed:
json.each{
it.date = Date.parse('HH:mm yyyy-MM-dd', it.date).format('yyyy-MM-dd HH:mm')
}

Related

how can I get value from json object in groovy

def RawRecordsDateRangeResponse = context.expand('${getRawRecordsForDateRange#Response}')
log.info RawRecordsDateRangeResponse
My response is:
{"2018-09-03":"https://dhap-dconnect-telemetry-data-dev2.s3.amazonaws.com/ULT/d83350d2-af56-11e8-b612-0242ac11001118/temperature/raw-2018-09-03.json"}
Here I want to get the value from the json response key as date.
Your response represents JSON document and it stores it in variable of type String. You can parse it using groovy.json.JsonSlurper class. Consider following example:
import groovy.json.JsonSlurper
def RawRecordsDateRangeResponse = context.expand('${getRawRecordsForDateRange#Response}')
def json = new JsonSlurper().parseText(RawRecordsDateRangeResponse)
def url = json.'2018-09-03'
println url
Output:
https://dhap-dconnect-telemetry-data-dev2.s3.amazonaws.com/ULT/d83350d2-af56-11e8-b612-0242ac11001118/temperature/raw-2018-09-03.json
If it's just the key of the JSON message you need rather than the value, you could use something like:
import groovy.json.JsonSlurper
def rawRecordsDateRangeResponse = '''{"2018-09-03":"https://dhap-dconnect-telemetry-data-dev2.s3.amazonaws.com/ULT/d83350d2-af56-11e8-b612-0242ac11001118/temperature/raw-2018-09-03.json"}'''
def json = new JsonSlurper().parseText(rawRecordsDateRangeResponse)
def date = json.collect({it.key})
print date
This produces [2018-09-03].

Scala dynamic String interpolation - Read from properties file

I tried to get the log message from message.properties and did string interpolation with that log message.In this time,The log message is not interpolated with original message.
I am not able to get string interpolated result and am getting the output as log message what i have specified in properties file
Here I dont want to hard code any log message in scala file,Instead of this,I want to get all message from properties file and redirected into application log after interpolating string value.
import com.typesafe.config.ConfigFactory
import grizzled.slf4j.Logging
object Test extends Logging {
def main(args: Array[String]){
val subjectArea="Member"
val messageProp = ConfigFactory.load("message.properties")
val log=messageProp.getString("log.subject.area")
debug(s"$log")
}
}
message.properties
log.subject.area=The Subject Area : $subjectArea
Console Output: The Subject Area : $subjectArea
i want this output : The Subject Area : Member
Thanks in advance!!!
Test.scala
message.propeties
This is not a string interpolation problem. You want a lightweight templating engine (e.g. http://jtwig.org/documentation/quick-start/application or something else). I feel that most of them would be an overkill if your problem is as simple as in the snippet you've provided.
If you want to do something more or less complex, then sure, go with template engines.
Otherwise, I'd just go with string substitution.
String interpolation only works with constants. To do what you want dynamically, you need to write some explicit processing yoruself (or use a template engine library). Something like this, perhaps?
val substPattern = """\$\{(.+?)\}""".r
import java.util.regex.Matcher.{ quoteReplacement => qq }
def processSubstitutions(
input: String,
vars: Map[String, String]
) = substPattern.replaceAllIn(
input, { m =>
val ref = m.group(1)
qq(vars.getOrElse(ref, ref)
}
)
val vars = Map("subjectArea" -> "Member")
val messageProp = ConfigFactory.load("message.properties")
val log=processSubstitutions(
messageProp.getString("log.subject.area"),
vars
)

Converting Date to String in groovy

I want to convert def date= new Date() to string so that i could make this expression def text=data+" "+"http://hjghjghj.ge(Service) /https:jsonparces getting data from taxservice:Successfully received response and then use text as an string i have tries toString() but it wasn't helpful any better ideas?
You don't even need toString(), just using it in a String concatenation transforms it, but you have to use String + Date, not Date + String, just like in Java. So either use
def text=""+date+" "+"http://hjghjghj.ge(Service) /https:jsonparces getting data from taxservice:Successfully received response
or
def text=(data as String)+" "+"http://hjghjghj.ge(Service) /https:jsonparces getting data from taxservice:Successfully received response
or
def text=data.toString()+" "+"http://hjghjghj.ge(Service) /https:jsonparces getting data from taxservice:Successfully received response`

How to convert soap xml response to delimited

I don't know hardly anything about XML. I have successfully gotten a SOAP response (using things like SOAPUi and Boomerang) from an asmx web service. It's a large file.
Now I need to get it to regular delimited columns. Is there a simple way to do this?
My file is attached here
Not sure if it is required one time transformation or do this job quite frequently.
So, adding the answer here with some more details.
Approach #1: Using on-line
As mentioned in the comments, you can use the on-line site to convert your xml data into csv.
Even it requires to do some pre-process with the message / response that you have i.e.,
save the data into file
remove headers or unwanted data etc or make it ready to be usable in the above mentioned online site.
The disadvantages in this approaches
requires some manual work
expose data on public, but at times may be possible to share
time taking
can not use it an automated fashion
difficult to repeat
Approach #2: Using Groovy Script
So, this approach addresses the disadvantages of #1 approach.
Here is the Groovy Script which reads previous soap request step's response, and gives the data into a csv file.
In your test case, add a new groovy script test step right after the soap request step which gives you the data and copy below script content into it. i.e., (Test Case -> Step 1: Soap Request where you are getting responseStep 2: Groovy Script (with below script))
Add a test case custom property, say OUTPUT_FILE_NAME and provide the file path for csv to be saved at. Even, if you do not provide this property, it will automatically saves the csv file chargedata.csv under System temp directory.
You may find the comments in-line
/**
* this script will read the previous step response
* extract the cdata at the given xpath
* read all the records and transfroms into csv file
**/
import com.eviware.soapui.support.XmlHolder
import groovy.xml.*
/**Define the output file name in test case custom property say OUTPUT_FILE_NAME and value as absolute file path
* otherwise, it write a file chargedata.csv in system temp directory
**/
def outputFileName = context.testCase.getPropertyValue('OUTPUT_FILE_NAME') ?: System.getProperty("java.io.tmpdir")+ '/chargedata.csv'
//csv field separator - change it if needed
def delimiter = ','
/**
* Below statement will fetch the previous request step response.
*/
def response = context.testCase.testStepList[context.currentStepIndex - 1].testRequest.response.responseContent
//Create the xml holder object to get the xpath value which cdata in this case
def responseHolder = new XmlHolder(response)
def xpath = '//*:Charges_FileResponse/*:Charges_FileResult'
//Get the cdata part from above xpath which is a string
def data = responseHolder.getNodeValue(xpath)
//This again parses the xml inside of cdata
def chargeRecords = new XmlParser().parseText(data)
//This is going hold all the data from ChargeRecords
def chargeRecordsDataStructure = []
//This is to hold all the headers
def headers = [] as Set
/**
* This is to create Charge data
**/
def buildChargeDataStructure = { charge ->
def chargeDataStructure = new Expando()
charge.children().each {
def elementName = it.name()
def elementText = it.value().join()
chargeDataStructure[elementName] = elementText
//Add to field name to the list if not already added
(elementName in headers) ?: headers << elementName
}
chargeDataStructure
}
/**
* this is to create a csv row in string format
**/
def createRow = { recordDataStructure ->
def row = new StringBuffer()
headers.each {
if (row) {
row += delimiter + recordDataStructure[it] ?: ''
} else {
row += recordDataStructure[it] ?: ''
}
}
row.toString()+'\n'
}
//Build the whole data structure of Charge Records
chargeRecords.Charge.each { charge ->
chargeRecordsDataStructure << buildChargeDataStructure( charge )
}
//Build the rows
def rows = new StringBuffer()
rows << headers.join(',') +'\n'
chargeRecordsDataStructure.each { rows << createRow (it)}
//Write the rows into file
new File(outputFileName).text = rows

Setting json key using bind variables in groovy

I've this code by which I am trying to set the value of a Key (a json node id). But its not setting the value. Log.info statement is showing right values.
Key= context.expand( '${#Project#key}' )
Value= context.expand( '${#Project#value}' )
Binding binding = new Binding()
binding.setVariable("v", "$Value")
binding.setVariable("k", "$Key")
log.info(binding.getVariable("v")) // gives me the value 1234
log.info(binding.getVariable("k")) // gives me the value request.id
def SetKey = new GroovyShell(binding).evaluate( "k=v")
Can someone please comment on whats wrong in this code. and how can I correct it.
Edit: Explanation of the issue
In SoapUI I've some json nodes saved in data source like this request.id and request.app.id and there expected values in Value column which I am fetching through Key and Value above. I am hoping to change the value of a json node to its respective value at run time. So for each iteration of data source, it should set the correct value of that particular json node. Befor the above code I've parsed my json request by json slurper and after the above code I am building the json again by Json builder and running the request. Parsing and running the request works fine, its just that I couldnt set the value of the json node.
If your keys are just dotted names, you could simply use some string manipulation, no need to involve the Groovy parser.
For example, if they all begin with request:
def steps = Key.split(/\./)
if (steps.size() < 2 || steps[0] != 'request') {
throw ...
}
def obj = request
if (steps.size() > 2) {
steps[1..-2].each {
obj = obj[it]
}
}
obj[steps[-1]] = Value

Resources