json to json conversion by ExecuteScript processor - groovy

I have following json data and i want to convert data in expected result by ExecuteScript nifi processor
{
"time": "2017-01-01T01:14:55+00:00",
"any": {
"nested": "data"
}
}
expected Result
{
"time": 1483233295,
"any": {
"nested": "data"
}
}
I am using following groovy code but getting some error please help to find the solution
var flowFile = session.get();
if (flowFile !== null) {
var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
flowFile = session.write(flowFile, new
StreamCallback(function(inputStream, outputStream) {
var inputJSON = IOUtils.toString(inputStream,StandardCharsets.UTF_8);
var contentObj = JSON.parse(inputJSON);
contentObj.time = flowFile.getAttribute("timestamp");
outputStream.write(JSON.stringify(contentObj).getBytes(StandardCharsets.UTF_8));
}));
session.transfer(flowFile, REL_SUCCESS);
}
getting error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
/home/jdoodle.groovy: 8: unable to resolve class StreamCallback
# line 8, column 36.
flowFile = session.write(flowFile, new
^
1 error

use ExecuteGroovyScript processor (it's optimized for groovy lang) with this kind of code:
import groovy.json.JsonSlurper
import groovy.json.JsonBuilder
def flowFile = session.get()
if (!flowFile) return
flowFile.write{rawIn, rawOut->
def json = rawIn.withReader("UTF-8"){ r-> new JsonSlurper().parse(r) }
json.time = Date.parse("yyyy-MM-dd'T'HH:mm:ssX", json.time).getTime()/1000
rawOut.withWriter("UTF-8"){ w-> new JsonBuilder(json).writeTo(w) }
}
REL_SUCCESS << flowFile
this code converts format of the field time with Date format to unix epoch time inside json content of flowfile.

Related

Getting Empty Array Using Groovy Script in Nifi

I have an requirement where I need to parse the data into the required format
Input:
{
"Message" : "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\n"
}
Here the content and delimiters are not fixed. The fixed part is only /nRecord keyword on which I am writing the Script. But I am not getting desired Output using Groovy.
desired Output:
[
{
"Record 1": "nRequired data is missing"
},
{
"Record 2": "processing failed"
}
]
I have written Groovy Script for the same but I am getting empty array.
import org.apache.commons.io.IOUtils
import groovy.json.*
import java.util.ArrayList
import java.nio.charset.*
import java.nio.charset.StandardCharsets
import groovy.json.JsonSlurper
import groovy.json.JsonBuilder
def flowFile = session.get()
if(!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
splitted = text.split('\nRecord')
int j = splitted.size()
final1 = []
for (int i=0;i<j-1;i++)
{
k = "Record " + splitted[i+1]
valid = k.replaceAll("\\n|\"|\\n|}","")
final1.add("{\"" + valid.replaceFirst(":",'":"')+ "\"}" )
}
def json = JsonOutput.toJson(final1)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
flowFile = session.putAttribute(flowFile, "error", e.getMessage())
session.transfer(flowFile, REL_FAILURE)
}
Can you please help me with the same.
Thank you.
I would use a regex with a simple trick:
import groovy.json.*
def json = new JsonSlurper().parseText '{ "Message" : "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\nRecord 3:\nprocessing failed badly\n" }'
String msg = json.Message.replaceAll( /\n+(Record \d+:)/, '=$1' ) // THE trick!
List res = ( msg =~ /(?m)(Record \d+):([^=]+)*/ ).collect{ _, rec, text -> [ (rec):text.trim() ] }
assert JsonOutput.toJson( res ) == '[{"Record 1":"Required data is missing."},{"Record 2":"processing failed"},{"Record 3":"processing failed badly"}]'

NiFi processor to search and replace multiple words in a txt file

I have txt file which I am reading using FetchSFTP in NiFi. Also, I have key and values in json format, as shown below received after REST call and JoltTransformJSON:
[{
"Key": "k2s2e2",
"Value": "Ottawa"
}, {
"Key": "60601",
"Value": "Chicago"
}, {
"Key": "",
"Value": "London"
}]
How can I replace all the occurrences of matching key from above to its value in txt file.
Example: abc.txt
000 apple stocks at k2s2e2 888
9000 samsung stocks at 60601 9990377
88 nokia devivces at 78889 790888071 hgj 7
Output:
000 apple stocks at Ottawa 888
9000 samsung stocks at Chicago 9990377
88 nokia devivces at 78889 790888071 hgj 7
My attempt using ExecuteGroovyScript:
import static groovy.json.JsonOutput.toJson
import java.nio.charset.StandardCharsets
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
class KeyValue{
String key
String value
}
private findAndReplace(KeyValueList) {
def response
KeyValueList.each {
def srcExp = it.key
def replaceText = it.value
def inputFilepath = "C:\\Project\\abc.txt"
def outputFilepath = "C:\\Project\\abc_output.txt"
new File(outputFilepath).withWriter { w ->
new File(inputFilepath).eachLine { line ->
w << line.replaceAll(srcExp , replaceText ) << '\n'
}
response = w
}
new File(inputFilepath).text= new File(outputFilepath).text
}
return response;
}
def flowFile = session.get()
if(!flowFile) return
def KeyValueList = []
//try {
def is = flowFile.read().withReader("UTF-8"){ new JsonSlurper().parse(it) }
is.each {
if(it.Key != "") {
KeyValue keyValue = new KeyValue(key:it.Key,value:it.Value)
KeyValueList.add(keyValue)
}
}
def retval = findAndReplace(KeyValueList)
flowFile = session.write(flowFile, {outputStream ->
outputStream.write(retval.toString().getBytes(StandardCharsets.UTF_8))
} as OutputStreamCallback)
session.transfer(flowFile, REL_SUCCESS)
//}catch(Exception e) {
// log.info(e.getMessage())
// REL_FAILURE << flowFile
//}
it's not a response to your question. just a try to fix your code.
if i understand correctly you are trying to
read json from flowfile
read text from some file path
write text with replacement to flowfile
code for ExecuteGroovyScript processor
import groovy.json.JsonSlurper
def ff = session.get()
if(!ff) return
ff.write{rawIn, rawOut->
def keyValueList = rawIn.withReader("UTF-8"){ new JsonSlurper().parse(it) }
new File('c:/Project/abc.txt').withReader("UTF-8"){reader->
rawOut.withWriter("UTF-8"){writer->
reader.eachLine{line->
keyValueList.each{ if(it.Key) line = line.replaceAll(it.Key, it.Value) }
writer << line << '\n'
}
}
}
}
REL_SUCCESS << ff
don't have time to test it...

groovy - replace values in json when loading from jmx

I have a piece a code as below. This is loading data from JMX to monitoring tool. Unfortunately, there are some 'NaN' values, which are not read properly by monitoring tool, and this interrupts transfer. My intention is to replace 'on fly' values 'NaN' to 'null'.
Has anybody any idea how to do it?
BR
import javax.security.auth.Subject;
import javax.security.auth.login.LoginContext;
import java.security.PrivilegedAction;
def hostName = hostProps.get("system.hostname")
def jmxPort = "10002"
def query = "Hadoop:service=hiveserver2,name=hiveserver2"
def metricsURL = "http://${hostName}:${jmxPort}/jmx?qry=${query}"
def ad = true
public class FetchMetrics implements PrivilegedAction {
def url;
public FetchMetrics(String url) {
this.url = url;
}
public Object run() {
URL urlObject = new URL(url);
def con = (HttpURLConnection) urlObject.openConnection();
return con.getInputStream();
}
}
lc = new LoginContext("Client");
lc.login();
Subject subject = lc.getSubject();
PrivilegedAction action = new FetchMetrics(metricsURL);
def metrics = Subject.doAsPrivileged(subject, action, null);
def jsonSlurper = new JsonSlurper()
def deviceMetrics = jsonSlurper.parse(metrics)
// Hit out endpoint and get our metrics
//deviceMetrics = jsonSlurper.parse(new URL(metricsURL))
deviceMetrics.'beans'[0].each {
println it
}
return 0
simplest way to use LAX json slurper, however it will parse NaN as a string "NaN"...
import groovy.json.JsonSlurper
import groovy.json.JsonParserType
import groovy.json.JsonBuilder
def jsonSlurper = new JsonSlurper().setType( JsonParserType.LAX )
def json = jsonSlurper.parseText('{ "a":111, "b": NaN}')
println new JsonBuilder(json).toPrettyString()
prints
{
"a": 111,
"b": "NaN"
}

how reading nutch generated content data on the segment folder using java

I am trying to read the content data inside the segment folder. I think the content data file is written in a custom format
I experimented with nutch's Content class, but it does not recognize the format.
import java.io.IOException;
import org.apache.commons.cli.Options;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.nutch.protocol.Content;
import org.apache.nutch.util.NutchConfiguration;
public class ContentReader {
public static void main(String[] args) throws IOException {
// Setup the parser
Configuration conf = NutchConfiguration.create();
Options opts = new Options();
GenericOptionsParser parser = new GenericOptionsParser(conf, opts, args);
String[] remainingArgs = parser.getRemainingArgs();
FileSystem fs = FileSystem.get(conf);
String segment = remainingArgs[0];
Path file = new Path(segment, Content.DIR_NAME + "/part-00000/data");
SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
Text key = new Text();
Content content = new Content();
// Loop through sequence files
while (reader.next(key, content)) {
try {
System.out.write(content.getContent(), 0,
content.getContent().length);
} catch (Exception e) {
}
}
}
}
org.apache.nutch.segment.SegmentReader
has a map reduce implementation that reads content data in the segment directory.
spark/scala code to read data from the segments content folder.
How I read from the content folder in my project.
I have created a case class page which holds data read from the content folder
case class Page(var url: String, var title: String = null
,var contentType: String = null, var rawHtml: String = null,var language: String = null
,var metadata: Map[String,String])
Code to read from content folder
import org.apache.commons.lang3.StringUtils
import org.apache.hadoop.io.{Text, Writable}
import org.apache.nutch.crawl.{CrawlDatum, Inlinks}
import org.apache.nutch.parse.ParseText
import org.apache.nutch.protocol.Content
val contentDF = spark.sparkContext.sequenceFile(path.contentLocation, classOf[Text], classOf[Writable])
.map { case (x, y) => (x.toString, extract(y.asInstanceOf[Content])) }
/** converts Content object to Page **/
def extract(content: Content): Page = {
try {
val parsed = Page(content.getUrl)
var charset: String = getCharsetFromContentType(content.getContentType)
if (StringUtils.isBlank(charset)) {
charset = "UTF-8"
}
parsed.rawHtml = Try(new String(content.getContent, charset)).getOrElse(new String(content.getContent, "UTF-8"))
parsed.contentType = Try(content.getMetadata.get("Content-Type")).getOrElse("text/html")
// parsed.isHomePage = Boolean.valueOf(content.getMetadata.get("isHomePage"))
parsed.metadata = content.getMetadata.names().map(name => (name,content.getMetadata.get(name))).toMap
Try {
if (StringUtils.isNotBlank(content.getMetadata.get("Content-Language")))
parsed.language = content.getMetadata.get("Content-Language")
else if (StringUtils.isNotBlank(content.getMetadata.get("language")))
parsed.language = content.getMetadata.get("language")
else parsed.language = content.getMetadata.get("lang")
}
parsed
} catch {
case e: Exception =>
LOG.error("ERROR while extracting data from Content ", e)
null
}
}
/**Get Html ContentType **/
def getCharsetFromContentType(contentType: String): String = {
var result: String = "UTF-8"
Try {
if (StringUtils.isNotBlank(contentType)) {
val m = charsetPattern.matcher(contentType)
result = if (m.find) m.group(1).trim.toUpperCase else "UTF-8"
}
}
result
}

SOAPUI: Validate response against xsd schema file

How can I validate a SOAP response against an XSD file that defines the response schema. the web service I'm calling has an XMLDocument as input and output, so can't use WSDL for response schema validation.
I case you still need this (valid for SOAP UI version 2.5.1):
File, Preferences, Editor Setting, Validate Response.
Use script assertion:
def project = messageExchange.modelItem.testStep.testCase.testSuite.project
def wsdlcontext = project.getInterfaceAt(0).getDefinitionContext()
def validator = new com.eviware.soapui.impl.wsdl.support.wsdl.WsdlValidator(wsdlcontext);
def errors = validator.assertRequest(messageExchange, false)
assert errors.length < 1
You can use the groovy script for validation the response against the xsd file.
Here is the way to validate
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.SchemaFactory;
import javax.xml.XMLConstants;
//Read your xsd file and get the conten into a variable like below.
def xsdContent = "Some Schema Standard";
//Take the response into another variable that you have to validate.
def actualXMLResponse = "Actual XML Response ";
//create a SchemaFactory object
def factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
//Create a given schema object with help of factory
def schema = factory.newSchema(new StreamSource(new StringReader(xsdContent ));
//Create a validator
def validator = schema.newValidator();
//now validate the actual response against the given schema
try {
validator.validate(new StreamSource(new StringReader(actualXMLResponse )));
} catch(Exception e) {
log.info (e);
assert false;
}
I hope this will help you :-)
This not worked me caused try not working
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.SchemaFactory;
import javax.xml.XMLConstants;
//Read your xsd file and get the conten into a variable like below.
// trim - XSD SCHEME no spaces
def xsdscheme = context.expand('${Properties-XSD_Scheme_Black_and_White#XSDSchemeWhite}')
def xsdscheme2 = xsdscheme.replace(' ', '')
xsdscheme2 = xsdscheme2.replaceAll("[\n\r]", "");
log.info "RES2 TRIMED " + xsdscheme2
def xsdContent = xsdscheme2;
//Take the response into another variable that you have to validate.
Res = context.expand('${#TestCase#WhiteListDecoded}');
def Res2 = Res.replace(' ', '')
Res2 = Res2.replaceAll("[\n\r]", "");
log.info "RES2 TRIMED " + Res2
def actualXMLResponse = Res2
//create a SchemaFactory object
def factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
//Create a given schema object with help of factory
def schema = factory.newSchema(new StreamSource(new StringReader(xsdContent ));
//Create a validator
def validator = schema.newValidator();
//now validate the actual response against the given schema
try {
validator.validate(new StreamSource(new StringReader(actualXMLResponse )));
} catch(Exception e) {
log.info (e);
assert false;
}

Resources