I have a problem with the below scenario:
-- I have a GPathResult "body" to which I want to append some more xml (nodes and children)
-- Some parts are common so I am trying to have them kept in an outer closure "commonNode" I can insert wherever I need
// some more code here to get body
def commonNode = {
return {
node2() {
child("childValue")
}
}
}
body.appendNode(
{
node1("value1")
commonNode()
node3("value3")
}
)
What I want to get after I would call XmlUtil.serialize(body) is this:
...
<body>
<node1>value</node1>
<node2>
<child>childValue</child>
</node2>
<node3>value3</node3>
<body>
...
however structure is missing from the result entirely, so I guess there is something wrong with the way I call the outer closure "commonNode()".
Hope someone has an answer. Let me know if you need further details.
This works:
import groovy.xml.*
def xml = '<body/>'
def body = new XmlSlurper().parseText( xml )
def commonNode = {
node2 {
child "childValue"
}
}
body.appendNode {
node1 "value1"
commonNode.delegate = delegate
commonNode()
node3 "value3"
}
println XmlUtil.serialize( body )
Related
I have a simple Groovy method that uses Groovy's MarkupBuilder to print HTML, very simplified version below:
void writeHtmlFile(<args>) {
def writer = new FileWriter(fileName.toFile())
def html = new MarkupBuilder(writer)
html.mkp.yieldUnescaped '<!DOCTYPE html>'
html.mkp.xmlDeclaration(version: "1.0", encoding: "utf-8")
html.html {
head { ... }
body(id: 'main') {
h1 "Report Title"
}
}
writer.flush()
writer.close()
}
This works well. Say I wanted to call a method after the h1 that does some calculations and adds more to the MarkupBuilder. How do I get the elements defined in the called method added to the MarkupBuilder? Here's something I tried that doesn't cause an exception, but also doesn't work (the resulting HTML has no <h2> element):
Closure testNested() {
println '---'
return { h2 "here's a subheading" }
}
// .... other stuff from above example not repeated ...
html.html {
head {...}
body(id: 'main') {
h1 "Report Title"
testNested()
}
I know I can easily do this inline. I'm trying to deepen my understanding of how Groovy uses closures and delegates in DSLs and clearly I'm missing something.
Consider the following code, which executes fine for me, using Groovy 2.4.5.
The builder pattern is a bit tricky because it can be viewed as hierarchical data and/or code, depending on your perspective. With practice, one can switch perspectives as necessary.
import groovy.xml.*
void testNested(def html) {
html.h2("here's a subheading from testNested")
}
void writeHtmlFile(def fileName) {
def writer = new FileWriter(fileName)
def html = new MarkupBuilder(writer)
html.mkp.yieldUnescaped '<!DOCTYPE html>'
html.mkp.xmlDeclaration(version: "1.0", encoding: "utf-8")
html.html {
body(id: 'main') {
h1 "Report Title"
testNested(html)
}
}
writer.flush()
writer.close()
}
writeHtmlFile("out.html")
Instead of having to declare all the properties in a map from an object like:
prop1: object.prop1
Can't you just drop the object in there like below somehow? Or what would be a proper way to achieve this?
results: [
object,
values: [
test: 'subject'
]
]
object.properties will give you a class as well
You should be able to do:
Given your POGO object:
class User {
String name
String email
}
def object = new User(name:'tim', email:'tim#tim.com')
Write a method to inspect the class and pull the non-synthetic properties from it:
def extractProperties(obj) {
obj.getClass()
.declaredFields
.findAll { !it.synthetic }
.collectEntries { field ->
[field.name, obj."$field.name"]
}
}
Then, map spread that into your result map:
def result = [
value: true,
*:extractProperties(object)
]
To give you:
['value':true, 'name':'tim', 'email':'tim#tim.com']
If you don't mind using a few libraries here's an option where you convert the object to json and then parse it back out as a map. I added mine to a baseObject which in your case object would extend.
class BaseObject {
Map asMap() {
def jsonSlurper = new groovy.json.JsonSlurperClassic()
Map map = jsonSlurper.parseText(this.asJson())
return map
}
String asJson(){
def jsonOutput = new groovy.json.JsonOutput()
String json = jsonOutput.toJson(this)
return json
}
}
Also wrote it without the json library originally. This is like the other answers but handles cases where the object property is a List.
class BaseObject {
Map asMap() {
Map map = objectToMap(this)
return map
}
def objectToMap(object){
Map map = [:]
for(item in object.class.declaredFields){
if(!item.synthetic){
if (object."$item.name".hasProperty('length')){
map."$item.name" = objectListToMap(object."$item.name")
}else if (object."$item.name".respondsTo('asMap')){
map << [ (item.name):object."$item.name"?.asMap() ]
} else{
map << [ (item.name):object."$item.name" ]
}
}
}
return map
}
def objectListToMap(objectList){
List list = []
for(item in objectList){
if (item.hasProperty('length')){
list << objectListToMap(item)
}else {
list << objectToMap(item)
}
}
return list
}
}
This seems to work well
*:object.properties
I have a function like this:
private downloadAllFiles() {
sftpRetriever.listFiles().findAll {
filter.isResponse(it) || filter.isResponseTurned(it)
}.each { String fileName ->
log.info 'Downloading file: {}', fileName
sftpRetriever.downloadFile(fileName)
log.info 'File downloaded'
removeRemoteFile(fileName)
}
}
I am looking for a simple way of modyfing this closure inside of that function so if the size() of findAll is 0 it will simply log 'No more files to download' and .each won't be executed. Is there any simple way to make it in single closure? It is really simply task if I divide it in several parts, but trying to learn closures here and improve my expressiveness :) Thank you in advance for your help.
Take a look at creature below :) It works due the fact that each returns the collection on which it's invoked (+ elvis operator and quite nice Groovy's truth evaluation):
def printContents(Collection collection) {
collection.each {
println it
} ?: println('Collection is empty')
}
printContents([1,2,'hello'])
printContents([])
I don't like this syntax but it's the shorter version which came to my mind.
You can also use metaprogramming to add the method provided by Steinar. It must be added to metaClass before first use but you'll avoid an effort to make extension module:
Collection.metaClass.doIfEmpty { Closure ifEmptyClosure ->
if (delegate.empty) {
ifEmptyClosure()
}
return delegate
}
def printContents(Collection collection) {
collection.doIfEmpty {
println "Collection is empty"
}.each {
println it
}
}
printContents([1,2,'hello'])
printContents([])
One rather generic and reusable option is to extend Collection using an extension module. This is surprisingly easy to do and is even recognized in IDE's (at least in IntelliJ) so you get code completion, etc.
For example, write an the extension class for collections which will perform the closure if the collection is empty. In addtion, it should always return the collection to allow further chaining:
package stackoverflow
class CollectionExtension {
static <T extends Collection> T doIfEmpty(T self, Closure closure) {
if (self.empty) {
closure()
}
return self
}
}
You will also need to tell groovy that this file is an extension module. Add a property file as a resource on the classpath: META-INF/services/org.codehaus.groovy.runtime.ExtensionModule (note: this name and location is mandatory for extension modules, i.e. you cannot change it).
moduleName=stackoverflow-module
moduleVersion=1.0
extensionClasses=stackoverflow.CollectionExtension
Finally a simple test script to show how this can be used:
def printContents(Collection collection) {
collection.doIfEmpty {
println "Collection is empty"
}.each {
println it
}
}
printContents([1,2,'hello'])
printContents([])
Output:
1
2
hello
Collection is empty
You may try the following piece of code:
def l1 = [1,2,3,4]
def l2 = [5,6,7,8]
def m(list) {
list.findAll { it < 5}.with { l ->
size > 0 ?
l.each { e ->
println e
}
:
println('Zero elements found')
}
}
m(l1)
m(l2)
No better idea at the moment.
My goal is to parse a large XML file and persist objects to DB based on the XML data, and to do it quickly. The operation needs to be transactional so I can rollback in case there is a problem parsing the XML or an object that gets created cannot be validated.
I am using the Grails Executor plugin to thread the operation. The problem is that each thread I create within the service has its own transaction and session. If I create 4 threads and 1 fails the session for the 3 that didn't fail may have already flushed, or they may flush in the future.
I was thinking if I could tell each thread to use the "current" Hibernate session that would probably fix my problem. Another thought I had was that I could prevent all sessions from flushing until it was known all completed without errors. Unfortunately I don't know how to do either of these things.
There is an additional catch too. There are many of these XML files to parse, and many that will be created in the future. Many of these XML files contain data that when parsed would create an object identical to one that was already created when a previous XML file was parsed. In such a case I need to make a reference to the existing object. I have added a transient isUnique variable to each class to address this. Using the Grails unique constraint does not work because it does not take hasMany relationships into account as I have outlined in my question here.
The following example is very simple compared to the real thing. The XML file's I'm parsing have deeply nested elements with many attributes.
Imagine the following domain classes:
class Foo {
String ver
Set<Bar> bars
Set<Baz> bazs
static hasMany = [bars: Bar, bazs: Baz]
boolean getIsUnique() {
Util.isUnique(this)
}
static transients = [
'isUnique'
]
static constraints = {
ver(nullable: false)
isUnique(
validator: { val, obj ->
obj.isUnique
}
)
}
}
class Bar {
String name
boolean getIsUnique() {
Util.isUnique(this)
}
static transients = [
'isUnique'
]
static constraints = {
isUnique(
validator: { val, obj ->
obj.isUnique
}
)
}
}
class Baz {
String name
boolean getIsUnique() {
Util.isUnique(this)
}
static transients = [
'isUnique'
]
static constraints = {
isUnique(
validator: { val, obj ->
obj.isUnique
}
)
}
}
And here is my Util.groovy class located in my src/groovy folder. This class contains the methods I use to determine if an instance of a domain class is unique and/or retrieve the already existing equal instance:
import org.hibernate.Hibernate
class Util {
/**
* Gets the first instance of the domain class of the object provided that
* is equal to the object provided.
*
* #param obj
* #return the first instance of obj's domain class that is equal to obj
*/
static def getFirstDuplicate(def obj) {
def objClass = Hibernate.getClass(obj)
objClass.getAll().find{it == obj}
}
/**
* Determines if an object is unique in its domain class
*
* #param obj
* #return true if obj is unique, otherwise false
*/
static def isUnique(def obj) {
getFirstDuplicate(obj) == null
}
/**
* Validates all of an object's constraints except those contained in the
* provided blacklist, then saves the object if it is valid.
*
* #param obj
* #return the validated object, saved if valid
*/
static def validateWithBlacklistAndSave(def obj, def blacklist = null) {
def propertiesToValidate = obj.domainClass.constraints.keySet().collectMany{!blacklist?.contains(it)? [it] : []}
if(obj.validate(propertiesToValidate)) {
obj.save(validate: false)
}
obj
}
}
And imagine XML file "A" is similar to this:
<foo ver="1.0">
<!-- Start bar section -->
<bar name="bar_1"/>
<bar name="bar_2"/>
<bar name="bar_3"/>
...
<bar name="bar_5000"/>
<!-- Start baz section -->
<baz name="baz_1"/>
<baz name="baz_2"/>
<baz name="baz_3"/>
...
<baz name="baz_100000"/>
</foo>
And imagine XML file "B" is similar to this (identical to XML file "A" except one new bar added and one new baz added). When XML file "B" is parsed after XML file "A" three new objects should be created 1.) A Bar with name = bar_5001 2.) A Baz with name = baz_100001, 3.) A Foo with ver = 2.0 and a list of bars and bazs equal to what is shown, reusing instances of Bar and Baz that already exist from the import of XML file A:
<foo ver="2.0">
<!-- Start bar section -->
<bar name="bar_1"/>
<bar name="bar_2"/>
<bar name="bar_3"/>
...
<bar name="bar_5000"/>
<bar name="bar_5001"/>
<!-- Start baz section -->
<baz name="baz_1"/>
<baz name="baz_2"/>
<baz name="baz_3"/>
...
<baz name="baz_100000"/>
<baz name="baz_100001"/>
</foo>
And a service similar to this:
class BigXmlFileUploadService {
// Pass in a 20MB XML file
def upload(def xml) {
String rslt = null
def xsd = Util.getDefsXsd()
if(Util.validateXmlWithXsd(xml, xsd)) { // Validate the structure of the XML file
def fooXml = new XmlParser().parseText(xml.getText()) // Parse the XML
def bars = callAsync { // Make a thread for creating the Bar objects
def bars = []
for(barXml in fooXml.bar) { // Loop through each bar XML element inside the foo XML element
def bar = new Bar( // Create a new Bar object
name: barXml.attribute("name")
)
bar = retrieveExistingOrSave(bar) // If an instance of Bar that is equal to this one already exists then use it
bars.add(bar) // Add the new Bar object to the list of Bars
}
bars // Return the list of Bars
}
def bazs = callAsync { // Make a thread for creating the Baz objects
def bazs = []
for(bazXml in fooXml.baz) { // Loop through each baz XML element inside the foo XML element
def baz = new Baz( // Create a new Baz object
name: bazXml.attribute("name")
)
baz = retrieveExistingOrSave(baz) // If an instance of Baz that is equal to this one already exists then use it
bazs.add(baz) // Add the new Baz object to the list of Bazs
}
bazs // Return the list of Bazs
}
bars = bars.get() // Wait for thread then call Future.get() to get list of Bars
bazs = bazs.get() // Wait for thread then call Future.get() to get list of Bazs
def foo = new Foo( // Create a new Foo object with the list of Bars and Bazs
ver: fooXml.attribute("ver")
bars: bars
bazs: bazs
).save()
rslt = "Successfully uploaded ${xml.getName()}!"
} else {
rslt = "File failed XSD validation!"
}
rslt
}
private def retrieveExistingOrSave(def obj, def existingObjCache) {
def dup = Util.getFirstDuplicate(obj)
obj = dup ?: Util.validateWithBlacklistAndSave(obj, ["isUnique"])
if(obj.errors.allErrors) {
log.error "${obj} has errors ${obj.errors}"
throw new RuntimeException() // Force transaction to rollback
}
obj
}
}
So the question is how do I get everything that happens inside of my service's upload method to act as it happened in a single session so EVERYTHING that happens can be rolled back if any one part fails?
You might not be able to do what you're trying to do.
First, a Hibernate session is not thread-safe:
A Session is an inexpensive, non-threadsafe object that should be used once and then discarded for: a single request, a conversation or a single unit of work. ...
Second, I don't think executing SQL queries in parallel will provide much benefit. I looked at how PostgreSQL's JDBC driver works and all the methods that actually run the queries are synchronized.
The slowest part of what you're doing is likely the XML processing so I'd recommend parallelizing that and doing persistence on a single thread. You could create several workers to read from the XML and add the objects to some sort of queue. Then have another worker that owns the Session and saves the objects as they're parsed.
You may also want to take a look at the Hibernate's batch processing doc page. Flushing after each insert is not the fastest way.
And finally, I don't know how your objects are mapped but you might run into problems saving Foo after all the child objects. Adding the objects to foo's collection will cause Hibernate to set the foo_id reference on each object and you'll end up with an update query for every object you inserted. You probably want to make foo first, and do baz.setFoo(foo) before each insert.
Service can be optimized to address some of the pain points:
I agree with #takteek, parsing the xml would be time consuming. So, plan to make that part async.
You do not need flush on each creation of child object. See below for the optimization.
Service class would look something like:
// Pass in a 20MB XML file
def upload(def xml) {
String rslt = null
def xsd = Util.getDefsXsd()
if (Util.validateXmlWithXsd(xml, xsd)) {
def fooXml = new XmlParser().parseText(xml.getText())
def foo = new Foo().save(flush: true)
def bars = callAsync {
saveBars(foo, fooXml)
}
def bazs = callAsync {
saveBazs(foo, fooXml)
}
//Merge the detached instances and check whether the child objects
//are populated or not. If children are
//Can also issue a flush, but we do not need it yet
//By default domain class is validated as well.
foo = bars.get().merge() //Future returns foo
foo = bazs.get().merge() //Future returns foo
//Merge the detached instances and check whether the child objects
//are populated or not. If children are
//absent then rollback the whole transaction
handleTransaction {
if(foo.bars && foo.bazs){
foo.save(flush: true)
} else {
//Else block will be reached if any of
//the children is not associated to parent yet
//This would happen if there was a problem in
//either of the thread, corresponding
//transaction would have rolled back
//in the respective sessions. Hence empty associations.
//Set transaction roll-back only
TransactionAspectSupport
.currentTransactionStatus()
.setRollbackOnly()
//Or throw an Exception and
//let handleTransaction handle the rollback
throw new Exception("Rolling back transaction")
}
}
rslt = "Successfully uploaded ${xml.getName()}!"
} else {
rslt = "File failed XSD validation!"
}
rslt
}
def saveBars(Foo foo, fooXml) {
handleTransaction {
for (barXml in fooXml.bar) {
def bar = new Bar(name: barXml.attribute("name"))
foo.addToBars(bar)
}
//Optional I think as session is flushed
//end of method
foo.save(flush: true)
}
foo
}
def saveBazs(Foo foo, fooXml) {
handleTransaction {
for (bazXml in fooXml.baz) {
def baz = new Baz(name: bazXml.attribute("name"))
foo.addToBazs(baz)
}
//Optional I think as session is flushed
//end of method
foo.save(flush: true)
}
foo
}
def handleTransaction(Closure clos){
try {
clos()
} catch (e) {
TransactionAspectSupport.currentTransactionStatus().setRollbackOnly()
}
if (TransactionAspectSupport.currentTransactionStatus().isRollbackOnly())
TransactionAspectSupport.currentTransactionStatus().setRollbackOnly()
}
Imagine I have this structure:
class Foo {
String bar
}
Now imagine I have several instance of Foo whose bar value is baz_1, baz_2, and zab_3.
I want to write a collect statement that only collects the bar values which contain the text baz. I cannot get it to work, but it would look something like this:
def barsOfAllFoos = Foo.getAll().bar
assert barsOfAllFoos == [ 'baz_1', 'baz_2', 'zab_3' ]
def barsWithBaz = barsOfAllFoos.collect{ if( it.contains( "baz" ) { it } ) } // What is the correct syntax for this?
assert barsWithBaz == [ 'baz_1', 'baz_2' ]
You need findAll:
barsOfAllFoos.findAll { it.contains 'baz' }
If you want to both filter and transform there's lots of ways to do this. After 1.8.1 I'd go with #findResults and a closure that returns null for the elements I want to skip.
def frob(final it) { "frobbed $it" }
final barsWithBaz = barsOfAllFoos.findResults {
it.contains('baz')? frob(it) : null
}
In earlier versions you can use #findAll and #collect
final barsWithBaz = barsOfAllFoos
. findAll { it.contains('baz') }
. collect { frob(it) }
Or #sum
final barsWithBaz = barsOfAllFoos.sum([]) {
it.contains('baz')? [frob(it)] : []
}
Or #inject
final barsWithBaz = barsOfAllFoos.inject([]) {
l, it -> it.contains('baz')? l << frob(it) : l
}
Using findResults did not work for me... If you want to collect a transformed version of the values matching the condition (for instance a regex search of many lines) you can use collect followed by find or findAll as follows.
def html = """
<p>this is some example data</p>
<script type='text/javascript'>
form.action = 'http://www.example.com/'
// ...
</script>
"""
println("Getting url from html...")
// Extract the url needed to upload the form
def url = html.split("\n").collect{line->
def m = line =~/.*form\.action = '(.+)'.*/
if (m.matches()) {
println "Match found!"
return m[0][1]
}
}.find()
println "url = '${url}'"
This returns the part of the line matching the given pattern.
Getting url from html...
Match found!
url = 'http://www.example.com/'