dynamic template generation and formatting using freemarker - spring-integration

My goal is to format a collection of java map to a string (basically a csv) using free marker or anything else that would do smartly. I want to generate the template using a configuration data stored in database and managed from an admin application.
The configuration will tell me at what position a given data (key in hash map) need to go and also if any script need to run on this data before applying it at a given position. Several positions may be blank if the data in not in map.
I am thinking to use free-marker to build this generic tool and would appreciate if you could share how I should go about this.
Also would like to know if there is any built is support in spring-integration for building such process as the application is a SI application.

I am no freemarker expert, but a quick look at their quick start docs led me here...
public class FreemarkerTransformerPojo {
private final Configuration configuration;
private final Template template;
public FreemarkerTransformerPojo(String ftl) throws Exception {
this.configuration = new Configuration(Configuration.VERSION_2_3_23);
this.configuration.setDirectoryForTemplateLoading(new File("/"));
this.configuration.setDefaultEncoding("UTF-8");
this.template = this.configuration.getTemplate(ftl);
}
public String transform(Map<?, ?> map) throws Exception {
StringWriter writer = new StringWriter();
this.template.process(map, writer);
return writer.toString();
}
}
and
public class FreemarkerTransformerPojoTests {
#Test
public void test() throws Exception {
String template = System.getProperty("user.home") + "/Development/tmp/test.ftl";
OutputStream os = new FileOutputStream(new File(template));
os.write("foo=${foo}, bar=${bar}".getBytes());
os.close();
FreemarkerTransformerPojo transformer = new FreemarkerTransformerPojo(template);
Map<String, String> map = new HashMap<String, String>();
map.put("foo", "baz");
map.put("bar", "qux");
String result = transformer.transform(map);
assertEquals("foo=baz, bar=qux", result);
}
}
From a Spring Integration flow, send a message with a Map payload to
<int:transformer ... ref="fmTransformer" method="transform" />
Or you could do it with a groovy script (or other supported scripting language) using Spring Integration's existing scripting support without writing any code (except the script).

Related

Spring Integration - Customize ObjectMapper used by WebFlux OutboundGateway

How do we customize the Jackson ObjectMapper used by WebFlux OutboundGateway? The normal customization done via Jackson2ObjectMapperBuilder or Jackson2ObjectMapperBuilderCustomizer is NOT respected.
Without this customization, LocalDate is serialized as SerializationFeature.WRITE_DATES_AS_TIMESTAMPS. Sample output - [2022-10-20] and there is NO way to customize the format
I assume you really talk about Spring Boot auto-configuration which is applied to the WebFlux instance. Consider to use an overloaded WebFlux.outboundGateway(String uri, WebClient webClient) to be able to auto-wire a WebClient.Builder which might be already configured with the mentioned customized ObjectMapper.
Registering a bean of type com.fasterxml.jackson.databind.module.SimpleModule will automatically be used by the pre-configured ObjectMapper bean. In SimpleModule, it is possible to register custom serialization and deserialization specifications.
To put that into code, a very simple solution would be the following:
#Bean
public SimpleModule odtModule() {
SimpleModule module = new SimpleModule();
JsonSerializer<LocalDate> serializer = new JsonSerializer<>() {
#Override
public void serialize(LocalDate odt, JsonGenerator jgen, SerializerProvider provider) throws IOException {
String formatted = odt.format(DateTimeFormatter.ISO_LOCAL_DATE);
jgen.writeString(formatted);
}
};
JsonDeserializer<LocalDate> deserializer = new JsonDeserializer<>() {
#Override
public LocalDate deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException {
return LocalDate.parse(jsonParser.getValueAsString());
}
};
module.addSerializer(LocalDate.class, serializer);
module.addDeserializer(LocalDate.class, deserializer);
return module;
}
Note that using lambdas for the implementations has sometimes resulted in weird behaviors for me, so I tend not to do that.

How to write JavaRDD to marklogic database

I am evaluating spark with marklogic database. I have read a csv file, now i have a JavaRDD object which i have to dump into marklogic database.
SparkConf conf = new SparkConf().setAppName("org.sparkexample.Dataload").setMaster("local");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> data = sc.textFile("/root/ml/workArea/data.csv");
SQLContext sqlContext = new SQLContext(sc);
JavaRDD<Record> rdd_records = data.map(
new Function<String, Record>() {
public Record call(String line) throws Exception {
String[] fields = line.split(",");
Record sd = new Record(fields[0], fields[1], fields[2], fields[3],fields[4]);
return sd;
}
});
This JavaRDD object i want to write to marklogic database.
Is there any spark api available for faster writing to the marklogic database ?
Lets say, If we could not write JavaRDD directly to marklogic then what is the currect approach to achieve this ?
Here is the code which i am using to write the JavaRDD data to marklogic database, let me know if it is wrong way to do that.
final DatabaseClient client = DatabaseClientFactory.newClient("localhost",8070, "MLTest");
final XMLDocumentManager docMgr = client.newXMLDocumentManager();
rdd_records.foreachPartition(new VoidFunction<Iterator<Record>>() {
public void call(Iterator<Record> partitionOfRecords) {
while (partitionOfRecords.hasNext()) {
Record record = partitionOfRecords.next();
System.out.println("partitionOfRecords - "+record.toString());
String docId = "/example/"+record.getID()+".xml";
JAXBContext context = JAXBContext.newInstance(Record.class);
JAXBHandle<Record> handle = new JAXBHandle<Record>(context);
handle.set(record);
docMgr.writeAs(docId, handle);
}
}
});
client.release();
I have used java client api to write the data, but i am getting below exception even though POJO class Record is implementing Serializable interface. Please let me know what could be the reason & how to solve that.
org.apache.spark.sparkexception task not Serializable .
The easiest way to get data into MarkLogic is via HTTP and the client REST API - specifically the /v1/documents endpoints - http://docs.marklogic.com/REST/client/management .
There are a variety of ways to optimize this, such as via a write set, but based on your question, I think the first thing to decide is - what kind of document do you want to write for each Record? Your example shows 5 columns in the CSV - typically, you'll write either a JSON or XML document with 5 fields/elements, each named based on the column index. So you'd need to write a little code to generate that JSON/XML, and then use whatever HTTP client you prefer (and one option is the MarkLogic Java Client API) to write that document to MarkLogic.
That addresses your question of how to write a JavaRDD to MarkLogic - but if your goal is to get data from a CSV into MarkLogic as fast as possible, then skip Spark and use mlcp - https://docs.marklogic.com/guide/mlcp/import#id_70366 - which involves zero coding.
Modified example from spark streaming guide, Here you will have to implement connection and writing logic specific to database.
public void send(JavaRDD<String> rdd) {
rdd.foreachPartition(new VoidFunction<Iterator<String>>() {
#Override
public void call(Iterator<String> partitionOfRecords) {
// ConnectionPool is a static, lazily initialized pool of
Connection connection = ConnectionPool.getConnection();
while (partitionOfRecords.hasNext()) {
connection.send(partitionOfRecords.next());
}
ConnectionPool.returnConnection(connection); // return to the pool
// for future reuse
}
});
}
I'm wondering if you just need to make sure everything you access inside your VoidFunction that was instantiated outside it is serializable (see this page). DatabaseClient and XMLDocumentManager are of course not serializable, as they're connected resources. You're right, however, to not instantiate DatabaseClient inside your VoidFunction as that would be less efficient (though it would work). I don't know if the following idea would work with spark. But I'm guessing you could create a class that keeps hold of a singleton DatabaseClient instance:
public static class MLClient {
private static DatabaseClient singleton;
private MLClient() {}
public static DatabaseClient get(DatabaseClientFactory.Bean connectionInfo) {
if ( connectionInfo == null ) {
throw new IllegalArgumentException("connectionInfo cannot be null");
}
if ( singleton == null ) {
singleton = connectionInfo.newClient();
}
return singleton;
}
}
then you just create a serializable DatabaseClientFactory.Bean outside your VoidFunction so your auth info is still centralized
DatabaseClientFactory.Bean connectionInfo =
new DatabaseClientFactory.Bean();
connectionInfo.setHost("localhost");
connectionInfo.setPort(8000);
connectionInfo.setUser("admin");
connectionInfo.setPassword("admin");
connectionInfo.setAuthenticationValue("digest");
Then inside your VoidFunction you could get that singleton DatabaseClient and new XMLDocumentManager like so:
DatabaseClient client = MLClient.get(connectionInfo);
XMLDocumentManager docMgr = client.newXMLDocumentManager();

Lost headers when using UnZipResultSplitter

I'm using the Spring Integration Zip extension and it appears that I'm losing headers I've added upstream in the flow. I'm guessing that they are being lost in UnZipResultSplitter.splitUnzippedMap() as I don't see anything that explicitly copies them over.
I seem to recall that this is not unusual with splitters but I can't determine what strategy one should use in such a case.
Yep!
It looks like a bug.
The splitter contract is like this:
if (item instanceof Message) {
builder = this.getMessageBuilderFactory().fromMessage((Message<?>) item);
}
else {
builder = this.getMessageBuilderFactory().withPayload(item);
builder.copyHeaders(headers);
}
So, if those splitted items are messages already, like in case of our UnZipResultSplitter, we just use message as is without copying headers from upstream.
Please, raise a JIRA ticket (https://jira.spring.io/browse/INTEXT) on the matter.
Meanwhile let's consider some workaround:
public class MyUnZipResultSplitter {
public List<Message<Object>> splitUnzipped(Message<Map<String, Object>> unzippedEntries) {
final List<Message<Object>> messages = new ArrayList<Message<Object>>(unzippedEntries.size());
for (Map.Entry<String, Object> entry : unzippedEntries.getPayload().entrySet()) {
final String path = FilenameUtils.getPath(entry.getKey());
final String filename = FilenameUtils.getName(entry.getKey());
final Message<Object> splitMessage = MessageBuilder.withPayload(entry.getValue())
.setHeader(FileHeaders.FILENAME, filename)
.setHeader(ZipHeaders.ZIP_ENTRY_PATH, path)
.copyHeaders(unzippedEntries/getHeaders())
.build();
messages.add(splitMessage);
}
return messages;
}
}

How to get the Gherkin feature description runtime in java

I need to report the feature description for the scenario that is being executes to report to other system.
Was able to get the scenario name from cucumber.api.Scenario; how I can the feature description ?
Is there any interface that I can use?
Using cucumber-Jvm, get the feature description runtime; as each scenario being executed might be from different feature files.
You can get the description of a feature by retrieving the Gherkin feature from CucumberFeature:
List<CucumberFeature> cucumberFeatures = new ArrayList<>();
FeatureBuilder featureBuilder = new FeatureBuilder(cucumberFeatures);
featureBuilder.parse(new FileResource(featureFile.getParentFile(), featureFile), new ArrayList());
for (CucumberFeature feature: cucumberFeatures) {
// Here we retrieve the Gherkin model
Feature f = feature.getGherkinFeature();
// Here we get name and description of the feature.
System.out.format("%s: %s%n", f.getName(), f.getDescription());
}
Another solution is to implement your own formatter, and do the parsing with Gherkin directly:
public class MyFormatter implements Formatter {
private List<Feature> features = new ArrayList<>();
public static void main(String... args) throws Exception {
OutputStreamWriter out = new OutputStreamWriter(System.out, "UTF-8");
// Read the feature file into a string.
File f = new File("/path/to/file.feature");
String input = FixJava.readReader(new FileReader(f));
// Parse the gherkin string with our own formatter.
MyFormatter formatter = new MyFormatter();
Parser parser = new Parser(formatter);
parser.parse(input, f.getPath(), 0);
for (Feature feature: formatter.features) {
System.out.format("%s: %s%n", feature.getName(), feature.getDescription());
}
}
#Override
public void feature(Feature feature) {
features.add(feature);
}
// ...
// follow all the Formatter methods to implement.
}

Global Static Dictionary Initialization from Database in Webapi

I want to Initialize a global Dictionary from Database in my web Api. Do i need to inject my DBContext in Global.Asax or Owin Startup. Any example would be much appreciated.
Any kind initialization purposes can be made in your custom defined OWIN Startup class class, like this:
using Microsoft.Owin;
using Microsoft.Owin.Security.OAuth;
using Owin;
using System;
[assembly: OwinStartup(typeof(WebAPIRestWithNest.Startup))]
namespace YourNamespace
{
public class Startup
{
public Dictionary<string, string> Table {get; private set;}
public void Configuration(IAppBuilder app)
{
// token generation
app.UseOAuthAuthorizationServer(new OAuthAuthorizationServerOptions
{
AllowInsecureHttp = false,
TokenEndpointPath = new PathString("/token"),
AccessTokenExpireTimeSpan = TimeSpan.FromHours(8),
Provider = new SimpleAuthorizationServerProvider()
});
// token consumption
app.UseOAuthBearerAuthentication(new OAuthBearerAuthenticationOptions());
app.UseWebApi(WebApiConfig.Register());
Table = ... Connect from DB and fill your table logic ...
}
}
}
After that you can use your Startup.Table property from your application.
In general, it is bad practice to access objects using static field in the asp.net applications because this may lead to bugs that are hardly detected and reproduced: especially this is true for non-immutable/not-thread-safe objects like Dictionary.
I assume you want to cache some DB data in memory to avoid excessive SQL queries. It is good idea to use standard asp.net caching for this purpose:
public IDictionary GetDict() {
var dict = HttpRuntime.Cache.Get("uniqueCacheKey") as IDictionary;
if (pvtData==null) {
dict = doLoadDictionaryFromDB(); // your code that loads data from DB
HttpRuntime.Cache.Add(cacheKey, dict,
null, Cache.NoAbsoluteExpiration,
new TimeSpan(0,5,0), // cache at least for 5 minutes after last access
CacheItemPriority.Normal, null);
}
return dict;
}
This approach allows you to select appropriate expiration policy (without the need to reinventing the wheel with static dictionary).
If you still want to use static dictionary, you can populate it on the application start (global.asax):
void Application_Start(object sender, EventArgs e)
{
// your code that initializes dictionary with data from DB
}

Resources