Spring Integration Cassandra persistence workflow - cassandra

I try to realize the following workflow with Spring Integration:
1) Poll REST API
2) store the POJO in Cassandra cluster
It's my first try with Spring Integration, so I'm still a bit overwhelmed about the mass of information from the reference. After some research, I could make the following work.
1) Poll REST API
2) Transform mapped POJO JSON result into a string
3) save string into file
Here's the code:
#Configuration
public class ConsulIntegrationConfig {
#InboundChannelAdapter(value = "consulHttp", poller = #Poller(maxMessagesPerPoll = "1", fixedDelay = "1000"))
public String consulAgentPoller() {
return "";
}
#Bean
public MessageChannel consulHttp() {
return MessageChannels.direct("consulHttp").get();
}
#Bean
#ServiceActivator(inputChannel = "consulHttp")
MessageHandler consulAgentHandler() {
final HttpRequestExecutingMessageHandler handler =
new HttpRequestExecutingMessageHandler("http://localhost:8500/v1/agent/self");
handler.setExpectedResponseType(AgentSelfResult.class);
handler.setOutputChannelName("consulAgentSelfChannel");
LOG.info("Created bean'consulAgentHandler'");
return handler;
}
#Bean
public MessageChannel consulAgentSelfChannel() {
return MessageChannels.direct("consulAgentSelfChannel").get();
}
#Bean
public MessageChannel consulAgentSelfFileChannel() {
return MessageChannels.direct("consulAgentSelfFileChannel").get();
}
#Bean
#ServiceActivator(inputChannel = "consulAgentSelfFileChannel")
MessageHandler consulAgentFileHandler() {
final Expression directoryExpression = new SpelExpressionParser().parseExpression("'./'");
final FileWritingMessageHandler handler = new FileWritingMessageHandler(directoryExpression);
handler.setFileNameGenerator(message -> "../../agent_self.txt");
handler.setFileExistsMode(FileExistsMode.APPEND);
handler.setCharset("UTF-8");
handler.setExpectReply(false);
return handler;
}
}
#Component
public final class ConsulAgentTransformer {
#Transformer(inputChannel = "consulAgentSelfChannel", outputChannel = "consulAgentSelfFileChannel")
public String transform(final AgentSelfResult json) throws IOException {
final String result = new StringBuilder(json.toString()).append("\n").toString();
return result;
}
This works fine!
But now, instead of writing the object to a file, I want to store it in a Cassandra cluster with spring-data-cassandra. For that, I commented out the file handler in the config file, return the POJO in transformer and created the following, :
#MessagingGateway(name = "consulCassandraGateway", defaultRequestChannel = "consulAgentSelfFileChannel")
public interface CassandraStorageService {
#Gateway(requestChannel="consulAgentSelfFileChannel")
void store(AgentSelfResult agentSelfResult);
}
#Component
public final class CassandraStorageServiceImpl implements CassandraStorageService {
#Override
public void store(AgentSelfResult agentSelfResult) {
//use spring-data-cassandra repository to store
LOG.info("Received 'AgentSelfResult': {} in Cassandra cluster...");
LOG.info("Trying to store 'AgentSelfResult' in Cassandra cluster...");
}
}
But this seems to be a wrong approach, the service method is never triggered.
So my question is, what would be a correct approach for my usecase? Do I have to implement the MessageHandler interface in my service component, and use a #ServiceActivator in my config. Or is there something missing in my current "gateway-approach"?? Or maybe there is another solution, that I'm not able to see..
Like mentioned before, I'm new to SI, so this may be a stupid question...
Nevertheless, thanks a lot in advance!

It's not clear how you are wiring in your CassandraStorageService bean.
The Spring Integration Cassandra Extension Project has a message-handler implementation.
The Cassandra Sink in spring-cloud-stream-modules uses it with Java configuration so you can use that as an example.

So I finally made it work. All I needed to do was
#Component
public final class CassandraStorageServiceImpl implements CassandraStorageService {
#ServiceActivator(inputChannel="consulAgentSelfFileChannel")
#Override
public void store(AgentSelfResult agentSelfResult) {
//use spring-data-cassandra repository to store
LOG.info("Received 'AgentSelfResult': {}...");
LOG.info("Trying to store 'AgentSelfResult' in Cassandra cluster...");
}
}
The CassandraMessageHandler and the spring-cloud-streaming seemed to be a to big overhead to my use case, and I didn't really understand yet... And with this solution, I keep control over what happens in my spring component.

Related

Basic Example using Spring Integration to copy file to another directory raising exception

I am a beginner in Spring Integration. I wrote this code which is in spring boot and it is raising exception "Bean named 'messageSource' is expected to be of type 'org.springframework.context.MessageSource' but was actually of type 'org.springframework.integration.file.FileReadingMessageSource'"
Code:
#Configuration
/*#EnableIntegration annotation designates this class as a Spring Integration configuration.*/
#EnableIntegration
public class SIConfig {
#Bean
public MessageChannel channel() {
return new DirectChannel();
}
//bydefault name of method
#Bean
public MessageSource messageSource() {
FileReadingMessageSource ms= new FileReadingMessageSource();
ms.setDirectory(new File("C:\\Users\\payal\\Pictures"));
ms.setFilter(new SimplePatternFileListFilter("*.mp4"));
return ms;
}
#Bean
public MessageHandler handler() {
FileWritingMessageHandler handler= new FileWritingMessageHandler(new File("C:\\Users\\payal\\Documents\\batch7"));
handler.setFileExistsMode(FileExistsMode.IGNORE);
handler.setExpectReply(false);
return handler;
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(messageSource(), configurer -> configurer.poller(Pollers.fixedDelay(10000)))
.channel(channel())
.handle(handler()).get();
}
}
Since using boot, versions are automatically managed
Uploaded code n GitHub too:
https://github.com/LearningNewTechnology/SpringIntegrationOne
Any help would be really appreciated.
Change the name of the bean to, e.g.
#Bean
public MessageSource myMessageSource() {
Spring framework (context) has another type of MessageSource and Spring Boot autoconfiguration creates a bean of that type with name messageSource so your bean is colliding with that.

spring batch integration file poller

I am trying to build a spring batch application that starts a job only after a file comes into a directory. For that I need a file poller and something like the snippet found in Spring reference manual:
public class FileMessageToJobRequest {
private Job job;
private String fileParameterName;
public void setFileParameterName(String fileParameterName) {
this.fileParameterName = fileParameterName;
}
public void setJob(Job job) {
this.job = job;
}
#Transformer
public JobLaunchRequest toRequest(Message<File> message) {
JobParametersBuilder jobParametersBuilder =
new JobParametersBuilder();
jobParametersBuilder.addString(fileParameterName,
message.getPayload().getAbsolutePath());
return new JobLaunchRequest(job, jobParametersBuilder.toJobParameters());
}
}
I would like to manage everything with configuration classes, but I can't really figure out how to make it work.
Your question isn't clear. Would be better to have something that works, then some your own PoC or attempt to reach the task.
But anyway that looks like you would like to avoid XML configuration and be only with Java & Annotation Configuration.
For this purpose I suggest you to take a look into Reference Manual and find this sample in the File Support chapter, too:
#Bean
#InboundChannelAdapter(value = "fileInputChannel", poller = #Poller(fixedDelay = "1000"))
public MessageSource<File> fileReadingMessageSource() {
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory(new File(INBOUND_PATH));
source.setFilter(new SimplePatternFileListFilter("*.txt"));
return source;
}

Spring integration with retry with ExceptionClassifierRetryPolicy

I am using int:request-handler-advice-chain with my service activator. It is working correctly with org.springframework.retry.policy.SimpleRetryPolicy however I would like to use org.springframework.retry.policy.ExceptionClassifierRetryPolicy to allow for a different number of retries based on the exception thrown by the service activator.
The problem I am having is that by the time the exception gets to the ExceptionClassifierRetryPolicy it is a
org.springframework.integration.MessageHandlingException
Can anyone advise on the best approach for get the cause (i.e my exception) from the MessageHandlingException made available to the ExceptionClassifierRetryPolicy?
Solution thanks to Artem's suggestion below:
Create a subclass of SubclassClassifier that returns the cause in the case of MessagingException
public class MessagingCauseExtractingSubclassClassifier extends SubclassClassifier<Throwable, RetryPolicy> {
private static final Logger LOG = LoggerFactory.getLogger(MessagingCauseExtractingSubclassClassifier.class);
public MessagingCauseExtractingSubclassClassifier(final Map<Class<? extends Throwable>, RetryPolicy> policyMap, final RetryPolicy retryPolicy) {
super(policyMap, retryPolicy);
}
#Override
public RetryPolicy classify(final Throwable throwable) {
Throwable t = throwable;
if (t instanceof MessagingException) {
t = t.getCause();
LOG.debug("Throwable is instanceof MessagingException so classifying cause type: {}", t.getClass());
}
return super.classify(t);
}
}
Then a new ExceptionClassifierRetryPolicy subclass that uses the new classifier and policyMap
public class MessasgeCauseExtractingExceptionClassifierRetryPolicy extends ExceptionClassifierRetryPolicy {
#Override
public void setPolicyMap(final Map<Class<? extends Throwable>, RetryPolicy> policyMap) {
final MessagingCauseExtractingSubclassClassifier classifier = new MessagingCauseExtractingSubclassClassifier(
policyMap, new NeverRetryPolicy());
setExceptionClassifier(classifier);
}
}
Currently this won't support retying on MessagingException but this is fine for our use case. Otherwise works perfectly.
The BinaryExceptionClassifier has traverseCauses option to analize the whole StackTrace until the proper condition.
Exactly this option is with one of SimpleRetryPolicy constructor:
public SimpleRetryPolicy(int maxAttempts, Map<Class<? extends Throwable>, Boolean> retryableExceptions,
boolean traverseCauses) {
Please, take a look if that variant is feasible for you.

How to write client proxy for SPI and what the difference between client and server proxies?

I have developed own idGenerator based on Hazelcast IdGenerator class (with storing each last_used_id into db). Now I want to run hazelcast cluster as a single java application and my web-application as other app (web-application restart shouldn't move id values to next block). I move MyIdGeneratorProxy and MyIdGeneratorService to new application, run it, run web-application as a hazelcast-client and get
IllegalArgumentException: No factory registered for service: ecs:impl:idGeneratorService
It was okay when client and server were the same application.
It seems it's unable to process without some clientProxy. I have compared IdGeneratorProxy and ClientIdGeneratorProxy and it looks the same. What is the idea? How to write client proxy for services? I have found no documentation yet. Is direction of investigations correct? I thought it is possible to divide hazelcast inner services (like a id generator service) and my business-processes. Should I store custom ClientProxy (for custom spi) in my web-application?
This is a demo how to create a client proxy, the missing part CustomClientProxy function call, is quit complicated(more like a server proxy,here is called ReadRequest, the server is called Operation), you can find a how AtomicLong implement.For every client proxy method you have to make a request.
#Test
public void client() throws InterruptedException, IOException
{
ClientConfig cfg = new XmlClientConfigBuilder("hazelcast-client.xml").build();
ServiceConfig serviceConfig = new ServiceConfig();
serviceConfig.setName(ConnectorService.NAME)
.setClassName(ConnectorService.class.getCanonicalName())
.setEnabled(true);
ProxyFactoryConfig proxyFactoryConfig = new ProxyFactoryConfig();
proxyFactoryConfig.setService(ConnectorService.NAME);
proxyFactoryConfig.setClassName(CustomProxyFactory.class.getName());
cfg.addProxyFactoryConfig(proxyFactoryConfig);
HazelcastInstance hz = HazelcastClient.newHazelcastClient(cfg);
Thread.sleep(1000);
for (int i = 0; i < 10; i++)
{
Connector c = hz.getDistributedObject(ConnectorService.NAME, "Connector:" + ThreadLocalRandom.current()
.nextInt(10000));
System.out.println(c.snapshot());
}
}
private static class CustomProxyFactory implements ClientProxyFactory
{
#Override
public ClientProxy create(String id)
{
return new CustomClientProxy(ConnectorService.NAME, id);
}
}
private static class CustomClientProxy extends ClientProxy implements Connector
{
protected CustomClientProxy(String serviceName, String objectName)
{
super(serviceName, objectName);
}
#Override
public ConnectorState snapshot()
{
return null;
}
#Override
public void loadState(ConnectorState state)
{
}
#Override
public boolean reconnect(HostNode node)
{
return false;
}
#Override
public boolean connect()
{
return false;
}
}
EDIT
In hazelcast the IdGenerate is implemented as a wrapper for AtomicLong, you should implement you IdGenerate by you own, instead of extend IdGenerate.
So you have to implement these(more like a todo list XD):
API
interface MyIdGenerate
Server
MyIdGenerateService
MyIdGenerateProxy
MyIdGenerateXXXOperation
Client
ClientMyIdGenerateFactory
ClientMyIdGenerateProxy
MyIdGenerateXXXRequest
I also made a sequence(same as IdGenerate) here, this is backed by zookeeper or redis,also it's easy to add a db backend,too.I will integrate to hazelcast if I got time.

Spring Integration 4 - configuring a LoadBalancingStrategy in Java DSL

I have a simple Spring Integration 4 Java DSL flow which uses a DirectChannel's LoadBalancingStrategy to round-robin Message requests to a number of possible REST Services (i.e. calls a REST service from one of two possible service endpoint URIs).
How my flow is currently configured:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
LoadBalancingStrategy loadBalancingStrategy = new RoundRobinLoadBalancingStrategy();
DirectChannel directChannel = new DirectChannel(loadBalancingStrategy);
return directChannel;
}
#Bean
public IntegrationFlow testLoadBalancing0Flow() {
return IntegrationFlows.from("test.load.balancing.ch")
.handle(restHandler0())
.channel("test.result.ch")
.get();
}
#Bean
public IntegrationFlow testLoadBalancing1Flow() {
return IntegrationFlows.from("test.load.balancing.ch")
.handle(restHandler1())
.channel("test.result.ch")
.get();
}
#Bean
public HttpRequestExecutingMessageHandler restHandler0() {
return createRestHandler(endpointUri0, 0);
}
#Bean
public HttpRequestExecutingMessageHandler restHandler1() {
return createRestHandler(endpointUri1, 1);
}
private HttpRequestExecutingMessageHandler createRestHandler(String uri, int order) {
HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(uri);
// handler configuration goes here..
handler.setOrder(order);
return handler;
}
My configuration works, but I am wondering whether there is a simpler/better way of configuring the flow using Spring Integration's Java DSL?
Cheers,
PM
First of all the RoundRobinLoadBalancingStrategy is the default one for the DirectChannel.
So, can get rid of the testLoadBalancingCh() bean definition at all.
Further, to avoid duplication for the .channel("test.result.ch") you can configure it on the HttpRequestExecutingMessageHandler as setOutputChannel().
From other side your configuration is so simple that I don't see reason to use DSL. You can achieve the same just with annotation configuration:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
return new DirectChannel();
}
#Bean(name = "test.result.ch")
public DirectChannel testResultCh() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "test.load.balancing.ch")
public HttpRequestExecutingMessageHandler restHandler0() {
return createRestHandler(endpointUri0, 0);
}
#Bean
#ServiceActivator(inputChannel = "test.load.balancing.ch")
public HttpRequestExecutingMessageHandler restHandler1() {
return createRestHandler(endpointUri1, 1);
}
private HttpRequestExecutingMessageHandler createRestHandler(String uri, int order) {
HttpRequestExecutingMessageHandler handler = new HttpRequestExecutingMessageHandler(uri);
// handler configuration goes here..
handler.setOrder(order);
handler.setOutputChannel(testResultCh());
return handler;
}
From other side there is MessageChannels builder factory to allow to simplify loadBalancer for your case:
#Bean(name = "test.load.balancing.ch")
public DirectChannel testLoadBalancingCh() {
return MessageChannels.direct()
.loadBalancer(new RoundRobinLoadBalancingStrategy())
.get();
}
However, I can guess that you want to avoid duplication within DSL flow definition to DRY, but it isn't possible now. That's because IntegrationFlow is linear to tie endoints bypassing the boilerplate code for standard objects creation.
As you see to achieve Round-Robin we have to duplicate, at least, inputChannel, to subscribe several MessageHandlers to the same channel. And we do that in the XML, via Annotations and, of course, from DSL.
I'm not sure that it will be so useful for real applications to provide a hook to configure several handlers using single .handle() for the same Round-Robin channel. Because the further downstream flow may not be so simple as your .channel("test.result.ch").
Cheers

Resources