Spring Integrtion XML and Java Config Conversion - spring-integration

I am very new to Spring Integration and my project is using File Support to read a file and load into data base.
I have XML config , trying to understand it's content.
<int-file:inbound-channel-adapter auto-startup= true channel="channelOne" directory="${xx}" filename-regex="${xx}" id="id" prevent-duplicates="false">
<int:poller fixed-delay="1000" receive-timeout="5000"/>
</int-file:inbound-channel-adapter>
<int:channel id="channelOne"/>
From the above piece, my understanding is :
We define a channel and
Then define inbound-channel-adapter - this will look into directory for the file and create a message with file as a payload.
I was able to convert this in JavaConfig as below :
#Bean
public MessageChannel fileInputChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "fileInputChannel", poller = #Poller(fixedDelay = "1000"))
public MessageSource<File> fileReadingMessageSource() {
FileReadingMessageSource sourceReader= new FileReadingMessageSource();
RegexPatternFileListFilter regexPatternFileListFilter = new RegexPatternFileListFilter(
file-regex);
//List<FileListFilter<File>> fileListFilter = new ArrayList<FileListFilter<File>>();
fileListFilter.add(regexPatternFileListFilter);
//CompositeFileListFilter compositeFileListFilter = new CompositeFileListFilter<File>(
fileListFilter);
sourceReader.setDirectory(new File(inputDirectorywhereFileComes));
sourceReader.setFilter(regexPatternFileListFilter );
return sourceReader;
}
Then the next piece of code , which literally I am struggling to understand and moreover to convert to JavaConfig.
Here is the next piece:
<int-file:outbound-gateway
delete-source-files="true"
directory="file:${pp}"
id="id"
reply-channel="channelTwo"
request-channel="channelOne"
temporary-file-suffix=".tmp"/>
<int:channel id="channelTwo"/>
<int:outbound-channel-adapter channel="channelTwo" id="id" method="load" ref="beanClass"/>
So from this piece , my understanding :
1: Define an output channel.
2: Define an outbound-gateway, which will write that message as a file again in directory(other one), also remove file from source directory. And finally it will call the method Load of Bean Class. This is our class and has load method which takes file as input and load it to DB.
I tried to covert it into Java Config. Here is my code:
#Bean
#ServiceActivator(inputChannel= "fileInputChannel")
public MessageHandler fileWritingMessageHandler() throws IOException, ParseException {
FileWritingMessageHandler handler = new FileWritingMessageHandler(new File(path to output directory));
handler.setFileExistsMode(FileExistsMode.REPLACE);
beaObject.load(new File(path to output directory or input directory:: Nothing Worked));
handler.setDeleteSourceFiles(true);
handler.setOutputChannel(fileOutputChannel());
return handler;
}
I am able to write this file to output folder also was able to delete from source. After that I am totally lost. I have to call method Load of my BeanClass(ref=class in XML ).
I tried a lot, but not able to get it. Read multiple times the integration File Support doc, but couldn't make it.
Note: When I tried , I got one error saying , the File Not Found Exception. I believe , I am able to call my method , but can not get the file.
This XML config is working perfectly fine.
Spring Integration with DSL also anyone can suggest, if possible.
Please help me to understand the basic flow and get this thing done. Any help and comments is really appreciable.
Thanks in advance.

First of all you need to understand that #Bean method is exactly for configuration and components definitions which are going to be used later at runtime. You definitely must not call a business logic in the #Bean. I mean that your beaObject.load() is totally wrong.
So, please, go first to Spring Framework Docs to understand what is #Bean and its parent #Configuration: https://docs.spring.io/spring/docs/5.1.2.RELEASE/spring-framework-reference/core.html#beans-java
Your #ServiceActivator for the FileWritingMessageHandler is really correct (when you remove that beaObject.load()). What you just need is to declare one more #ServiceActivator for calling your beaObject.load() at runtime when message appears in the fileOutputChannel:
#ServiceActivator(inputChannel= "fileOutputChannel")
public void loadFileIntoDb(File payload) {
this.beaObject.load(payload);
}
See https://docs.spring.io/spring-integration/docs/5.1.1.BUILD-SNAPSHOT/reference/html/configuration.html#annotations for more info.

Related

Mockito how can simule record doesn't exist

.exception.INSSTaxNotFoundException: INSS Tax not found with ID 1
Could someone help me?
I want mokite "inssTaxService.findById", I don't know how do.
I get this error: INSSTaxNotFoundException: INSS Tax not found with ID 1.
But I could like found the record and go on.
Can I do that in Service or Not?
#Test
void whenINSSTaxIdInformedThenReturnThisINSSTax() throws INSSTaxNotFoundException {
INSSTaxDTO expectedSavedInssTaxDTO = INSSTaxBuilder.builder().build().toINSSTaxDTO();
INSSTax expectedSavedInssTax = inssTaxMapper.toModel(expectedSavedInssTaxDTO);
when(inssTaxService.findById(expectedSavedInssTaxDTO.getId())).
thenReturn(expectedSavedInssTaxDTO);
assertEquals(expectedSavedInssTax.getId(), expectedSavedInssTaxDTO.getId());
assertEquals(expectedSavedInssTax.getDescription(), expectedSavedInssTaxDTO.getDescription());
assertEquals(expectedSavedInssTax.getSocialSecurityRatePercent(), expectedSavedInssTaxDTO.getSocialSecurityRatePercent());
}
What you might be missing is actually injecting the mock of inssTaxService inside your class which you are testing,
Your code would be something like this. Considering its a pure java code(not spring boot etc, you can change the code accordingly in that case).
Mock the service(Which i feel you have done else Mockito would have thrown and error)
InssTaxService mockedInssTaxService = Mockito.mock(InssTaxService.class);
//other impl on this mock for this e.g
when(mockedInssTaxService.findById(expectedSavedInssTaxDTO.getId())).
thenReturn(expectedSavedInssTaxDTO);
Inject the mocked object to the ClassToTest.
ClassToTest classToTest = new ClassToTest(mockedInssTaxService);
If you are using spring boot test you can use #MockBean or #Mock and #InjectMocks instead of new keyword

Spring Integration: Switch routing dynamically

A spring integration based converter consumes the messages from one system, checks, converts and sends it to the other one.
Should the target system be down, we stop the inbound adapters, but would also like to persist locally or forward the currently "in-flight" converted messages. For that would simply like to reroute the messages from the normal output channel to some "backup"-channel dynamically.
In the docs I have found only the option to route the messages based on their headers ( so on some step before in flow I would have to add those dynamically once the targer system is not availbale), or based on the payload type, which is not really my case. The case with adding dynamically some header, and then filtering it out down the pipe, or during de-/serializing still seems not the best approach for me. I would like rather to be able to turn a switch(on some internal Event) that would then reroute those "in-flight" messages to the "backup"-channel.
What would be a best SI approach to achive this? Thanks!
The router could not only be based on the the payload type or some header. You really can have a general POJO method invocation to return a channel, its name or some routing key which is mapped. That POJO method indeed can check some internal system state and produce this or that routing key.
So, you may have something like this in the router configuration:
.route(myRouter())
where your myRouter is something like this:
#Bean
MyRouter myRouter() {
return;
}
and its internal code might be like this:
public class MyRouter {
#Autowired
private SystemState systemState;
String route(Object payload) {
return this.systemState.isActive() ? "successChannel" : "backupChannel";
}
}
The same can be achieved a simple lambda definition:
.<Object, Boolean>route(p -> systemState().isActive(),
m -> m.channelMapping(true, "sucessChannel")
.channelMapping(false, "backupChannel"))
Also...
private final AtomicBoolean switcher = new AtomicBoolean();
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(() -> "foo", e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.route(s -> switcher.get() ? "foo" : "bar")
.get();
}

GlobalChannelInterceptor pass array of patterns

I am Spring Integration 4.3.13 and trying to pass patterns when configuring #GlobalChannelInterceptor
Here is the example
#Configuration
public class IntegrationConfig{
#Bean
#GlobalChannelInterceptor(patterns = "${spring.channel.interceptor.patterns:*}")
public ChannelInterceptor channelInterceptor(){
return new ChannelInterceptorImpl();
}
}
properties file has following values:
spring.channel.interceptor.patterns=*intchannel, *event
I am using direct channels with names that end with these two string
springintchannel
registrationevent
With the above config, both the channels should have interceptor configured but it is not getting configured.
The comma-separate value isn't support there currently.
I agree that we need to fix it, so feel free to raise a JIRA on the matter and we will file a solution from some other place.
Meanwhile you can do this as a workaround:
#Bean
public GlobalChannelInterceptorWrapper channelInterceptorWrapper(#Value("${spring.channel.interceptor.patterns:*}") String[] patterns) {
GlobalChannelInterceptorWrapper globalChannelInterceptorWrapper = new GlobalChannelInterceptorWrapper(channelInterceptor());
globalChannelInterceptorWrapper.setPatterns(patterns);
return globalChannelInterceptorWrapper;
}

Rename file in-place using Spring Integration FileWritingMessageHandler

I am attempting to write a simple test of renaming files in-place using a FileWritingMessageHandler, however I can't seem to figure out how to properly specify the target destination directory.
Since I am recursively scanning a directory tree I would ultimately like to simply read the parent path from the file payload and rename it using the FileNameGenerator, but that doesn't appear to work.
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
How do I properly determine the source file's location and use that in the handler?
Edit
Here is the channel adapter that scans for files. I had to use .setUseWatchService(true) to achieve recursive scanning.
#Bean
#InboundChannelAdapter(channel = "sourceFileChannel", poller = #Poller(fixedRate = "5000", maxMessagesPerPoll = "-1"))
public MessageSource<File> sourceFiles() {
CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
filters.addFilter(new SimplePatternFileListFilter(sourceFilenamePattern));
filters.addFilter(persistentFilter());
FileReadingMessageSource source = new FileReadingMessageSource();
source.setAutoCreateDirectory(true);
source.setDirectory(new File(sourceDirectory));
source.setFilter(filters);
source.setUseWatchService(true);
return source;
}
UPDATE
Artem helped me understand my mistake.
I was able to achieve the desired result by using the SpelExpressionParser as Artem outlined.
The key piece being:
new SpelExpressionParser().parseExpression("payload.parent")
Where "payload.parent" resolves to the file parent path properly.
#Bean
#ServiceActivator(inputChannel = "processingFileChannel")
public MessageHandler copyFileForProcessingOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new SpelExpressionParser().parseExpression("payload.parent"));
adapter.setDeleteSourceFiles(false);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
adapter.setFileNameGenerator(processingFileNameGenerator());
return adapter;
}
#Bean
public DefaultFileNameGenerator processingFileNameGenerator() {
DefaultFileNameGenerator defaultFileNameGenerator = new DefaultFileNameGenerator();
defaultFileNameGenerator.setExpression("'p_' + payload.name");
return defaultFileNameGenerator;
}
The 'payload.name' in the DefaultFileNameGenerator resolves properly, but 'payload.path' does not.
Well, I'm not sure what should be in your case, but for me that always returns the full, absolute path for source file, including the root directory to scan.
Since you have it there as a sourceDirectory, how about to use it in your processingFileNameGenerator to sever the root dir from the target path to use? For example if my root dir is /root and I get a file from the subdir /root/foo/my_file.txt, I could do payload.path.replaceFirst('/root', ''). So in the end I have just /foo/my_file.txt.
At least that is what I'm going to do in that JIRA to populate FileHeaders.FILENAME with the relative path from the provided directory to scan.
UPDATE
Oh! I see. No, that isn't going to work that way. See FileWritingMessageHandler ctors. The String once accepts static target directory. Root for our case. The code like new LiteralExpression("payload.path") isn't going to work a desired way, too. See LiteralExpression JavaDocs. It is just an Expression variant to always return the same static value. In your case it is payload.path.
If you are really going to evaluate against an incoming Message, you should use new SpeLexpressionParser().parseExpression("payload.path"). But as I said before, it returns the absolute path for any file in the sub-directories.

Sink component doesn't get the right data with kafka in spring cloud data flow

I am not a native English speaker but I try to express my question as clear as possible.
I encountered this problem which has confused me for two days and I still can't find the solution.
I have built a stream which will run in the Spring Could Data Flow in the Hadoop YARN.
The stream is composed of Http source,processor and file sink.
1.Http Source
The HTTP Source component has two output channels binding with two different destinations which are dest1 and dest2 defined in the application.properties.
spring.cloud.stream.bindings.output.destination=dest1
spring.cloud.stream.bindings.output2.destination=dest2
Below is the code snipet for HTTP source for your reference..
#Autowired
private EssSource channels; //EssSource is the interface for multiple output channels
##output channel 1:
#RequestMapping(path = "/file", method = POST, consumes = {"text/*", "application/json"})
#ResponseStatus(HttpStatus.ACCEPTED)
public void handleRequest(#RequestBody byte[] body, #RequestHeader(HttpHeaders.CONTENT_TYPE) Object contentType) {
logger.info("enter ... handleRequest1...");
channels.output().send(MessageBuilder.createMessage(body,
new MessageHeaders(Collections.singletonMap(MessageHeaders.CONTENT_TYPE, contentType))));
}
##output channel 2:
#RequestMapping(path = "/test", method = POST, consumes = {"text/*", "application/json"})
#ResponseStatus(HttpStatus.ACCEPTED)
public void handleRequest2(#RequestBody byte[] body, #RequestHeader(HttpHeaders.CONTENT_TYPE) Object contentType) {
logger.info("enter ... handleRequest2...");
channels.output2().send(MessageBuilder.createMessage(body,
new MessageHeaders(Collections.singletonMap(MessageHeaders.CONTENT_TYPE, contentType))));
}
2. Processor
The processor has two multiple input channels and two output channels binding with different destinations.
The destination binding is defined in application.properties in processor component project.
//input channel binding
spring.cloud.stream.bindings.input.destination=dest1
spring.cloud.stream.bindings.input2.destination=dest2
//output channel binding
spring.cloud.stream.bindings.output.destination=hdfsSink
spring.cloud.stream.bindings.output2.destination=fileSink
Below is the code snippet for Processor.
#Transformer(inputChannel = EssProcessor.INPUT, outputChannel = EssProcessor.OUTPUT)
public Object transform(Message<?> message) {
logger.info("enter ...transform...");
return "processed by transform1";;
}
#Transformer(inputChannel = EssProcessor.INPUT_2, outputChannel = EssProcessor.OUTPUT_2)
public Object transform2(Message<?> message) {
logger.info("enter ... transform2...");
return "processed by transform2";
}
3. The file sink component.
I use the official fil sink component from Spring.
maven://org.springframework.cloud.stream.app:file-sink-kafka:1.0.0.BUILD-SNAPSHOT
And I just add the destination binding in its applicaiton.properties file.
spring.cloud.stream.bindings.input.destination=fileSink
4.Finding:
The data flow I expected should like this:
Source.handleRequest() -->Processor.handleRequest()
Source.handleRequest2() -->Processor.handleRequest2() --> Sink.fileWritingMessageHandler();
Should only the string "processed by transform2" is saved to the file.
But after my testing, the data flow is actual like this:
Source.handleRequest() -->Processor.handleRequest() --> Sink.fileWritingMessageHandler();
Source.handleRequest2() -->Processor.handleRequest2() --> Sink.fileWritingMessageHandler();
Both the "processed by transform1" and "processed by transform2" string are saved to the file.
5.Question:
Although the destination for the output channel in Processor.handleRequest() binds to hdfsSink instead of fileSink,the data still flows to file Sink. I can't understand this and this is not what I want.
I only want the data from Processor.handleRequest2() flows to file sink instead of both.
If I don't do it right, could anyone tell me how to do it and what is the solution?
It has been confused me for 2 days.
Thanks you for your kindly help.
Alex
Is your stream definition something like this (where the '-2' versions are the ones with multiple channels) ?
http-source-2 | processor-2 | file-sink
Note that Spring Cloud Data Flow will override the destinations defined in applications.properties which is why, even if spring.cloud.stream.bindings.output.destination for the processor is set to hdfs-sink, it will actually match the input of file-sink.
The way destinations are configured from a stream definition is explained here (in the context of taps): http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-stream-tap-dsl
What you can do is to simply swap the meaning of channel 1 and 2 - use the side channel for hdfs. This is a bit brittle though - as the input/output channels of the Stream will be configured automatically and the other channels will be configured via application.properties - in this case it may be better to configure the side channel destinations via stream definition or at deployment time - see http://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_application_properties.
It seems to me that these could be just as well be 2 streams listening to separate endpoints, using regular components - given that data is supposed to be flowing side by side.

Resources