Spring Integration file adapter tests failing on macOS - spring-integration

I have a spring integration project with multiple tests that runs perfectly fine on Windows and Linux machines. Now, I've just bought a MacBook and tried running the tests on it. All of my file adapter tests (15) are failing.
The below example test verifies that when a file is placed in the inboundOutDirectory directory, that it's properly processed.
Test class example:
#RunWith(SpringRunner.class)
#ContextConfiguration(
initializers = ConfigFileApplicationContextInitializer.class,
classes = {StructuralEnricherIntegrationFlow.class, StructuralEnricherIntegrationFlowTests.Config.class})
#SpringIntegrationTest
#DirtiesContext
public class StructuralEnricherIntegrationFlowTests {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Autowired
private File inboundOutDirectory;
#Autowired
private File inboundProcessedDirectory;
#Autowired
private File inboundFailedDirectory;
#Autowired
private PollableChannel testChannel;
#After
public void tearDown() throws IOException {
FileUtils.cleanDirectory(inboundOutDirectory);
FileUtils.cleanDirectory(inboundProcessedDirectory);
FileUtils.cleanDirectory(inboundFailedDirectory);
}
#Test
public void testStructuralEnricherFlowEmptyResponse() throws Exception {
// given
String docId = "xxx";
// when
MessageHandler mockMessageHandler = mockMessageHandler().handleNextAndReply(m -> "{}");
this.mockIntegrationContext.substituteMessageHandlerFor("structuralEnricherEndpoint", mockMessageHandler);
String fileName = TestingUtils.createFileAggregated(docId, 1, ".json", inboundOutDirectory,
"{}");
// then
Message<String> receive = (Message<String>) testChannel.receive(3000);
JSONAssert.assertEquals(String.format("{xxx"),
receive.getPayload(), JSONCompareMode.LENIENT);
assertTrue(FileUtils.getFile(new File(this.inboundProcessedDirectory, fileName)).exists());
}
#Configuration
#EnableIntegration
public static class Config {
#Autowired
private File inboundOutDirectory;
#Bean
public PollableChannel testChannel() {
return new QueueChannel();
}
#Bean
public IntegrationFlow processedDirectory(#Value("${pattern}") String pattern) {
return IntegrationFlows
.from(Files.inboundAdapter(inboundOutDirectory)
.regexFilter(pattern)
.useWatchService(true)
.watchEvents(FileReadingMessageSource.WatchEventType.CREATE),
e -> e.poller(Pollers.fixedDelay(100)))
.transform(Files.toStringTransformer())
.channel("testChannel")
.get();
}
}
}
It seems that the testChannel never receives a message as I'm getting a NullPointerException at receive.getPayload(). I've tried increasing the timeout but it didn't help.
System info:
spring-boot-starter-parent 2.3.4.RELEASE
MacBook Pro 13 (2020) with macOS Big Sur 11.1
openjdk version "1.8.0_275"
The same test works for example on ubuntu with java version 1.8.0_275.
I'm not sure what's the problem so hopefully, you can help me out.
UPDATE
So it seems that the problem lies in the useWatchService(true) in the Files.inboundAdapter. This is how the processing of the document is currently triggered:
return IntegrationFlows
.from(Files.inboundAdapter(inboundOutDirectory)
.regexFilter(intermediatePattern)
.useWatchService(true)
.watchEvents(FileReadingMessageSource.WatchEventType.CREATE,
FileReadingMessageSource.WatchEventType.DELETE),
e -> e.poller(Pollers.fixedDelay(1000)
.taskExecutor(Executors.newFixedThreadPool(2))
.maxMessagesPerPoll(5)))
I have debugged this flow a bit more and it appears that when .useWatchService(true), then the adapter will pick up the file with a delay (it takes about 10 seconds). Not sure why it works differently on macOS. When I change it to .useWatchService(false), it works instantly.
Here is the util method that creates the files:
public static String createFileAggregated(String id, Integer documentId, String extension, File tmpDir, String content) throws Exception {
String filename = String.format("%s%04d%s", id, documentId, extension);
FileUtils.write(new File(tmpDir, filename), (Optional.ofNullable(content).orElse(filename)), "UTF-8", false);
return filename;
}

Related

What is the wrong in my integration flow and tasklet

This is my flows LS-GET(SFTP outbound gateway: download files from a remote SFTP server.)
and MessagingGateway.
#MessagingGateway
public interface IntegratedRemoteFileProcessMessagingGateway {
#Gateway(requestChannel = "getFlows.input")
void getFlows(final String remoteDirectory);
#Gateway(requestChannel = "moveFlows.input")
void moveFlows(final String remoteDirectory);
}
#Bean
public QueueChannelSpec getOutputChannel() {
return MessageChannels.queue();
}
#Bean
public IntegrationFlow getFlows() {
return f -> f
.enrichHeaders(h -> h
.headerExpression("originalPayload", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_DIRECTORY, "payload.toString()"))
.log(LoggingHandler.Level.INFO, "eu.haee", "'Header originalPayload=' + headers[originalPayload]")
.handle(Sftp.outboundGateway(sessionFactory, Command.LS.getCommand(), "payload")
.autoCreateDirectory(false)
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.filter(new SftpSimplePatternFileListFilter("*.xml"))
.options(Option.NAME_ONLY, Option.RECURSIVE))
.split()
.log(LoggingHandler.Level.INFO, "eu.haee", "'LS Payload= ' + payload.toString()")
.enrichHeaders(h -> h
.headerExpression("originalRemoteFile", "payload.toString()")
.headerExpression(FileHeaders.REMOTE_FILE, "payload.toString()"))
.handle(Sftp.outboundGateway(sessionFactory, Command.GET.getCommand(), "headers['originalPayload'] + headers['file_remoteFile']")
.autoCreateLocalDirectory(false)
.charset("UTF-8")
.fileNameExpression("headers['file_remoteFile']")
.localDirectory(new File(flowsConfiguration.localDirectory()))
.localFilenameExpression(new FunctionExpression<Message<?>>(m -> {
IntegrationMessageHeaderAccessor accessor = new IntegrationMessageHeaderAccessor(m);
final String remoteFileName = (String) accessor.getHeader("file_remoteFile");
final int extensionIndex = remoteFileName.lastIndexOf('.');
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMddHHmmss.SSSSS", Locale.GERMAN);
return String.format("%s_MYC(%s-%d)%s", remoteFileName.substring(0, extensionIndex),
ZonedDateTime.of(LocalDateTime.now(), ZoneId.of("Europe/Berlin")).format(formatter),
(new SecureRandom()).nextInt(99999),
remoteFileName.substring(extensionIndex));
}))
.options(Option.PRESERVE_TIMESTAMP)
.remoteFileSeparator("/"))
.channel("getOutputChannel");
}
This is my spring-batch tasklet and Junit. MessagingGateway injected via tasklet constructor.
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
for (Endpoint endpoint : endpoints) {
final String remoteDirectory = endpoint.getEpConUri();
logger.info("ProcessRemoteFilesFlowsTasklet {} dealer at {} remote files process starting",
endpoint.getId().getDlrCd(), remoteDirectory);
flowsMessagingGateway.getFlows(remoteDirectory);
}
return RepeatStatus.FINISHED;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
controlChannel.send(new GenericMessage<>("#getPoller.start()"));
logger.info("GetPollerRemoteFilesFlowsTasklet poller starting...");
return RepeatStatus.FINISHED;
}
#Autowired
private IntegratedRemoteFileProcessMessagingGateway flowsMessagingGateway;
#Autowired
private EndpointRepository endpointRepository;
#Test
public void getFlows() {
flowsMessagingGateway.getFlows("/c07va00011/iris/import/");
Uninterruptibles.sleepUninterruptibly(60, TimeUnit.SECONDS);
}
When I execute getFlows test code. I met exception. but file downloaded to my local computer.
I have no idea. I've tried many variants but didn't get any progress.
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.getFlows.input'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=/c07va00011/iris/import/, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#2e64ae1a, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#2e64ae1a, id=bd393cb7-42d0-03b2-674d-40e3cf9211de, timestamp=1609844917799}]
...
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
... 145 common frames omitted
#EnableIntegration placed every spring-integration related configuration classes. #IntegrationComponentScan also placed my main flow configuration class (with string arrays of package names to scan).
If #EnableIntegration annotation located in multiple classes what happens?
Should I merge all spring-batch and spring-integration configuration classes into one?
Also, I've tested ControlBus(Send messages to the poller in the spring-batch tasklet) and got the same exception.
11:57:36.481 [main] ERROR o.s.batch.core.step.AbstractStep - Encountered an error executing step startGetPollerRemoteFilesStep in job integratedFilesProcessJob2
org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.controlChannel'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=#getPoller.start(), headers={id=539a27d0-9bce-062d-8664-53aae14b5680, timestamp=1609930656454}]
#Lazy, #DependsOn also not working. (#Lazy added to the ControlBus, #DependsOn added in the spring service class: Spring-batch jobs also manually starts/stops by rest API calls.)
#Autowired
public BatchFileServiceConfiguration(JobBuilderFactory jobBuilderFactory,
StepBuilderFactory stepBuilderFactory,
PropertyConfiguration propertyConfiguration,
#Qualifier("sourceBatchTransactionManager") PlatformTransactionManager sourceBatchTransactionManager,
#Qualifier("sourceBatchEntityManagerFactory") EntityManagerFactory sourceBatchEntityManagerFactory,
#Qualifier("processFileTaskExecutor") TaskExecutor processFileTaskExecutor,
BatchEndpointRepository batchEndpointRepository,
RemoteFileProcessMessagingGateway remoteFileProcessMessagingGateway,
#Lazy #Qualifier("controlChannel") MessageChannel controlChannel) {
this.jobBuilderFactory = jobBuilderFactory;
this.stepBuilderFactory = stepBuilderFactory;
this.propertyConfiguration = propertyConfiguration;
this.sourceBatchTransactionManager = sourceBatchTransactionManager;
this.sourceBatchEntityManagerFactory = sourceBatchEntityManagerFactory;
this.processFileTaskExecutor = processFileTaskExecutor;
this.batchEndpointRepository = batchEndpointRepository;
this.remoteFileProcessMessagingGateway = remoteFileProcessMessagingGateway;
this.controlChannel = controlChannel;
#Service
#DependsOn({"lsFlows", "getFlows", "moveFlows", "moveFailedFlows", "getPollableFlows"})
public class FileServiceImpl implements FileService {
These exceptions were never occurred in the spring-integration stand-alone applications.

Basic Example using Spring Integration to copy file to another directory raising exception

I am a beginner in Spring Integration. I wrote this code which is in spring boot and it is raising exception "Bean named 'messageSource' is expected to be of type 'org.springframework.context.MessageSource' but was actually of type 'org.springframework.integration.file.FileReadingMessageSource'"
Code:
#Configuration
/*#EnableIntegration annotation designates this class as a Spring Integration configuration.*/
#EnableIntegration
public class SIConfig {
#Bean
public MessageChannel channel() {
return new DirectChannel();
}
//bydefault name of method
#Bean
public MessageSource messageSource() {
FileReadingMessageSource ms= new FileReadingMessageSource();
ms.setDirectory(new File("C:\\Users\\payal\\Pictures"));
ms.setFilter(new SimplePatternFileListFilter("*.mp4"));
return ms;
}
#Bean
public MessageHandler handler() {
FileWritingMessageHandler handler= new FileWritingMessageHandler(new File("C:\\Users\\payal\\Documents\\batch7"));
handler.setFileExistsMode(FileExistsMode.IGNORE);
handler.setExpectReply(false);
return handler;
}
#Bean
public IntegrationFlow flow() {
return IntegrationFlows.from(messageSource(), configurer -> configurer.poller(Pollers.fixedDelay(10000)))
.channel(channel())
.handle(handler()).get();
}
}
Since using boot, versions are automatically managed
Uploaded code n GitHub too:
https://github.com/LearningNewTechnology/SpringIntegrationOne
Any help would be really appreciated.
Change the name of the bean to, e.g.
#Bean
public MessageSource myMessageSource() {
Spring framework (context) has another type of MessageSource and Spring Boot autoconfiguration creates a bean of that type with name messageSource so your bean is colliding with that.

Unable to set multiple expectation with MockWebServiceServer

I am using a MockWebServiceServer to test the REST APIs. I am passing the values to it using #Runwith(Parameterized.class).
#RunWith(Parameterized.class)
public class MyAPITest {
protected static MockWebServiceServer mockServer;
private Message message;
public MyAPITest(Message messageIn) {
this.message = messageIn;
}
#BeforeClass
public static void setup(){
mockServer = MockWebServiceServer.createServer(applicationContext);
}
#Test
public final void testMethod() throws Throwable {
Source reqPayload1 = new StringSource("...");
Source reqPayload2 = new StringSource("...");
Source resPayload1 = new StringSource("...");
Source resPayload2 = new StringSource("...");
mockServer.expect(RequestMatchers.payload(reqPayload1 )).andRespond(ResponseCreators.withPayload(resPayload1 ));
//Unable to add below line as it throws exception. Unable to set multiple expectation
//mockServer.expect(RequestMatchers.payload(reqPayload2 )).andRespond(ResponseCreators.withPayload(resPayload2 ));
myClass.onMessage(this.message);
mockServer.verify();
}
#Parameters
public static Collection<Object[]> getParameters() {
//Read input data from file
}
}
Code works fine when I've only 1 input.
But it throws exception when I've more than one input.
java.lang.IllegalStateException: Can not expect another connection, the test is already underway
at org.springframework.util.Assert.state(Assert.java:385)
at org.springframework.ws.test.client.MockWebServiceMessageSender.expectNewConnection(MockWebServiceMessageSender.java:64)
at org.springframework.ws.test.client.MockWebServiceServer.expect(MockWebServiceServer.java:162)
at com.rakuten.gep.newsletter.batch.ExacttargetMQJobTest.testOnMessage(ExacttargetMQJobTest.java:82)
I'm using Spring 3.2. I want to test my api with multiple inputs.

Spring Integration Cassandra persistence workflow

I try to realize the following workflow with Spring Integration:
1) Poll REST API
2) store the POJO in Cassandra cluster
It's my first try with Spring Integration, so I'm still a bit overwhelmed about the mass of information from the reference. After some research, I could make the following work.
1) Poll REST API
2) Transform mapped POJO JSON result into a string
3) save string into file
Here's the code:
#Configuration
public class ConsulIntegrationConfig {
#InboundChannelAdapter(value = "consulHttp", poller = #Poller(maxMessagesPerPoll = "1", fixedDelay = "1000"))
public String consulAgentPoller() {
return "";
}
#Bean
public MessageChannel consulHttp() {
return MessageChannels.direct("consulHttp").get();
}
#Bean
#ServiceActivator(inputChannel = "consulHttp")
MessageHandler consulAgentHandler() {
final HttpRequestExecutingMessageHandler handler =
new HttpRequestExecutingMessageHandler("http://localhost:8500/v1/agent/self");
handler.setExpectedResponseType(AgentSelfResult.class);
handler.setOutputChannelName("consulAgentSelfChannel");
LOG.info("Created bean'consulAgentHandler'");
return handler;
}
#Bean
public MessageChannel consulAgentSelfChannel() {
return MessageChannels.direct("consulAgentSelfChannel").get();
}
#Bean
public MessageChannel consulAgentSelfFileChannel() {
return MessageChannels.direct("consulAgentSelfFileChannel").get();
}
#Bean
#ServiceActivator(inputChannel = "consulAgentSelfFileChannel")
MessageHandler consulAgentFileHandler() {
final Expression directoryExpression = new SpelExpressionParser().parseExpression("'./'");
final FileWritingMessageHandler handler = new FileWritingMessageHandler(directoryExpression);
handler.setFileNameGenerator(message -> "../../agent_self.txt");
handler.setFileExistsMode(FileExistsMode.APPEND);
handler.setCharset("UTF-8");
handler.setExpectReply(false);
return handler;
}
}
#Component
public final class ConsulAgentTransformer {
#Transformer(inputChannel = "consulAgentSelfChannel", outputChannel = "consulAgentSelfFileChannel")
public String transform(final AgentSelfResult json) throws IOException {
final String result = new StringBuilder(json.toString()).append("\n").toString();
return result;
}
This works fine!
But now, instead of writing the object to a file, I want to store it in a Cassandra cluster with spring-data-cassandra. For that, I commented out the file handler in the config file, return the POJO in transformer and created the following, :
#MessagingGateway(name = "consulCassandraGateway", defaultRequestChannel = "consulAgentSelfFileChannel")
public interface CassandraStorageService {
#Gateway(requestChannel="consulAgentSelfFileChannel")
void store(AgentSelfResult agentSelfResult);
}
#Component
public final class CassandraStorageServiceImpl implements CassandraStorageService {
#Override
public void store(AgentSelfResult agentSelfResult) {
//use spring-data-cassandra repository to store
LOG.info("Received 'AgentSelfResult': {} in Cassandra cluster...");
LOG.info("Trying to store 'AgentSelfResult' in Cassandra cluster...");
}
}
But this seems to be a wrong approach, the service method is never triggered.
So my question is, what would be a correct approach for my usecase? Do I have to implement the MessageHandler interface in my service component, and use a #ServiceActivator in my config. Or is there something missing in my current "gateway-approach"?? Or maybe there is another solution, that I'm not able to see..
Like mentioned before, I'm new to SI, so this may be a stupid question...
Nevertheless, thanks a lot in advance!
It's not clear how you are wiring in your CassandraStorageService bean.
The Spring Integration Cassandra Extension Project has a message-handler implementation.
The Cassandra Sink in spring-cloud-stream-modules uses it with Java configuration so you can use that as an example.
So I finally made it work. All I needed to do was
#Component
public final class CassandraStorageServiceImpl implements CassandraStorageService {
#ServiceActivator(inputChannel="consulAgentSelfFileChannel")
#Override
public void store(AgentSelfResult agentSelfResult) {
//use spring-data-cassandra repository to store
LOG.info("Received 'AgentSelfResult': {}...");
LOG.info("Trying to store 'AgentSelfResult' in Cassandra cluster...");
}
}
The CassandraMessageHandler and the spring-cloud-streaming seemed to be a to big overhead to my use case, and I didn't really understand yet... And with this solution, I keep control over what happens in my spring component.

Spring integration with retry with ExceptionClassifierRetryPolicy

I am using int:request-handler-advice-chain with my service activator. It is working correctly with org.springframework.retry.policy.SimpleRetryPolicy however I would like to use org.springframework.retry.policy.ExceptionClassifierRetryPolicy to allow for a different number of retries based on the exception thrown by the service activator.
The problem I am having is that by the time the exception gets to the ExceptionClassifierRetryPolicy it is a
org.springframework.integration.MessageHandlingException
Can anyone advise on the best approach for get the cause (i.e my exception) from the MessageHandlingException made available to the ExceptionClassifierRetryPolicy?
Solution thanks to Artem's suggestion below:
Create a subclass of SubclassClassifier that returns the cause in the case of MessagingException
public class MessagingCauseExtractingSubclassClassifier extends SubclassClassifier<Throwable, RetryPolicy> {
private static final Logger LOG = LoggerFactory.getLogger(MessagingCauseExtractingSubclassClassifier.class);
public MessagingCauseExtractingSubclassClassifier(final Map<Class<? extends Throwable>, RetryPolicy> policyMap, final RetryPolicy retryPolicy) {
super(policyMap, retryPolicy);
}
#Override
public RetryPolicy classify(final Throwable throwable) {
Throwable t = throwable;
if (t instanceof MessagingException) {
t = t.getCause();
LOG.debug("Throwable is instanceof MessagingException so classifying cause type: {}", t.getClass());
}
return super.classify(t);
}
}
Then a new ExceptionClassifierRetryPolicy subclass that uses the new classifier and policyMap
public class MessasgeCauseExtractingExceptionClassifierRetryPolicy extends ExceptionClassifierRetryPolicy {
#Override
public void setPolicyMap(final Map<Class<? extends Throwable>, RetryPolicy> policyMap) {
final MessagingCauseExtractingSubclassClassifier classifier = new MessagingCauseExtractingSubclassClassifier(
policyMap, new NeverRetryPolicy());
setExceptionClassifier(classifier);
}
}
Currently this won't support retying on MessagingException but this is fine for our use case. Otherwise works perfectly.
The BinaryExceptionClassifier has traverseCauses option to analize the whole StackTrace until the proper condition.
Exactly this option is with one of SimpleRetryPolicy constructor:
public SimpleRetryPolicy(int maxAttempts, Map<Class<? extends Throwable>, Boolean> retryableExceptions,
boolean traverseCauses) {
Please, take a look if that variant is feasible for you.

Resources