Remote directory for sftp outbound gateway with DSL - spring-integration

I'm having an issue with the SFTP outbound gateway using DSL.
I want to use a outbound gateway to send a file, then continue my flow.
The problem is that I have an exception telling me:
IllegalArgumentException: 'remoteDirectoryExpression' is required
I saw that I can use a RemoteFileTemplate where I can set the sftp session factory plus the remote directory information, but the directory I wan't is defined in my flow by the code put in the header just before the launch of the batch.
#Bean
public IntegrationFlow orderServiceFlow() {
return f -> f
.handleWithAdapter(h -> h.httpGateway("myUrl")
.httpMethod(HttpMethod.GET)
.expectedResponseType(List.class)
)
.split()
.channel(batchLaunchChannel());
}
#Bean
public DirectChannel batchLaunchChannel() {
return MessageChannels.direct("batchLaunchChannel").get();
}
#Bean
public IntegrationFlow batchLaunchFlow() {
return IntegrationFlows.from(batchLaunchChannel())
.enrichHeaders(h -> h
.headerExpression("oiCode", "payload")
)
.transform((GenericTransformer<String, JobLaunchRequest>) message -> {
JobParameters jobParameters = new JobParametersBuilder()
.addDate("exec_date", new Date())
.addString("oiCode", message)
.toJobParameters();
return new JobLaunchRequest(orderServiceJob, jobParameters);
})
.handle(new JobLaunchingMessageHandler(jobLauncher))
.enrichHeaders(h -> h
.headerExpression("jobExecution", "payload")
)
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(Sftp.outboundGateway(sftpSessionFactory,
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.get();
}
I don't see how I can tell my outbound gateway which will be the directory depending what is in my header.

The Sftp.outboundGateway() has an overloaded version with the RemoteFileTemplate. So, you need to instantiate SftpRemoteFileTemplate bean and configure its:
/**
* Set the remote directory expression used to determine the remote directory to which
* files will be sent.
* #param remoteDirectoryExpression the remote directory expression.
*/
public void setRemoteDirectoryExpression(Expression remoteDirectoryExpression) {
This one can be like FunctionExpression:
setRemoteDirectoryExpression(m -> m.getHeaders().get("remoteDireHeader"))

Before I got an answer, I came with this solution but I'm not sure it's a good one.
// flow //
.handle((p, h) -> {
//Logic to retreive file...
return new File("");
})
.handle(
Sftp.outboundGateway(
remoteFileTemplate(new SpelExpressionParser().parseExpression("headers['oiCode']")),
AbstractRemoteFileOutboundGateway.Command.PUT,
"payload")
)
.handle(// next steps //)
.get();
public RemoteFileTemplate remoteFileTemplate(Expression directory) throws Exception {
RemoteFileTemplate template = new SftpRemoteFileTemplate(sftpSessionFactory);
template.setRemoteDirectoryExpression(directory);
template.setAutoCreateDirectory(true);
template.afterPropertiesSet();
return template;
}
But this provoke a warn because of exception thrown by the ExpresionUtils
java.lang.RuntimeException: No beanFactory

Related

Spring Integration read and process a file without polling

I'm currently trying to write and integration flow then reads a csv file and processes it in chunks (Calls API for enrichment) then writes in back out as a new csv. I currently have an example working perfectly except that it is polling a directory. What I would like to do is be able to pass the file-path and file-name to the integration flow in the headers and then just perform the operation on that one file.
Here is my code for the polling example that works great except for the polling.
#Bean
#SuppressWarnings("unchecked")
public IntegrationFlow getUIDsFromTTDandOutputToFile() {
Gson gson = new GsonBuilder().disableHtmlEscaping().create();
return IntegrationFlows
.from(Files.inboundAdapter(new File(inputFilePath))
.filter(getFileFilters())
.preventDuplicates(true)
.autoCreateDirectory(true),
c -> c
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1)
)
)
.log(Level.INFO, m -> "TTD UID 2.0 Integration Start" )
.split(Files.splitter())
.channel(c -> c.executor(Executors.newFixedThreadPool(7)))
.handle((p, h) -> new CSVUtils().csvColumnSelector((String) p, ttdColNum))
.channel("chunkingChannel")
.get();
}
#Bean
#ServiceActivator(inputChannel = "chunkingChannel")
public AggregatorFactoryBean chunker() {
log.info("Initializing Chunker");
AggregatorFactoryBean aggregator = new AggregatorFactoryBean();
aggregator.setReleaseStrategy(new MessageCountReleaseStrategy(batchSize));
aggregator.setExpireGroupsUponCompletion(true);
aggregator.setGroupTimeoutExpression(new ValueExpression<>(100L));
aggregator.setOutputChannelName("chunkingOutput");
aggregator.setProcessorBean(new DefaultAggregatingMessageGroupProcessor());
aggregator.setSendPartialResultOnExpiry(true);
aggregator.setCorrelationStrategy(new CorrelationStrategyIml());
return aggregator;
}
#Bean
public IntegrationFlow enrichFlow() {
return IntegrationFlows.from("chunkingOutput")
.handle((p, h) -> gson.toJson(new TradeDeskUIDRequestPayloadBean((Collection<String>) p)))
.enrichHeaders(eh -> eh.async(false)
.header("accept", "application/json")
.header("contentType", "application/json")
.header("Authorization", "Bearer [TOKEN]")
)
.log(Level.INFO, m -> "Sending request of size " + batchSize + " to: " + TTD_UID_IDENTITY_MAP)
.handle(Http.outboundGateway(TTD_UID_IDENTITY_MAP)
.requestFactory(
alliantPooledHttpConnection.get_httpComponentsClientHttpRequestFactory())
.httpMethod(HttpMethod.POST)
.expectedResponseType(TradeDeskUIDResponsePayloadBean.class)
.extractPayload(true)
)
.log(Level.INFO, m -> "Writing response to output file" )
.handle((p, h) -> ((TradeDeskUIDResponsePayloadBean) p).printMappedBodyAsCSV2())
.handle(Files.outboundAdapter(new File(outputFilePath))
.autoCreateDirectory(true)
.fileExistsMode(FileExistsMode.APPEND)
//.appendNewLine(true)
.fileNameGenerator(m -> m.getHeaders().getOrDefault("file_name", "outputFile") + "_out.csv")
)
.get();
}
public class CorrelationStrategyIml implements CorrelationStrategy {
#Override
public Object getCorrelationKey(Message<?> message) {
return message.getHeaders().getOrDefault("", 1);
}
}
#Component
public class CSVUtils {
#ServiceActivator
String csvColumnSelector(String inputStr, Integer colNum) {
return StringUtils.commaDelimitedListToStringArray(inputStr)[colNum];
}
}
private FileListFilter<File> getFileFilters(){
ChainFileListFilter<File> cflf = new ChainFileListFilter<>();
cflf.addFilter(new LastModifiedFileListFilter(30));
cflf.addFilter(new AcceptOnceFileListFilter<>());
cflf.addFilter(new SimplePatternFileListFilter(fileExtention));
return cflf;
}
If you know the file, then there is no reason in any special component from the framework. You just start your flow from a channel and send a message to it with File object as a payload. That message is going to be carried on to the slitter in your flow and everything is going to work OK.
If you really want to have a high-level API on the matter, you can expose a #MessagingGateway as a beginning of that flow and end-user is going to call your gateway method with desired file as an argument. The framework will create a message on your behalf and send it to the message channel in the flow for processing.
See more info in docs about gateways:
https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#integration-flow-as-gateway
And also a DSL definition starting from some explicit channel:
https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-channels

How to route using message headers in Spring Integration DSL Tcp

I have 2 server side services and I would like route messages to them using message headers, where remote clients put service identification into field type.
Is the code snippet, from server side config, the correct way? It throws cast exception indicating that route() see only payload, but not the message headers. Also all example in the Spring Integration manual shows only payload based decisioning.
#Bean
public IntegrationFlow serverFlow( // common flow for all my services, currently 2
TcpNetServerConnectionFactory serverConnectionFactory,
HeartbeatServer heartbeatServer,
FeedServer feedServer) {
return IntegrationFlows
.from(Tcp.inboundGateway(serverConnectionFactory))
.<Message<?>, String>route((m) -> m.getHeaders().get("type", String.class),
(routeSpec) -> routeSpec
.subFlowMapping("hearbeat", subflow -> subflow.handle(heartbeatServer::processRequest))
.subFlowMapping("feed", subflow -> subflow.handle(feedServer::consumeFeed)))
.get();
}
Client side config:
#Bean
public IntegrationFlow heartbeatClientFlow(
TcpNetClientConnectionFactory clientConnectionFactory,
HeartbeatClient heartbeatClient) {
return IntegrationFlows.from(heartbeatClient::send, e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.enrichHeaders(c -> c.header("type", "heartbeat"))
.log()
.handle(outboundGateway(clientConnectionFactory))
.handle(heartbeatClient::receive)
.get();
}
#Bean
public IntegrationFlow feedClientFlow(
TcpNetClientConnectionFactory clientConnectionFactory) {
return IntegrationFlows.from(FeedClient.MessageGateway.class)
.enrichHeaders(c -> c.header("type", "feed"))
.log()
.handle(outboundGateway(clientConnectionFactory))
.get();
}
And as usual here is the full demo project code, ClientConfig and ServerConfig.
There is no standard way to send headers over raw TCP. You need to encode them into the payload somehow (and extract them on the server side).
The framework provides a mechanism to do this for you, but it requires extra configuration.
See the documentation.
Specifically...
The MapJsonSerializer uses a Jackson ObjectMapper to convert between a Map and JSON. You can use this serializer in conjunction with a MessageConvertingTcpMessageMapper and a MapMessageConverter to transfer selected headers and the payload in JSON.
I'll try to find some time to create an example of how to use it.
But, of course, you can roll your own encoding/decoding.
EDIT
Here's an example configuration to use JSON to convey message headers over TCP...
#SpringBootApplication
public class TcpWithHeadersApplication {
public static void main(String[] args) {
SpringApplication.run(TcpWithHeadersApplication.class, args);
}
// Client side
public interface TcpExchanger {
public String exchange(String data, #Header("type") String type);
}
#Bean
public IntegrationFlow client(#Value("${tcp.port:1234}") int port) {
return IntegrationFlows.from(TcpExchanger.class)
.handle(Tcp.outboundGateway(Tcp.netClient("localhost", port)
.deserializer(jsonMapping())
.serializer(jsonMapping())
.mapper(mapper())))
.get();
}
// Server side
#Bean
public IntegrationFlow server(#Value("${tcp.port:1234}") int port) {
return IntegrationFlows.from(Tcp.inboundGateway(Tcp.netServer(port)
.deserializer(jsonMapping())
.serializer(jsonMapping())
.mapper(mapper())))
.log(Level.INFO, "exampleLogger", "'Received type header:' + headers['type']")
.route("headers['type']", r -> r
.subFlowMapping("upper",
subFlow -> subFlow.transform(String.class, p -> p.toUpperCase()))
.subFlowMapping("lower",
subFlow -> subFlow.transform(String.class, p -> p.toLowerCase())))
.get();
}
// Common
#Bean
public MessageConvertingTcpMessageMapper mapper() {
MapMessageConverter converter = new MapMessageConverter();
converter.setHeaderNames("type");
return new MessageConvertingTcpMessageMapper(converter);
}
#Bean
public MapJsonSerializer jsonMapping() {
return new MapJsonSerializer();
}
// Console
#Bean
#DependsOn("client")
public ApplicationRunner runner(TcpExchanger exchanger,
ConfigurableApplicationContext context) {
return args -> {
System.out.println("Enter some text; if it starts with a lower case character,\n"
+ "it will be uppercased by the server; otherwise it will be lowercased;\n"
+ "enter 'quit' to end");
Scanner scanner = new Scanner(System.in);
String request = scanner.nextLine();
while (!"quit".equals(request.toLowerCase())) {
if (StringUtils.hasText(request)) {
String result = exchanger.exchange(request,
Character.isLowerCase(request.charAt(0)) ? "upper" : "lower");
System.out.println(result);
}
request = scanner.nextLine();
}
scanner.close();
context.close();
};
}
}

Spring Integration: MessageSource doesn't honor errorChannel header

I've the following flow:
#Resource(name = S3_CLIENT_BEAN)
private MessageSource<InputStream> messageSource;
public IntegrationFlow fileStreamingFlow() {
return IntegrationFlows.from(s3Properties.getFileStreamingInputChannel())
.enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true))
.handle(String.class, (fileName, h) -> {
if (messageSource instanceof S3StreamingMessageSource) {
S3StreamingMessageSource s3StreamingMessageSource = (S3StreamingMessageSource) messageSource;
ChainFileListFilter<S3ObjectSummary> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilters(...);
s3StreamingMessageSource.setFilter(chainFileListFilter);
return s3StreamingMessageSource.receive();
}
return messageSource.receive();
}, spec -> spec
.requiresReply(false) // in case all messages got filtered out
)
.channel(s3Properties.getFileStreamingOutputChannel())
.get();
}
I found that if s3StreamingMessageSource.receive throws an exception, the error ends up in the error channel configured for the previous flow in the pipeline, not the S3_ERROR_CHANNEL that's configured for this flow. Not sure if it's related to this question.
The s3StreamingMessageSource.receive() is called from the SourcePollingChannelAdapter:
protected Message<?> receiveMessage() {
return this.source.receive();
}
This one called from the AbstractPollingEndpoint:
private boolean doPoll() {
message = this.receiveMessage();
...
this.handleMessage(message);
...
}
That handleMessage() does this:
this.messagingTemplate.send(getOutputChannel(), message);
So, that is definitely still far away from the mentioned .enrichHeaders(spec -> spec.header(ERROR_CHANNEL, S3_ERROR_CHANNEL, true)) downstream.
However you still can catch an exception in that S3_ERROR_CHANNEL. Pay attention to the second argument of the IntegrationFlows.from():
IntegrationFlows.from(s3Properties.getFileStreamingInputChannel(),
e -> e.poller(Pollers.fixedDelay(...)
.errorChannel(S3_ERROR_CHANNEL)))
Or, according your current you have somewhere a global poller, so configure an errorChannel there.

Configure error handling and retry for Http.outboundGateway spring dsl

I have a requirement where i need to make a rest call when it fails i have to retry 3 times and depending on the status code received i need perform different action, i couldn't find a proper spring integration dsl example. How to configure the error handler and retry
#Bean
public IntegrationFlow performCreate() {
return IntegrationFlows.from("createFlow")
.handle(Http.outboundGateway("http://localhost:8080/create")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class)
.requestFactory(simpleClientHttpRequestFactory())
.errorHandler(??)
)
.log(LoggingHandler.Level.DEBUG, "response", m -> m.getPayload())
.log(LoggingHandler.Level.DEBUG, "response", m -> m.getHeaders())
.get();
}
private SimpleClientHttpRequestFactory simpleClientHttpRequestFactory() {
SimpleClientHttpRequestFactory simpleClientHttpRequestFactory = new SimpleClientHttpRequestFactory();
simpleClientHttpRequestFactory.setReadTimeout(5000);
simpleClientHttpRequestFactory.setConnectTimeout(5000);
return simpleClientHttpRequestFactory;
}
The Java DSL .handle() has a second argument - Consumer<GenericEndpointSpec<?>> and that one can be configured with the:
/**
* Configure a list of {#link Advice} objects to be applied, in nested order, to the
* endpoint's handler. The advice objects are applied only to the handler.
* #param advice the advice chain.
* #return the endpoint spec.
*/
public S advice(Advice... advice) {
One of those advises is in the Spring Integration box - RequestHandlerRetryAdvice: https://docs.spring.io/spring-integration/docs/5.0.4.RELEASE/reference/html/messaging-endpoints-chapter.html#retry-advice
https://docs.spring.io/spring-integration/docs/5.0.4.RELEASE/reference/html/java-dsl.html#java-dsl-endpoints
.handle(Http.outboundGateway("http://localhost:8080/create")
.httpMethod(HttpMethod.GET)
.expectedResponseType(String.class)
.requestFactory(simpleClientHttpRequestFactory()),
e -> e.advice(retryAdvice())
...
#Bean
public RequestHandlerRetryAdvice retryAdvice() {
RequestHandlerRetryAdvice requestHandlerRetryAdvice = new RequestHandlerRetryAdvice();
requestHandlerRetryAdvice.setRecoveryCallback(errorMessageSendingRecoverer());
return requestHandlerRetryAdvice;
}
#Bean
public ErrorMessageSendingRecoverer errorMessageSendingRecoverer() {
return new ErrorMessageSendingRecoverer(recoveryChannel())
}
#Bean
public MessageChannel recoveryChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow handleRecovery() {
return IntegrationFlows.from("recoveryChannel")
.log(LoggingHandler.Level.ERROR, "error",
m -> m.getPayload())
.get();
}

JMS Outbound Gateway request destination - post process on success

I am using a JMS Outbound Gateway to send messages to a request queue and receive messages from a separate response queue. I would like to add functionality so that a call is made to a specific bean's method once a message has been successfully sent to the request queue.
I am using spring-integration 4.0.4 APIs and spring-integration-java-dsl 1.0.0 APIs for this and, I have so far been able to achieve the above functionality as follows:
#Configuration
#EnableIntegration
public class IntegrationConfig {
...
#Bean
public IntegrationFlow requestFlow() {
return IntegrationFlows
.from("request.ch")
.routeToRecipients(r ->
r.ignoreSendFailures(false)
.recipient("request.ch.1", "true")
.recipient("request.ch.2", "true"))
.get();
}
#Bean
public IntegrationFlow sendReceiveFlow() {
return IntegrationFlows
.from("request.ch.1")
.handle(Jms.outboundGateway(cachingConnectionFactory)
.receiveTimeout(45000)
.requestDestination("REQUEST_QUEUE")
.replyDestination("RESPONSE_QUEUE")
.correlationKey("JMSCorrelationID"), e -> e.requiresReply(true))
.channel("response.ch").get();
}
#Bean
public IntegrationFlow postSendFlow() {
return IntegrationFlows
.from("request.ch.2")
.handle("requestSentService", "fireRequestSuccessfullySentEvent")
.get();
}
...
}
Now, although the above configuration works, I have noticed that the only apparent reason request.ch.1 is called before request.ch.2 seems to be because of the channel names' alphabetical order and not because of the order in which they where added to the RecipientListRouter itself. Is this correct? Or am I missing something here?
* EDIT below shows solution using Aggregator between JMS Outbound/Inbound Adapters approach (without Messaging Gateway) *
Integration Config:
#Configuration
#EnableIntegration
public class IntegrationConfig {
...
#Bean
public IntegrationFlow reqFlow() {
return IntegrationFlows
.from("request.ch")
.enrichHeaders(e -> e.headerChannelsToString())
.enrichHeaders(e -> e.headerExpression(IntegrationMessageHeaderAccessor.CORRELATION_ID, "headers['" + MessageHeaders.REPLY_CHANNEL + "']"))
.routeToRecipients(r -> {
r.ignoreSendFailures(false);
r.recipient("jms.req.ch", "true");
r.recipient("jms.agg.ch", "true");
})
.get();
}
#Bean
public IntegrationFlow jmsReqFlow() {
return IntegrationFlows
.from("jms.req.ch")
.handle(Jms.outboundAdapter(cachingConnectionFactory)
.destination("TEST_REQUEST_CH")).get();
}
#Bean
public IntegrationFlow jmsPostReqFlow() {
return IntegrationFlows
.from("jms.req.ch")
.handle("postSendService", "postSendProcess")
.get();
}
#Bean
public IntegrationFlow jmsResFlow() {
return IntegrationFlows
.from(Jms.inboundAdapter(cachingConnectionFactory).destination(
"TEST_RESPONSE_CH"),
c -> c.poller(Pollers.fixedRate(1000).maxMessagesPerPoll(10)))
.channel("jms.agg.ch").get();
}
#Bean
public IntegrationFlow jmsAggFlow() {
return IntegrationFlows
.from("jms.agg.ch")
.aggregate(a -> {
a.outputProcessor(g -> {
List<Message<?>> l = new ArrayList<Message<?>>(g.getMessages());
Message<?> firstMessage = l.get(0);
Message<?> lastMessage = (l.size() > 1) ? l.get(l.size() - 1) : firstMessage;
Message<?> messageOut = MessageBuilder.fromMessage(lastMessage)
.setHeader(MessageHeaders.REPLY_CHANNEL, (String) firstMessage.getHeaders().getReplyChannel())
.build();
return messageOut;
});
a.releaseStrategy(g -> g.size() == 2);
a.groupTimeout(45000);
a.sendPartialResultOnExpiry(false);
a.discardChannel("jms.agg.timeout.ch");
}, null)
.channel("response.ch")
.get();
}
}
#Bean
public IntegrationFlow jmsAggTimeoutFlow() {
return IntegrationFlows
.from("jms.agg.timeout.ch")
.handle(Message.class, (m, h) -> new ErrorMessage(new MessageTimeoutException(m), h))
.channel("error.ch")
.get();
}
}
Cheers,
PM
H-m... Looks like. It is really a bug in the DslRecipientListRouter logic: https://github.com/spring-projects/spring-integration-java-dsl/issues/9
Will be fixed soon and released over a couple of days.
Thank you for pointing that out!
BTW. your logic isn't correct a bit: Even when we fix that RecipientListRouter, the second recipinet will receive the same request message only after JmsOutboundGateway will have received the reply, not just after request has been sent to the request-queue.
It is blocked request-reply process. And there is no a hook to get a point between reqeust and reply in the JmsOutboundGateway.
Is that OK for you?

Resources