How to implement distributed lock around poller in Spring Integration using ZooKeeper - spring-integration

Spring Integration has ZooKeeper support as documented in https://docs.spring.io/spring-integration/reference/html/zookeeper.html
However this document is so vague.
It suggests adding below bean but does not give details on how to start/stop a poller when the node is granted leadership.
#Bean
public LeaderInitiatorFactoryBean leaderInitiator(CuratorFramework client) {
return new LeaderInitiatorFactoryBean()
.setClient(client)
.setPath("/siTest/")
.setRole("cluster");
}
Do we have any example on how to ensure below poller is run only once in a cluster at any time using zookeeper?
#Component
public class EventsPoller {
public void pullEvents() {
//pull events should be run by only one node in the cluster at any time
}
}

The LeaderInitiator emits an OnGrantedEvent and OnRevokedEvent, when it becomes leader and its leadership is revoked.
See https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#endpoint-roles and the next https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#leadership-event-handling for more info about those events handling and how it affects your components in the particular role.
Although I agree that Zookkeper chapter must have some link to that SmartLifecycleRoleController chapter. Feel free to raise a JIRA on the matter and contribution is welcome!
UPDATE
This is what I did in our test:
#RunWith(SpringRunner.class)
#DirtiesContext
public class LeaderInitiatorFactoryBeanTests extends ZookeeperTestSupport {
private static CuratorFramework client;
#Autowired
private PollableChannel stringsChannel;
#BeforeClass
public static void getClient() throws Exception {
client = createNewClient();
}
#AfterClass
public static void closeClient() {
if (client != null) {
client.close();
}
}
#Test
public void test() {
assertNotNull(this.stringsChannel.receive(10_000));
}
#Configuration
#EnableIntegration
public static class Config {
#Bean
public LeaderInitiatorFactoryBean leaderInitiator(CuratorFramework client) {
return new LeaderInitiatorFactoryBean()
.setClient(client)
.setPath("/siTest/")
.setRole("foo");
}
#Bean
public CuratorFramework client() {
return LeaderInitiatorFactoryBeanTests.client;
}
#Bean
#InboundChannelAdapter(channel = "stringsChannel", autoStartup = "false", poller = #Poller(fixedDelay = "100"))
#Role("foo")
public Supplier<String> inboundChannelAdapter() {
return () -> "foo";
}
#Bean
public PollableChannel stringsChannel() {
return new QueueChannel();
}
}
}
And I have in logs something like this:
2018-12-14 10:12:33,542 DEBUG [Curator-LeaderSelector-0] [org.springframework.integration.support.SmartLifecycleRoleController] - Starting [leaderInitiatorFactoryBeanTests.Config.inboundChannelAdapter.inboundChannelAdapter] in role foo
2018-12-14 10:12:33,578 DEBUG [Curator-LeaderSelector-0] [org.springframework.integration.support.SmartLifecycleRoleController] - Stopping [leaderInitiatorFactoryBeanTests.Config.inboundChannelAdapter.inboundChannelAdapter] in role foo

Related

Spring Integration - #InboundChannelAdapter polling

I am new to Spring Integration. We are creating our application using Spring Integration Annotations.
I have configured an #InboundChannelAdapter with poller fixed delay of 5 seconds. But the problem is as soon as I start my application on weblogic, the adapter starts polling and hits endpoint with practically no message.
We need to call a rest service and then trigger this adapter.
Is there a way to implement the same?
TIA!
Set the autoStartup property to false and use a control bus to start/stop it.
#SpringBootApplication
#IntegrationComponentScan
public class So59469573Application {
public static void main(String[] args) {
SpringApplication.run(So59469573Application.class, args);
}
}
#Component
class Integration {
#Autowired
private ApplicationContext context;
#InboundChannelAdapter(channel = "channel", autoStartup = "false",
poller = #Poller(fixedDelay = "5000"))
public String foo() {
return "foo";
}
#ServiceActivator(inputChannel = "channel")
public void handle(String in) {
System.out.println(in);
}
#ServiceActivator(inputChannel = "controlChannel")
#Bean
public ExpressionControlBusFactoryBean controlBus() {
return new ExpressionControlBusFactoryBean();
}
}
#MessagingGateway(defaultRequestChannel = "controlChannel")
interface Control {
void send(String control);
}
#RestController
class Rest {
#Autowired
Control control;
#PostMapping("/foo/{command}")
public void trigger(#PathVariable String command) {
if ("start".equals(command)) {
control.send("#'integration.foo.inboundChannelAdapter'.start()");
}
}
}

spring integration sftp error: Dispatcher has no subscribers

I'm trying to use the outbound gateway to download files from sftp server,
my config:
#IntegrationComponentScan
#EnableIntegration
#Configuration
public class FtpConfig {
#Bean(name = "myGateway")
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handlerLs() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "mget", "payload");
sftpOutboundGateway.setLocalDirectory(new File("/Users/xxx/Documents/"));
return sftpOutboundGateway;
}
#MessagingGateway
public interface OutboundGatewayOption {
#Gateway(requestChannel = "sftpChannel")
List<File> mget(String dir);
}
#Bean
public MessageChannel sftpChannel() {
return new DirectChannel();
}
}
and the execute bean:
#Service
public class DownloadService implements InitializingBean{
#Autowired
FtpConfig.OutboundGatewayOption gatewayOption;
#Override
public void afterPropertiesSet() throws Exception {
List<File> files = gatewayOption.mget("/sftp/server/path");
}
}
and I got this exception:org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'application.sftpChannel'.;
Qestion :how can I add the 'subscribers' ?
You can't perform messaging operator in the afterPropertiesSet(). That's too early : some beans might not be initialized yet. And that exception confirms the problem.
You have to implement SmartLifecicle instead and do the same in the start().

spring integration : solutions/tips on connect multiple sftp server?

My spring batch project needs to download files from multiple sftp servers.
the sftp host/port/filePath is config in application.properties file. I consider using the spring integration 'sftp out-bound gateway' to connect these servers and download files. but Im don't how to do this kind of configuration(I'm using java config, ) and make it work? i guess I need some way to define multiple session factory according to the number of sftp server info config in application.properties file.
properties file:
sftp.host=host1,host2
sftp.user=user1,user2
sftp.pwd=pwd1,pwd2
config class:
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory1() {
...
}
#Bean(name = "myGateway1")
#ServiceActivator(inputChannel = "sftpChannel1")
public MessageHandler handler1() {
...
}
#MessagingGateway
public interface DownloadGateway1 {
#Gateway(requestChannel = "sftpChannel1")
List<File> start(String dir);
}
#Bean(name="sftpChannel1")
public MessageChannel sftpChannel1() {
return new DirectChannel();
}
Right, the server is specified in the session factory, not the gateway. The framework does provide a delegating session factory, allowing it to be selected from one of the configured factories for each message sent to the gateway. See Delegating Session Factory.
EDIT
Here's an example:
#SpringBootApplication
public class So46721822Application {
public static void main(String[] args) {
SpringApplication.run(So46721822Application.class, args);
}
#Value("${sftp.name}")
private String[] names;
#Value("${sftp.host}")
private String[] hosts;
#Value("${sftp.user}")
private String[] users;
#Value("${sftp.pwd}")
private String[] pwds;
#Autowired
private DelegatingSessionFactory<?> sessionFactory;
#Autowired
private SftpGateway gateway;
#Bean
public ApplicationRunner runner() {
return args -> {
try {
this.sessionFactory.setThreadKey("one"); // use factory "one"
this.gateway.send(new File("/tmp/f.txt"));
}
finally {
this.sessionFactory.clearThreadKey();
}
};
}
#Bean
public DelegatingSessionFactory<LsEntry> sessionFactory() {
Map<Object, SessionFactory<LsEntry>> factories = new LinkedHashMap<>();
for (int i = 0; i < this.names.length; i++) {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory();
factory.setHost(this.hosts[i]);
factory.setUser(this.users[i]);
factory.setPassword(this.pwds[i]);
factories.put(this.names[i], factory);
}
// use the first SF as the default
return new DelegatingSessionFactory<LsEntry>(factories, factories.values().iterator().next());
}
#ServiceActivator(inputChannel = "toSftp")
#Bean
public SftpMessageHandler handler() {
SftpMessageHandler handler = new SftpMessageHandler(sessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression("foo"));
return handler;
}
#MessagingGateway(defaultRequestChannel = "toSftp")
public interface SftpGateway {
void send(File file);
}
}
with properties...
sftp.name=one,two
sftp.host=host1,host2
sftp.user=user1,user2
sftp.pwd=pwd1,pwd2

onApplicationEvent() is never invoked on DelayHandler

I'm using Spring Boot and Spring Integration Java DSL in my #Configuration class. One of the flows is using DelayHandler with MessageStore, by means of .delay(String groupId, String expression, Consumer endpointConfigurer):
#Bean
public IntegrationFlow errorFlow() {
return IntegrationFlows.from(errorChannel())
...
.delay(...)
...
.get();
}
I was hoping to utilize the reschedulePersistedMessages() functionality of DelayHandler, but I found out the onApplicationEvent(ContextRefreshedEvent event) which invokes it is actually never invoked (?)
I'm not sure, but I suspect this is due to the fact DelayHandler is not registered as a Bean, so registerListeners() in AbstractApplicationContext is not able to automatically register DelayHandler (and registration of non-bean listeners via ApplicationEventMulticaster.addApplicationListener(ApplicationListener listener) is not done for DelayHandler.
Currently I'm using a rather ugly workaround of registering my own listener Bean into which I inject the integration flow Bean, and then invoking the onApplicationEvent() manually after locating the DelayHandler:
#Override
public void onApplicationEvent(ContextRefreshedEvent event) {
Set<Object> integrationComponents = errorFlow.getIntegrationComponents();
for (Object component : integrationComponents) {
if (component instanceof DelayerEndpointSpec) {
Tuple2<ConsumerEndpointFactoryBean, DelayHandler> tuple2 = ((DelayerEndpointSpec) component).get();
tuple2.getT2().onApplicationEvent(event);
return;
}
}
}
Well, yes. This test-case confirm the issue:
#ContextConfiguration
#RunWith(SpringJUnit4ClassRunner.class)
#DirtiesContext
public class DelayerTests {
private static MessageGroupStore messageGroupStore = new SimpleMessageStore();
private static String GROUP_ID = "testGroup";
#BeforeClass
public static void setup() {
messageGroupStore.addMessageToGroup(GROUP_ID, new GenericMessage<>("foo"));
}
#Autowired
private PollableChannel results;
#Test
public void testDelayRescheduling() {
Message<?> receive = this.results.receive(10000);
assertNotNull(receive);
assertEquals("foo", receive.getPayload());
assertEquals(1, messageGroupStore.getMessageGroupCount());
assertEquals(0, messageGroupStore.getMessageCountForAllMessageGroups());
}
#Configuration
#EnableIntegration
public static class ContextConfiguration {
#Bean
public IntegrationFlow delayFlow() {
return flow ->
flow.delay(GROUP_ID, (String) null,
e -> e.messageStore(messageGroupStore)
.id("delayer"))
.channel(c -> c.queue("results"));
}
}
}
Here we go: https://github.com/spring-projects/spring-integration-java-dsl/issues/59.
As a workaround we can do this in our #Configuration:
#Autowired
private ApplicationEventMulticaster multicaster;
#PostConstruct
public void setup() {
this.multicaster.addApplicationListenerBean("delayer.handler");
}
Pay attention to the beanName to register. This is exactly that .id("delayer") from our flow definition plus the .handler suffix for the DelayHandler bean definition.

Spring Integration Kafka Manual Acknowledgment

I'm having issues using manual acknowledgements with the KafkaTopicOffsetManager. When acknowledge() is called, the topic begins to get spammed repeatedly. Kafka has log.cleaner.enable set to true and the topic is using cleanup.policy=compact. Thanks for any help.
Config:
#Bean
public ZookeeperConfiguration zookeeperConfiguration() {
ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration(kafkaConfig.getZookeeperAddress());
zookeeperConfiguration.setClientId("clientId");
return zookeeperConfiguration;
}
#Bean
public ConnectionFactory connectionFactory() {
return new DefaultConnectionFactory(zookeeperConfiguration());
}
#Bean
public TestMessageHandler messageListener() {
return new TestMessageHandler();
}
#Bean
public OffsetManager offsetManager() {
ZookeeperConnect zookeeperConnect = new ZookeeperConnect(kafkaConfig.getZookeeperAddress());
OffsetManager offsetManager = new KafkaTopicOffsetManager(zookeeperConnect, kafkaConfig.getTopic() + "_OFFSET");
return offsetManager;
}
#Bean
public KafkaMessageListenerContainer kafkaMessageListenerContainer() {
KafkaMessageListenerContainer kafkaMessageListenerContainer = new KafkaMessageListenerContainer(connectionFactory(), kafkaConfig.getTopic());
kafkaMessageListenerContainer.setMessageListener(messageListener());
kafkaMessageListenerContainer.setOffsetManager(offsetManager());
return kafkaMessageListenerContainer;
}
Listener:
public class TestMessageHandler implements AcknowledgingMessageListener {
private static final Logger logger = LoggerFactory.getLogger(TestMessageHandler.class);
#Override
public void onMessage(KafkaMessage message, Acknowledgment acknowledgment) {
logger.info(message.toString());
acknowledgment.acknowledge();
}
}
The KafkaTopicOffsetManager needs its own topic to maintain the offset of the actual topic being consumed.
If you don't want to deal with decoding the message payload yourself (its painful in my opinion), extend listener from abstract class AbstractDecodingAcknowledgingMessageListener and provide org.springframework.integration.kafka.serializer.common.StringDecoder as the decoder.
public class TestMessageHandlerDecoding extends AbstractDecodingAcknowledgingMessageListener {
public TestMessageHandlerDecoding(Decoder keyDecoder, Decoder payloadDecoder) {
super(keyDecoder, payloadDecoder);
}
#Override
public void doOnMessage(Object key, Object payload, KafkaMessageMetadata metadata, Acknowledgment acknowledgment) {
LOGGER.info("payload={}",payload);
}

Resources