spring-integration amqp outbound adapter race condition? - spring-integration

We've got a rather complicated spring-integration-amqp use case in one of our production applications and we've been seeing some "org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers" exceptions on startup. After the initial errors on startup, we don't see those exceptions anymore from the same components. This is seeming like some kind of startup race condition on components that depend on AMQP outbound adapters and that end up using them early in the lifecycle.
I can reproduce this by calling a gateway that sends to a channel wired to an outbound adapter in a PostConstruct method.
config:
package gadams;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.annotation.IntegrationComponentScan;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.dsl.amqp.Amqp;
import org.springframework.integration.dsl.channel.MessageChannels;
import org.springframework.messaging.MessageChannel;
#SpringBootApplication
#IntegrationComponentScan
public class RabbitRace {
public static void main(String[] args) {
SpringApplication.run(RabbitRace.class, args);
}
#Bean(name = "HelloOut")
public MessageChannel channelHelloOut() {
return MessageChannels.direct().get();
}
#Bean
public Queue queueHello() {
return new Queue("hello.q");
}
#Bean(name = "helloOutFlow")
public IntegrationFlow flowHelloOutToRabbit(RabbitTemplate rabbitTemplate) {
return IntegrationFlows.from("HelloOut").handle(Amqp.outboundAdapter(rabbitTemplate).routingKey("hello.q"))
.get();
}
}
gateway:
package gadams;
import org.springframework.integration.annotation.Gateway;
import org.springframework.integration.annotation.MessagingGateway;
#MessagingGateway
public interface HelloGateway {
#Gateway(requestChannel = "HelloOut")
void sendMessage(String message);
}
component:
package gadams;
import javax.annotation.PostConstruct;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.DependsOn;
import org.springframework.stereotype.Component;
#Component
#DependsOn("helloOutFlow")
public class HelloPublisher {
#Autowired
private HelloGateway helloGateway;
#PostConstruct
public void postConstruct() {
helloGateway.sendMessage("hello");
}
}
In my production use case, we have a component with a PostConstruct method where we're using a TaskScheduler to schedule a bunch of components with some that depend on AMQP outbound adapters, and some of those end up executing immediately. I've tried putting bean names on the IntegrationFlows that involve an outbound adapter and using #DependsOn on the beans that use the gateways and/or the gateway itself, but that doesn't get rid of the errors on startup.

That everything called Lifecycle. Any Spring Integration endpoints start listen for or produce messages only when their start() is performed.
Typically for standard default autoStartup = true it is done in the ApplicationContext.finishRefresh(); as a
// Propagate refresh to lifecycle processor first.
getLifecycleProcessor().onRefresh();
To start producing messages to the channel from the #PostConstruct (afterPropertiesSet()) is really very early, because it is does far away from the finishRefresh().
You really should reconsider your producing logic and that implementation into SmartLifecycle.start() phase.
See more info in the Reference Manual.

Related

Mockito.when method doesn't manage my service call

i'm trying to make Unit Test testing a simple GET controller method apply MockMvc.perform method but when the controller receive a request the method Mockito.when seems to doesn't manage the method call of MenuService and the test throw an exception. The Exception says menuServiceMock is null
I'm working with Mockito MockMvc JUnit
import org.junit.runner.RunWith;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.web.WebAppConfiguration;
import org.springframework.test.web.servlet.MockMvc;
import edu.AG.LandingPageSanpietro.domain.Menu;
import edu.AG.LandingPageSanpietro.service.MenuService;
import java.util.Arrays;
import static org.hamcrest.Matchers.*;
import static org.hamcrest.Matchers.is;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.model;
#RunWith(SpringJUnit4ClassRunner.class)
#WebAppConfiguration
class MenuControllerTest {
private MockMvc mockMvc;
#Autowired
private MenuService menuServiceMock;
#Test
public void testHomeController1() throws Exception {
Menu first=new Menu("titolo1","descrizione1","filename1");
Menu second=new Menu("titolo2","descrizione2","filename2");
Mockito.when(menuServiceMock.getMenus()).thenReturn(Arrays.asList(first, second));
mockMvc.perform(get("/manageMenu"))
.andExpect(status().isOk())
.andExpect(view().name("manageMenu"))
.andExpect(forwardedUrl("/src/main/resources/tamplates/manageMenu.html"))
.andExpect(model().attribute("menus", hasSize(2)));
}
My Controller
#GetMapping("/manageMenu")
public String chiamataGestisciMenu(Model model) {
model.addAttribute("menus", menuService.getMenus());
return "manageMenu";
}
The error
java.lang.NullPointerException: Cannot invoke "edu.AG.LandingPageSanpietro.service.MenuService.getMenus()" because "this.menuServiceMock" is null
at edu.AG.LandingPageSanpietro.controller.MenuControllerTest.testHomeController1(MenuControllerTest.java:44)
I can't understand why when() method doesn't manage my menuServiceMock.getMenus() request for returning the specified list.
Use the annotation #Mock from mockito instead of #Autowired.
Seems like you have not initialized MockMvc. Try Autowiring it or initialize it in #Before method:
#RunWith(SpringJUnit4ClassRunner.class)
#WebMvcTest(controllers = ControllerToBeTested.class)
class MenuControllerTest {
#Autowired
private MockMvc mockMvc;
...
or you can even initialize it in #Before lik this:
#Before
public void setup() {
this.mockMvc = MockMvcBuilders.standaloneSetup(new ControllerToBeTested()).build();
}

How to manually stop or start the #InboundChannelAdapter by ourself?

Helo Everyone,
i am working on spring cloud data flow
i created the application and it is working fine but we have one requirement like it will be able to start/stop whenever we need..
if i keep autoStartup="false" it is not starting at first,but i dont know how to start or stop after that.
Most of the places having only xml config.
Tried some code with some online articles but it didn't work.
can anyone know how to resolve this and if is there any example means it will be really helpful.
Actually if autoStartup is false and using CommandLineRunner i can able to start the service.but the same thing if i tried with rest endpointit is throwing error.
Below is the code snippet.
below snippet is from my code.
package com.javatechie.service;
import com.javatechie.impl.ProductBuilder;
import com.javatechie.TbeSource;
import com.javatechie.model.Product;
import com.javatechie.stopping.StopPollingAdvice;
import lombok.SneakyThrows;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Scope;
import org.springframework.integration.annotation.EndpointId;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.annotation.Poller;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.config.ExpressionControlBusFactoryBean;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.handler.LoggingHandler;
import org.springframework.integration.scheduling.PollerMetadata;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.support.GenericMessage;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.scheduling.support.PeriodicTrigger;
import org.springframework.stereotype.Component;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;
import java.util.function.Supplier;
#Component
#EnableIntegration
#RestController
public class ProcessingCl {
#Autowired
ApplicationContext ac;
#Bean
#Scope("singleton")
private ProductBuilder dataAccess() {
return new ProductBuilder();
}
#Bean
#EndpointId("inboundtest")
#InboundChannelAdapter(channel = TbeSource.PR1, poller = #Poller(fixedDelay = "100", errorChannel = "errorchannel"),autoStartup = "false")
public Supplier<Product> getProductSource(ProductBuilder dataAccess)
{
return ()->dataAccesss.getNext();
}
#Bean
MessageChannel controlChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "controlChannel")
ExpressionControlBusFactoryBean controlBus() {
ExpressionControlBusFactoryBean expressionControlBusFactoryBean = new ExpressionControlBusFactoryBean();
return expressionControlBusFactoryBean;
}
#Bean
CommandLineRunner commandLineRunner(#Qualifier("controlChannel") MessageChannel controlChannel) {
return (String[] args) -> {
System.out.println("Starting incoming file adapter: ");
boolean sent = controlChannel.send(new GenericMessage<>("#inboundtest.start()"));
System.out.println("Sent control message successfully? " + sent);
while (System.in.available() == 0) {
Thread.sleep(50);
}
};
}
#GetMapping("/stop11")
void test() {
controlChannel().send(new GenericMessage<>("#inboundtest.stop()"));
System.out.println("it is in call method to stop");
}
#GetMapping("/start11")
void test1() {
controlChannel().send(new GenericMessage<>("#inboundtest.start()"));
System.out.println("it is in call method to start");
}
}
Following is the error message.
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for channel 'unknown.channel.name'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers, failedMessage=GenericMessage [payload=#inboundtest.start(), headers={id=82c74865-2f68-c192-7d48-501bf3b28e02, timestamp=1608643036650}], failedMessage=GenericMessage [payload=#inboundtest.start(), headers={id=82c74865-2f68-c192-7d48-501bf3b28e02, timestamp=1608643036650}]] with root cause
org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
The control bus, #EndpointId and command as #inboundtest.start() is the way to go.
Only the problem that you don't show what is a subscriber for that TbeSource.PR1 channel.
That's probably where you are missing the point of Spring Integration: you start producing messages from that polling channel adapter, but there is no something like service activator having that TbeSource.PR1 channel as input.
I see you say it is Data Flow, but it is not clear from your description and configuration where is the binding for a TbeSource.PR1 destination. Do you have something like #EnableBinding and respective Source definition?

Spring reactive file integration

I am trying to use spring-integration-file to poll a directory and create a reactive stream from files placed in this directory. This is working for the most part, but when I place a file but have no subscriber in place I get an exception. To demonstrate the problem I have written a small demo application:
import org.reactivestreams.Publisher;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.MediaType;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.integration.dsl.Pollers;
import org.springframework.integration.file.dsl.Files;
import org.springframework.messaging.Message;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;
import java.io.File;
#SpringBootApplication
#RestController
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#Bean
public Publisher<Message<File>> reactiveSource() {
return IntegrationFlows
.from(Files.inboundAdapter(new File("."))
.patternFilter("*.csv"),
e -> e.poller(Pollers.fixedDelay(1000)))
.channel("processFileChannel")
.toReactivePublisher();
}
#GetMapping(value = "/files", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> files() {
return Flux.from(reactiveSource())
.map(message -> message.getPayload().getAbsolutePath());
}
}
So if I now do a curl to localhost:8080/files and then place a csv file in the root directory of the project everything is fine, I see the path of the file as response to my curl. But when I don't do a curl and then place a file in the root directory I get the following exception:
java.lang.IllegalStateException: The [bean 'reactiveSource.channel#0'; defined in: 'com.example.demo.DemoApplication';
from source: 'bean method reactiveSource'] doesn't have subscribers to accept messages
at org.springframework.util.Assert.state(Assert.java:97)
at org.springframework.integration.channel.FluxMessageChannel.doSend(FluxMessageChannel.java:61)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
... 38 more
I thought one of the attributes of reactive streams was that when there was no subscriber the stream would not start due to the stream being lazy. But apparently this is not the case. Could someone explain what I would need to do to have the stream not start if there is no subscriber?
If you use one of the latest version, then you can use a FluxMessageChannel channel instead of that DirectChannel for the "processFileChannel". This way a SourcePollingChannel adapter will becomes reactive and indeed the source is not going to be polled until a subscription happens to that FluxMessageChannel.
You then create a Flux in your files() API from this FluxMessageChannel - no need in the .toReactivePublisher().
See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#source-polling-channel-adapter
The point is that .toReactivePublisher() just makes an integration flow as a Publisher exactly at this point. Everything before this point is in regular, imperative way and works independently from the downstream logic.
UPDATE
Something like this:
#Bean
FluxMessageChannel filesChannel() {
return new FluxMessageChannel();
}
#Bean
public IntegrationFlow reactiveSource() {
return IntegrationFlows
.from(Files.inboundAdapter(new File("."))
.patternFilter("*.csv"),
e -> e.poller(Pollers.fixedDelay(1000)))
.channel(filesChannel())
.get();
}
#GetMapping(value = "/files", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> files() {
return Flux.from(filesChannel())
.map(message -> ((File) message.getPayload()).getAbsolutePath());
}

Micronaut: Unable to inject a bean inside Quartz job

I am trying to inject bean (Either data access GORM service or any other bean) in the Quartz Job implemented class, but it always shows null. Same beans (GORM or other beans) are able to inject without any issues in other classes.
can you please help me to retrieve any bean in this Job class.
My Quartz job
#Singleton
#Slf4j
class MyQuartzJob implements Job {
#Inject
MyHttpBean myHttpBean // unable to inject
#Inject
ApplicationContext appContext // unable to inject
#Inject
MyGORMService myGormService // unable to inject
}
#Singleton
#Slf4j
class MyHttpBean {
// business logic
}
code to invoke QuartzJob
#Singleton
#Context
#Slf4j
#CompileStatic
class MasterScheduler{
#PostConstruct
void init(){
// Quartz Job initialization code written here. This works fine.
}
}
my build.gradle
dependencyManagement {
imports {
mavenBom 'io.micronaut:micronaut-bom:1.3.2'
}
}
dependencies {
annotationProcessor("io.micronaut:micronaut-inject-java:1.3.2")
annotationProcessor("io.micronaut:micronaut-inject-groovy:1.3.2")
implementation("io.micronaut:micronaut-inject:1.3.2")
// https://mvnrepository.com/artifact/org.quartz-scheduler/quartz
compile group: 'org.quartz-scheduler', name: 'quartz', version: '2.3.0'
... other dependencies
}
Java version: 1.8
Note: I am using Micronaut scheduling capabilities, but I need distributed execution support & hence moving to Quartz ...
you need to define a JobFactory.
package io.micronaut.quartz;
import io.micronaut.context.BeanContext;
import org.quartz.Job;
import org.quartz.JobDetail;
import org.quartz.Scheduler;
import org.quartz.SchedulerException;
import org.quartz.spi.JobFactory;
import org.quartz.spi.TriggerFiredBundle;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.inject.Singleton;
#Singleton
public class MicronautJobFactory implements JobFactory {
private final Logger log = LoggerFactory.getLogger(getClass());
private final BeanContext beanContext;
public MicronautJobFactory(BeanContext beanContext) {
this.beanContext = beanContext;
}
#Override
public Job newJob(TriggerFiredBundle bundle, Scheduler scheduler) throws SchedulerException {
JobDetail jobDetail = bundle.getJobDetail();
Class<? extends Job> jobClass = jobDetail.getJobClass();
try {
if (log.isDebugEnabled()) {
log.debug(
"Producing instance of Job '" + jobDetail.getKey() +
"', class=" + jobClass.getName());
}
return beanContext.getBean(jobClass);
} catch (Exception e) {
SchedulerException se = new SchedulerException(
"Problem instantiating class '"
+ jobDetail.getJobClass().getName() + "'", e);
throw se;
}
}
}
afterward when you define your scheduler you can will then have to add your factory builder to the quartz scheduler when you define it. This replaces the default builder that comes with Quartz.
All your jobs going forward will have to be defined using #Prototype or #Singleton annotation. By Default Quartz uses newInstance() which circumvents object defined by Micronaut, but this will defer the object building to Micronaut when the job is declared.
scheduler.setJobFactory(jobFactory);

Spring Batch thread-safe Map job repository

the Spring Batch docs say of the Map-backed job repository:
Note that the in-memory repository is volatile and so does not allow restart between JVM instances. It also cannot guarantee that two job instances with the same parameters are launched simultaneously, and is not suitable for use in a multi-threaded Job, or a locally partitioned Step. So use the database version of the repository wherever you need those features.
I would like to use a Map job repository, and I do not care about restarting, prevention of concurrent job executions, etc. but I do care about being able to use multi-threading and local partitioning.
My batch application has some partitioned steps, and at first glance it seems to run just fine with a Map-backed job repository.
What is the reason it said to be not possible with MapJobRepositoryFactoryBean? Looking at the implementation of Map DAOs, they are using ConcurrentHashMap. Is this not thread-safe ?
I would advise you to follow the documentation, rather than relying on implementation details. Even if the maps are individually thread-safe, there might be race conditions in changes than involve more than one of these maps.
You can use an in-memory database very easily. Example
#Grapes([
#Grab('org.springframework:spring-jdbc:4.0.5.RELEASE'),
#Grab('com.h2database:h2:1.3.175'),
#Grab('org.springframework.batch:spring-batch-core:3.0.6.RELEASE'),
// must be passed with -cp, for whatever reason the GroovyClassLoader
// is not used for com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver
//#Grab('org.codehaus.jettison:jettison:1.2'),
])
import org.h2.jdbcx.JdbcDataSource
import org.springframework.batch.core.Job
import org.springframework.batch.core.JobParameters
import org.springframework.batch.core.Step
import org.springframework.batch.core.StepContribution
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory
import org.springframework.batch.core.launch.JobLauncher
import org.springframework.batch.core.scope.context.ChunkContext
import org.springframework.batch.core.step.tasklet.Tasklet
import org.springframework.batch.repeat.RepeatStatus
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.context.annotation.AnnotationConfigApplicationContext
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.core.io.ResourceLoader
import org.springframework.jdbc.datasource.init.DatabasePopulatorUtils
import org.springframework.jdbc.datasource.init.ResourceDatabasePopulator
import javax.annotation.PostConstruct
import javax.sql.DataSource
#Configuration
#EnableBatchProcessing
class AppConfig {
#Autowired
private JobBuilderFactory jobs
#Autowired
private StepBuilderFactory steps
#Bean
public Job job() {
return jobs.get("myJob").start(step1()).build()
}
#Bean
Step step1() {
this.steps.get('step1')
.tasklet(new MyTasklet())
.build()
}
#Bean
DataSource dataSource() {
new JdbcDataSource().with {
url = 'jdbc:h2:mem:temp_db;DB_CLOSE_DELAY=-1'
user = 'sa'
password = 'sa'
it
}
}
#Bean
BatchSchemaPopulator batchSchemaPopulator() {
new BatchSchemaPopulator()
}
}
class BatchSchemaPopulator {
#Autowired
ResourceLoader resourceLoader
#Autowired
DataSource dataSource
#PostConstruct
void init() {
def populator = new ResourceDatabasePopulator()
populator.addScript(
resourceLoader.getResource(
'classpath:/org/springframework/batch/core/schema-h2.sql'))
DatabasePopulatorUtils.execute populator, dataSource
}
}
class MyTasklet implements Tasklet {
#Override
RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
println 'TEST!'
}
}
def ctx = new AnnotationConfigApplicationContext(AppConfig)
def launcher = ctx.getBean(JobLauncher)
def jobExecution = launcher.run(ctx.getBean(Job), new JobParameters([:]))
println "Status is: ${jobExecution.status}"

Resources