Scenario: I have 3 spring cloud streaming apps
1'st: unmarshalls XML payload into JAXB object
2'nd: Converts JAXB payload into our domain POJO
3'rd: Validate domain object
I am trying to test the 3'rd app. I have included the 1'st and 2'nd applications as test dependencies. I have added:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
Now I have about 20 xml files with various validation scenario's. The first test works fine. I am able pick up the expected message of the channel using:
final Message<PaymentInstruction> mceMessage =
(Message<PaymentInstruction>) collector.forChannel(
validationBindings.mce()).take();
The 2'nd test that is run is where I have an issue. The test just sits at 'take'.
I have done some digging in the spring-integration-core-4.3.8.jar and have traced the issue to org.springframework.integration.dispatcher.AbstractDispatcher:
#Override
public synchronized boolean addHandler(MessageHandler handler) {
Assert.notNull(handler, "handler must not be null");
Assert.isTrue(this.handlers.size() < this.maxSubscribers, "Maximum subscribers exceeded");
boolean added = this.handlers.add(handler);
if (this.handlers.size() == 1) {
this.theOneHandler = handler;
}
else {
this.theOneHandler = null;
}
return added;
}
There is a handler that was added for the first test, so it assigns null to 'this.theOneHandler'
My options are:
Refactor the code in the other 2 projects so that I can do the unmarshalling and creating of my domain object without the need for the spring cloud app code
I can create an individual unit test class per test case, however I'd rather not go this route as the startup time for spring boot will be high that will be multiplied by the number of test cases
Do I have some missing configuration that would allow me to have these multiple handlers or am I breaking the way I want to use spring cloud streaming?
Environment:
Java 8 Update 131
org.springframework.cloud:spring-cloud-dependencies:Dalston.RELEASE
org.springframework.boot:spring-boot-dependencies:1.5.2.RELEASE
theOneHandler is a dispatching optimization used when a channel has only one subscriber.
It sounds like you are binding all your services to the same channel, which is not what you want.
Bottom line is you can't do what you are trying to do because each of the services are using the same channel name (presumably input).
You would need to load each service in a separate application context and wire them together with bridges.
EDIT
You could also test them using an aggregated application.
Related
I am trying to test a Spring Integration flow that starts off from a message-driven-channel-adapter configured as:
<int-jms:message-driven-channel-adapter id="myAdapter" ... />
My test goes like:
#SpringJUnitConfig(locations = {"my-app-context.xml"})
#SpringIntegrationTest(noAutoStartup = {"myAdapter"})
public class MyIntegrationFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Test
public void myTest() {
...
MessageSource<String> messageSource = () -> new GenericMessage<>("foo");
mockIntegrationContext.substituteMessageSourceFor("myAdapter", messageSource);
...
}
}
I am however getting the following error:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'myAdapter' is expected to be of type 'org.springframework.integration.endpoint.SourcePollingChannelAdapter' but was actually of type 'org.springframework.integration.jms.JmsMessageDrivenEndpoint'
How should one specify an alternate source for the channel adapter for testing using the MockIntegrationContext, or by some other method?
The Message Driver Channel Adapter is really not a Source Polling Channel Adapter. So, the substituteMessageSourceFor() is indeed cannot be used for that type of components, which, essentially is a MessageProducerSupport implementation, not a SourcePollingChannelAdapter for a MessageSource.
The difference exists because not all protocols provides a listener-like hooks to spawn some self-managed task to subscribe to. The good example is JDBC, which is only passive system expecting requests. Therefore a polling channel adapter with a JdbcPollingChannelAdapter (which is a MessageSource) implementation must be used to interact with DB in event-driven manner.
Other systems (like JMS in your case) provides some listener (or consumer) API for what we can spawn a while task (see MessageListenerContainer in spring-jms) and let its MessageProducerSupport to emit messages to the channel.
Therefore you need to distinguish for yourself with what type of component you interact before choosing a testing strategy.
Since there is no extra layer in case of message-driver channel adapter, but rather some specific, self-managed MessageProducerSupport impl, we not only provide a particular mocking API, but even don't require to know anything else, but just standard unit testing feature and a message channel this endpoint is producing in the configuration.
So, the solution for you is something like:
#SpringIntegrationTest(noAutoStartup = {"myAdapter"}) - that's fully correct in your code: we really have to stop the real channel adapter to not pollute our testing environment.
You just need to inject into your test class a MessageChannel that id="myAdapter" is producing to. In your test code you just build a Message and send it into this channel. No need to worry about a MockIntegrationContext at all.
I have a quarkus application with an async endpoint that creates an entity with default properties, starts a new thread within the request method and executes a long running job and then returns the entity as a response for the client to track.
#POST
#Transactional
public Response startJob(#NonNull JsonObject request) {
// create my entity
JobsRecord job = new JobsRecord();
// set default properties
job.setName(request.getString("name"));
// make persistent
jobsRepository.persist(job);
// start the long running job on a different thread
Executor.execute(() -> longRunning(job));
return Response.accepted().entity(job).build();
}
Additionally, the long running job will make updates to the entity as it runs and so it must also be transactional. However, the database entity just doesn't get updated.
These are the issues I am facing:
I get the following warnings:
ARJUNA012094: Commit of action id 0:ffffc0a80065:f2db:5ef4e1c7:0 invoked while multiple threads active within it.
ARJUNA012107: CheckedAction::check - atomic action 0:ffffc0a80065:f2db:5ef4e1c7:0 commiting with 2 threads active!
Seems like something that should be avoided.
I tried using #Transaction(value = TxType.REQUIRES_NEW) to no avail.
I tried using the API Approach instead of the #Transactional approach on longRunning as mentioned in the guide as follows:
#Inject UserTransaction transaction;
.
.
.
try {
transaction.begin();
jobsRecord.setStatus("Complete");
jobsRecord.setCompletedOn(new Timestamp(System.currentTimeMillis()));
transaction.commit();
} catch (Exception e) {
e.printStackTrace();
transaction.rollback();
}
but then I get the errors: ARJUNA016051: thread is already associated with a transaction! and ARJUNA016079: Transaction rollback status is:ActionStatus.COMMITTED
I tried both the declarative and API based methods again this time with context propagation enabled. But still no luck.
Finally, based on the third approach, I thought keeping the #Transactional on the Http request handler and leaving longRunning as is without declarative or API based transaction approaches would work. However the database still does not get updated.
Clearly I am misunderstanding how JTA and context propagation works (among other things).
Is there a way (or even a design pattern) that allows me to update database entities asynchronously in a quarkus web application? Also why wouldn't any of the approaches I took have any effect?
Using quarkus 1.4.1.Final with ext: [agroal, cdi, flyway, hibernate-orm, hibernate-orm-panache, hibernate-validator, kubernetes-client, mutiny, narayana-jta, rest-client, resteasy, resteasy-jackson, resteasy-mutiny, smallrye-context-propagation, smallrye-health, smallrye-openapi, swagger-ui]
You should return an async type from your JAX-RS resource method, the transaction context will then be available when the async stage executes. There is some relevant documentation in the quarkus guide on context propagation.
I would start by looking at the one of the reactive examples such as the getting started quickstart. Try annotating each resource endpoint with #Transactional and the async code will run with a transaction context.
I have set up a simple Spring Integration flow which is composed of such steps:
poll a rest api periodically then
do some processing on the payload
and land it on a Kafka topic.
Please observe the code below:
#Component
public class MyIntegrationFlow extends IntegrationFlowAdapter {
#Override
protected IntegrationFlowDefinition<?> buildFlow() {
return from(() -> List.of("pathVariable1", "pathVariable2"), c -> c.poller(Pollers.fixedDelay(5, TimeUnit.SECONDS)))
.split()
.handle(httpRequest(), c -> c.advice(new RequestHandlerRetryAdvice()))
.transform(Tranformers.fromJson(Foo.class))
.filter(payload -> payload.isValid())
.log()
.transform(Tranformers.toJson())
.channel(Source.OUTPUT); // output channel for kafka topic
}
private HttpMessageHandlerSpec httpRequest() {
return Http.outboundGateway("http://somehost:8080/{pathVariable}")
.httpMethod(GET)
.uriVariable("pathVariable", Message::getPayload)
.expectedResponseType(String.class);
}
}
This works brilliantly, however, I am struggling to come up with some good tests.
How am I supposed to mock the external REST API?
How am I supposed to test that the retry policy does kick in and the desired number of http requests are made?
How do I change the MessageSource of the flow (list of path vars) that is polled periodically?
How do I check if the payload has successfully made it to the Kafka topic?
Too much questions and some of them requires too broad explanation. Anyway I think you can start from Spring Integration Testing Framework and its documentation.
How am I supposed to mock the external REST API?
I think you can just consider to use a Mock MVC from Spring Framework and its MockMvcClientHttpRequestFactory to inject into the HttpRequestExecutingMessageHandler based on the HttpMessageHandlerSpec.
retry policy does kick
Well, I guess the same mocked MVC endpoint can verify how many times it has been called and fail for first several times to initiate that retry.
How do I change the MessageSource
This is exactly a part of Spring Integration Testing Framework with its MockIntegration.mockMessageSource() and MockIntegrationContext: https://docs.spring.io/spring-integration/docs/5.1.6.RELEASE/reference/html/#mockintegration
made it to the Kafka topic?
Or you the mentioned MockIntegration.mockMessageHandler() to verify that endpoint for Kafka is called. Or use an Embedded Kafka from Spring Kafka project: https://docs.spring.io/spring-kafka/docs/2.2.7.RELEASE/reference/html/#embedded-kafka-annotation
I am trying to configure integration flow with gateway. Using Java DSL on kotlin.
Gateway config:
#MessagingGateway(name = "tdiOutSenderGateway")
interface TdiOutSenderGateway {
fun send(packet: PhasorEnricher.Packet)
}
Flow config:
#Bean
open fun tdiOutSendFlow() = IntegrationFlows
.from(TdiOutSenderGateway::class.java)
.transform(tdiOutSenderRouter())
.get()!!
got send is not supported, because no request channel has been configured
docs: request channel will be autoconfigured.
#IntegrationComponentScan exists
https://docs.spring.io/spring-integration/reference/htmlsingle/#java-dsl-gateway readed and used as base
double checked: I am using exactly this gateway
Is there any additional setting I missed?
Of course, transform should return something, there should be route or handle.
But even I fixed #1 I faced with that: Void kotlin functions return Unit. Spring integrations checks Unit == null which is false, tries to find the next channel and throw an error. The fix is to use kotlin lambda and return null explicitly.
After working with Spring Integration and kotlin these 8 months I decided to try to create Kotlin DSL for Spring Integration (https://github.com/spring-projects/spring-integration/issues/3016)
Kotlin Unit: https://kotlinlang.org/docs/reference/functions.html#unit-returning-functions
P.S.: Of course I solved it earlier, not after 8 months.
I have below application:
Its windows console .NET 3.0 application
I'm creating 20 workloads and assigning them to threadpool to process.
Each thread in ThreadPool creates WCF Client and calls service with request created using workload assigned.
Sometimes on production servers[12 core machines], I get following exception:
There was an error reflecting type 'xyz' while invoking operation using WCF client. This starts appearing in all threads. After sometime it suddenly disappears and starts appearing again.
Code:
Pseudo Code:
for(int i=0;i<20;i++)
{
MultiThreadedProcess proc =new MultThreadedProcess(someData[i]);
ThreadPool.QueueUserWorkItem(proc.CallBack,i);
}
In Class MultiThreadedProcess, I do something like this:
public void Callback(object index)
{
MyServiceClient client = new MyServiceClient();
MyServiceResponse response =client.SomeOperation(new MyServiceRequest(proc.SomeData));
client.close();
//Process Response
}
Can anyone suggest some resolutions for this problem?
If you can turn on diagnostic, appears to me serialization issue, there might be chance that certain data members/values are not able to de-serialized properly for operation call.