For long-running message-based requests, we can set the visibility timeout at the queue level, but there doesn't appear to be a way to override it at the message level. I'd like to be able to extend the time for long-running operations without just setting a queue-wide extra long timeout. The underlying SQS service provides the ChangeMessageVisibility function but it's not exposed. From what I can tell, RedisMQ doesn't support the concept at all.
Is there effectively ever a timeout and retry for RedisMQ?
Is it possible to expose the ability to extend visibility time from within a service? A long-running operation could then call it periodically to get enough time.
Redis MQ is not related to Amazon SQS or ServiceStack's Amazon SQS MQ other than they're different concrete implementations of ServiceStack's Messaging APIs but a ServiceStack AppHost can only have 1 IMessageService registered so I don't understand how using Redis MQ is relevant to your use of Amazon SQS?
Redis MQ is built on top of Redis's List and Pub/Sub primitives and doesn't contain any such MQ customizations.
If you're just referring to Amazon SQS MQ then there's a RegisterHandler overload that lets you specify the visibilityTimeoutSeconds per message Type so you could have long-running requests executed with a different Request DTO Type which would be my recommendation to keep them isolated. ServiceStack's Auto Mapping makes it easy to convert between Request DTOs with the same schema, e.g:
public object Any(MyRequest request) { ... }
public object Any(LongRunning request) => Any(request.ConvertTo<MyRequest>());
The SqsMqServer does have RequestFilter and ResponseFilter you can use to inspect the IMessage<T> that's returned by ServiceStack MQ which can be used to change the metadata sent in the SQS Message but not any of its custom message-level properties.
To enable fine-grained access to SQS I've added the following filters to SqsMqServer, SqsMqMessageFactory and SqsMqClient in this commit which will allow you to intercept and customize the requests going to and coming from Amazon SQS:
Action<SendMessageRequest,IMessage> SendMessageRequestFilter { get; set; }
Action<ReceiveMessageRequest> ReceiveMessageRequestFilter { get; set; }
Action<Amazon.SQS.Model.Message, IMessage> ReceiveMessageResponseFilter { get; set; }
Action<DeleteMessageRequest> DeleteMessageRequestFilter { get; set; }
Action<ChangeMessageVisibilityRequest> ChangeMessageVisibilityRequestFilter { get; set; }
This change is available from v5.4.1 that's now available on MyGet.
Related
I am trying to test a Spring Integration flow that starts off from a message-driven-channel-adapter configured as:
<int-jms:message-driven-channel-adapter id="myAdapter" ... />
My test goes like:
#SpringJUnitConfig(locations = {"my-app-context.xml"})
#SpringIntegrationTest(noAutoStartup = {"myAdapter"})
public class MyIntegrationFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#Test
public void myTest() {
...
MessageSource<String> messageSource = () -> new GenericMessage<>("foo");
mockIntegrationContext.substituteMessageSourceFor("myAdapter", messageSource);
...
}
}
I am however getting the following error:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'myAdapter' is expected to be of type 'org.springframework.integration.endpoint.SourcePollingChannelAdapter' but was actually of type 'org.springframework.integration.jms.JmsMessageDrivenEndpoint'
How should one specify an alternate source for the channel adapter for testing using the MockIntegrationContext, or by some other method?
The Message Driver Channel Adapter is really not a Source Polling Channel Adapter. So, the substituteMessageSourceFor() is indeed cannot be used for that type of components, which, essentially is a MessageProducerSupport implementation, not a SourcePollingChannelAdapter for a MessageSource.
The difference exists because not all protocols provides a listener-like hooks to spawn some self-managed task to subscribe to. The good example is JDBC, which is only passive system expecting requests. Therefore a polling channel adapter with a JdbcPollingChannelAdapter (which is a MessageSource) implementation must be used to interact with DB in event-driven manner.
Other systems (like JMS in your case) provides some listener (or consumer) API for what we can spawn a while task (see MessageListenerContainer in spring-jms) and let its MessageProducerSupport to emit messages to the channel.
Therefore you need to distinguish for yourself with what type of component you interact before choosing a testing strategy.
Since there is no extra layer in case of message-driver channel adapter, but rather some specific, self-managed MessageProducerSupport impl, we not only provide a particular mocking API, but even don't require to know anything else, but just standard unit testing feature and a message channel this endpoint is producing in the configuration.
So, the solution for you is something like:
#SpringIntegrationTest(noAutoStartup = {"myAdapter"}) - that's fully correct in your code: we really have to stop the real channel adapter to not pollute our testing environment.
You just need to inject into your test class a MessageChannel that id="myAdapter" is producing to. In your test code you just build a Message and send it into this channel. No need to worry about a MockIntegrationContext at all.
I have set my Domain Model objects to be independent of any service and infrastructure logic.
I am also using Domain Events to react to some changes in Domain Models.
Now my problem is how to raise those events from the Domain Model objects itself.
Currently I am using Udi Dahan's DomainEvents static class for this (I need evens to be handled exactly when they happen and not at a latter time).
The events are used for many things, like logging, updating the data in related services and other Domain Model objects and db, publishing messages to the MassTransit bus etc.
The DomainEvents static class uses Autofac scope that I inject at some point in it, to find the IMediatr instance and to publish the events, like this:
public static class DomainEvents
{
private static ILifetimeScope Scope;
public async static Task RaiseAsync<TDomainEvent>(TDomainEvent #event) where TDomainEvent : IDomainEvent
{
var mediator = Scope?.Resolve<IMediatorBus>();
if (mediator != null)
{
await mediator!.Publish(#event).ConfigureAwait(false);
}
else
{
Debug.WriteLine("Mediator not set for DomainEvents!");
}
}
public static void SetScope(ILifetimeScope scope)
{
Scope = scope;
}
}
This all works ok in a single-threaded environment, but the method DomainEvents.SetScope() is a possible racing problem in multhi-threaded environment.
Ie. When I introduce MassTransit and create message consumers, each Message consumer will set the current LifetimeScope to DomainEvents by that method, and here is the problem, each consumer will overwrite the lifetime scope with the new one.
Why I use DomainEvents static class? Because I don't want to pollute my Domain Model Objects with infrastructure stuff.
I thought about making DomainEvents non static (define an interface), but then I need them injected in every Domain Model Object and I'm still thinking about this, but maybe there is a better way.
I want to know if there is a better way to handle this?
Maybe some change in DomainEvents class? Or maybe remove the DomainEvents static class end use an interface or DomainService to do this.
The problem is I don't like static classes, but I also don't like pushing non domain-specific dependencies into Domain Model Objects.
Please help.
UPDATE
To better clarify the process and for what I use DomainEvents...
I have a long-running process that can take from few minutes to few hours/days to complete.
So the process is going like this:
I receive an message from MassTransit ie ProcessStartMessage(processId)
Get the ProcessData for (processId) from Db.
Construct an in-memory Domain Model ProcessTracker (singleton) and put all the data I loaded from DB in it. (in-memory cache)
I receive another message from Masstransit ie. ProcessStatusChanged(processId, data).
Forward this message data to in-memory singleton ProcessTracker to process.
ProcessTracker process the data.
For ProcessTracker to be able to process this data it instantiates many Domain Model Objects, each responsible to process some part of the data. (Note there is NO more db calls and entity hydration from db, it all happens in memory, also Domain Model is not mapped to any entity, it is not connected to any db object).
At some point I need to log what a Domain Model object in the chain has done, has it work finished or started, has reached some milestone etc. This is done by raising DomainEvents. I also need to notify the GUI of those events, so they are used to send Masstransit messages too.
Ie.(pseudo code):
public class ProcessTracker
{
private Step _currentStep;
public void ProcessData(data)
{
_currentStep.ProcessData(data);
DomainEvents.Raise(new ProcesTrackerDataProcessed());
...
}
}
public class Step
{
public Phase _currentPhase;
public void ProcessData(data)
{
if (data.IsManual && _someOtherCondition())
{
DomainEvents.Raise(new StepDataEvent1());
...
}
if(data.CanTransition)
{
DomainEvents.Raise(new TransitionToNewPhase(this, data));
}
_currentPhase.DoSomeWork(data);
DomainEvents.Raise(new StepDataProcessed(this, data));
...
}
}
About db updates, those are not transactional and not important to the process and the Domain Model Object state is kept only in memory, if the process crash the process MUST begin from the start (there is NO recovery).
To end the process:
I receive ProcessEnd from the MassTransit
The message data is forwarded to the ProcessTracker
ProcessTracker handles the data an nets a result of the proceess
The result of the process is saved to db
A message is sent to other parties in the process that notifies them of a process completion.
Ask yourself first what are you going to do when you raise an event from your domain model?
Normally it works like this:
Get a command
Load a domain object from a repository
Execute behaviour
(here probably) Raise an event
Persist the new domain object state
So, where your extra domain event handlers would fit? Are you going to execute some other database calls, send an email? Remember that it all happens now, when you haven't even persisted the changed state of your domain object. What if your persistence fails? It will happen after you executed all the domain handlers.
You should not execute more than one transaction when you handle a single command. The Aggregate pattern clearly tells you that the aggregate is the transaction boundary. You should raise domain events after you complete the transaction, or within the same technical transaction, but it should only persist the aggregate state and the event. Domain event reactions potentially trigger transactions for other domain objects, and it should be done outside of the scope of handling the current command.
The issue is not at all technical, it is a design problem.
If you use MassTransit, you can only make it (relatively) reliable if you handle the command in a message consumer. Then, you can use in-memory outbox, which will not send an event unless the consumer succeeded. It is still not guaranteed that the event will be published in case of the broker failure.
Unless you go to Event Sourcing, you have two 100% reliable options:
Use a transactional outbox pattern (NServiceBus has one and it's quite complex). It has limitations on what type of database you use.
Store the event to the same database as the domain object, in a different table, within the same transaction. Poll the table with DELETE INTO and emit events to the broker from there.
I'm trying to learn how to handle MQTT Messages in Spring-Integration.
Have created a converter, that subscribes with a single MqttPahoMessageDrivenChannelAdapter per MQTT Topic for consuming and converting the messages.
The problem is our data provider is planning to "speed-up" publishing messages on his side. So instead of having a few(<=10) topics each of which has messages with about 150 fields it is planned to publish each of those fields to the separate MQTT topic.
This means my converter would have to consume ca. 1000 mqtt topics, but I do not know whether:
Is spring-integration still a good choice for it. Cause afaik. the mentioned adapter uses the PAHO MqttClient that will consume the messages from all of the topics it is subscribed to in one single thread and creating 1000 instances of those adapters is an overkill.
If we stick further to spring-integration and use the provided components, would it be a good idea to create a single inbound adapter for all of the fields, that previously were in messages of one topic but moving the conversion away from the adapter bean to a separate bean ( that does the conversion) connected with an executer-channel to the adapter and thus executing the conversion of those fields on some threadpool in parallel.
Thanks in advance for your answers!
I think your idea makes sense.
For that purpose you need to implement a passthrough MqttMessageConverter and provide an MqttMessage as a payload and topic as a header:
public class PassThroughMqttMessageConverter implements MqttMessageConverter {
#Override
public Message<?> toMessage(String topic, MqttMessage mqttMessage) {
return MessageBuilder.withPayload(mqttMessage)
.setHeader(MqttHeaders.RECEIVED_TOPIC, topic)
.build();
}
#Override
public Object fromMessage(Message<?> message, Class<?> targetClass) {
return null;
}
#Override
public Message<?> toMessage(Object payload, MessageHeaders headers) {
return null;
}
}
So, you really will be able to perform a target conversion downstream, after a mentioned ExecutorChannel in the custom transformer.
You also may consider to implement a custom MqttPahoClientFactory (an extension of the DefaultMqttPahoClientFactory may work as well) and provide a custom ScheduledExecutorService to be injected into the MqttClient you are going create in the getClientInstance().
I'm familiarizing myself DDD lately and trying to get hold of the key concepts and I have a query incase of publishing domain events for the local subscribers, so can I assume that the event publisher, takes care of both publishing to remote subscribers thro AQMP while also leveraging observable to publish it towards the local subscribers, is this a scalable solution? or is there a familiar pattern to handle this?(Also advise if there is a reactive solution to the prblm, may be RxJava or the likes)
The locality of subscribers, local or remote, should be completely transparent to the domain entity raising an publishing the event.
Your thoughts are very similar to design I use. I use a quite primitive version of a local event publishers, which communicate via messaging middleware (AMQP to propagate events to all other local publishers).
I'm not doing that in Java, but in fact, for the local event "bus" I use Rx.Net (reactive extensions have an almost identical API in all languages, so RxJava would work)
a simplified version with transaction support ripped out would look like this:
public class EventHub
{
private readonly ISubject<object, object> _messages;
public EventHub()
{
var _subject = new Subject<object>();
m_messages = Subject.Synchronize(_subject);
}
public void Publish<T>(T message)
{
m_messages.OnNext(message);
}
public IObservable<T> AsObservable<T>()
{
return m_messages
.ObserveOn(ThreadPoolScheduler.Instance)
.OfType<T>()
.Synchronize()
.AsObservable();
}
}
Adding remote capability would be adding another subscriber, which would subscribe to all events and route them to other local EventHubs, and also listen to other eventhubs and publish their events on the local hub. It would not change the EventHub component.
I have a jsf view which shows some data from a managed-bean (viewscope) in a table, that is retrieved remotely.
Currently the data is updated via polling from the client-view using primefaces poll component.
This is not sufficient enough, since to much traffic is sent to client and now that primefaces supports server-push I only want to reload the data and push it to the client-view if data has been changed.
This should be realized via polling from the web-tier to application tier calling a method like hasChanged(...). If data is changed the web-tier pushes a notification to client to reload data.
Current client poll
client >> web-tier >> app-tier
client asks web-tier via ajax for data which again asks app-tier for data and updates view
Wished web-tier poll and push
client << web-tier >> app-tier
web-tier polls app-tier if data has changed and reloads on behalf and informs (pushes) client to update view
Approaches:
What is the best approach to realize the polling at the managed-bean in the web-tier?
TimerTask in the managed-bean
Spawning threads in a JSF managed bean for scheduled tasks using a timer
Additional EJB with Schedule annotation
Additional EJB with TimerService
other?
Edit:
Architecture: (3-tier)
Server1: database
Server2: app-tier (EAR with Remote EJB + Hibernate)
Server3: web-tier (WAR with JSF 2.0 + Primefaces 3.4)
Client: Browser
Based on my experience, I can recommend two routes, Spring Integration and CDI Events. I'd recommend the Spring route, but based on your current stack, I think CDI events have you covered. They cleanly help you achieve the Observer/Observable pattern and you can pull of a clean separation of tiers too. I must warn you though, this approach is effective only for a small-medium use case. Consider the following:
Design and implement an Event class that encapsulates all the information that is required to be delivered to consumers. Let's call it a FundsTransfer event Your event implementation should contain enough information to enable listeners filter only for events of interest.
A simple POJO
class FundsTransfer {
BigDecimal transferValue;
Date transferDate;
int transferCurrencyCode;
public FundsTransfer(BigDecimal transferValue, Date date, int currencyCode) {
//set accordingly
}
//setters and getters
}
Implement a business object at the business layer, call it Notifier. The function of polling should be delegated to this object. It will be in charge of creating and publishing objects of type event in response to changes on the server side. Depending on your requirements, this object could be a singleton to handle all Event types or you could have a group of Notifier types polling for different events.
//A sample implementation of your Observer object :
#Singleton //Defines a singleton EJB
public class PollerService {
#Inject
Event fundsTranferNotifier;
//this annotation specifies that the polling method should run every second.
#Schedule(second = "*/1", minute = "*", hour = "*", persistent = false)
public void pollIt() {
boolean found = this.pollingMethod(); // pollingMethod() will do the actual polling
if (found) { //based on the outcome of the polling method, fire the notifier method
blowWhistleOnTransfer();
}
}
public void blowWhistleOnTransfer() {
//this is the broadcast event.
fundsTransferNotifier.fire(new FundsTransfer(new BigDecimal("100000", new Date(), 855));
}
}
In the code above, I've used a Timer EJB as my Observer. See this for an introduction to EJB timers. Again, the Observer object will live in the app tier
The client tier will each have access to an listener object that will be notified when an event of interest occurs (that is has been published by a Notifier type). Then the listener can issue a push based on this event. Your listener object could be a POJO with #Named CDI annotation. In your listener object, simply implement a method with an #Observes annotation, with a parameter of the type of event the listener is interested in:
public void onNewTransfer(#Observes FundsTransfer transfer) {
if (transfer.compareTo(new BigDecimal("150000")) > 0) {
//push to view.
}
}
The above filtering is still quite crude compared to the message filtering options that CDI provides. As you can see from the tutorial I referenced earlier, you could create CDI qualifiers that can give you finer grained filtering on messages. Like I stated earlier, this is a bit heavy for large scale deployment, in which case I'd advise the spring integration route, if you're up to take the dependency on.
In summary, the model of the system will be:
One Poller(Notifier) Object (In the app layer)
|
|
|
Multiple Listener Objects (In the web tier)
---------------------------------------------------
| | | | | | | | | | | |