spring-boot: 2.4.4
spring-integration-redis: 5.4.5
#Bean
#Primary
public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory factory) {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(factory);
redisTemplate.setEnableTransactionSupport(true);
......
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
#Autowired
private RedisLockRegistry redisLock;
final Lock lock = redisLock.obtain(lockKey);
boolean lockFlag = lock.tryLock(2, TimeUnit.SECONDS);
i follow the code to JedisScriptingCommands
public <T> T eval(byte[] script, ReturnType returnType, int numKeys, byte[]... keysAndArgs) {
Assert.notNull(script, "Script must not be null!");
if (isQueueing() || isPipelined()) {
throw new UnsupportedOperationException();
}
try {
return (T) new JedisScriptReturnConverter(returnType)
.convert(connection.getJedis().eval(script, JedisConverters.toBytes(numKeys), keysAndArgs));
} catch (Exception ex) {
throw convertJedisAccessException(ex);
}
}
in this method, check if in transaction, if true throw UnsupportedOperationException.
i don't understand why does it not support transaction, and if there any way to bypass it?
Doesn't look like you really have something relevant to Spring Integration in your problem at all. Feels more like a question to Jedis driver.
Consider to use a Lettuce instead: https://lettuce.io/
Related
What do I need to do to prevent the following Exception which is presumably thrown by RabbitMQ.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:877)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:787)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:707)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:98)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1236)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:688)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1190)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1174)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1200(SimpleMessageListenerContainer.java:98)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1363)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'amqpLaunchSpringBatchJobFlow.channel#0'; nested exception is jp.ixam_drive.batch.service.JobExecutionRuntimeException: Failed to start job with name ads-insights-import and parameters {accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:449)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:171)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$400(AmqpInboundChannelAdapter.java:45)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:95)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:784)
... 10 common frames omitted
Caused by: jp.ixam_drive.batch.service.JobExecutionRuntimeException: Failed to start job with name ads-insights-import and parameters {accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}
at jp.ixam_drive.facebook.SpringBatchLauncher.launchJob(SpringBatchLauncher.java:42)
at jp.ixam_drive.facebook.AmqpBatchLaunchIntegrationFlows.lambda$amqpLaunchSpringBatchJobFlow$1(AmqpBatchLaunchIntegrationFlows.java:71)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
... 18 common frames omitted
Caused by: org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException: A job instance already exists and is complete for parameters={accessToken=<ACCESS_TOKEN>, id=act_1234567890, classifier=stats, report_run_id=1482330625184792, job_request_id=32}. If you want to run this job again, change the parameters.
at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:126)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:172)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy125.createJobExecution(Unknown Source)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:125)
at jp.ixam_drive.batch.service.JobOperationsService.launch(JobOperationsService.java:64)
at jp.ixam_drive.facebook.SpringBatchLauncher.launchJob(SpringBatchLauncher.java:37)
... 24 common frames omitted
when I have 2 instances of Spring Boot application both of which run the following code in parallel to execute Spring Batch Jobs?
#Configuration
#Conditional(AmqpBatchLaunchCondition.class)
#Slf4j
public class AmqpAsyncAdsInsightsConfiguration {
#Autowired
ObjectMapper objectMapper;
#Value("${batch.launch.amqp.routing-keys.async-insights}")
String routingKey;
#Bean
public IntegrationFlow amqpOutboundAsyncAdsInsights(AmqpTemplate amqpTemplate) {
return IntegrationFlows.from("async_ads_insights")
.<JobParameters, byte[]>transform(SerializationUtils::serialize)
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey(routingKey)).get();
}
#Bean
public IntegrationFlow amqpAdsInsightsAsyncJobRequestFlow(FacebookMarketingServiceProvider serviceProvider,
JobParametersToApiParametersTransformer transformer, ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, routingKey))
.<byte[], JobParameters>transform(SerializationUtils::deserialize)
.<JobParameters, ApiParameters>transform(transformer)
.<ApiParameters>handle((payload, header) -> {
String accessToken = (String) header.get("accessToken");
String id = (String) header.get("object_id");
FacebookMarketingApi api = serviceProvider.getApi(accessToken);
String reportRunId = api.asyncRequestOperations().getReportRunId(id, payload.toMap());
ObjectNode objectNode = objectMapper.createObjectNode();
objectNode.put("accessToken", accessToken);
objectNode.put("id", id);
objectNode.put("report_run_id", reportRunId);
objectNode.put("classifier", (String) header.get("classifier"));
objectNode.put("job_request_id", (Long) header.get("job_request_id"));
return serialize(objectNode);
}).channel("ad_report_run_polling_channel").get();
}
#SneakyThrows
private String serialize(JsonNode jsonNode) {
return objectMapper.writeValueAsString(jsonNode);
}
}
#Configuration
#Conditional(AmqpBatchLaunchCondition.class)
#Slf4j
public class AmqpBatchLaunchIntegrationFlows {
#Autowired
SpringBatchLauncher batchLauncher;
#Value("${batch.launch.amqp.routing-keys.job-launch}")
String routingKey;
#Bean(name = "batch_launch_channel")
public MessageChannel batchLaunchChannel() {
return MessageChannels.executor(Executors.newSingleThreadExecutor()).get();
}
#Bean
public IntegrationFlow amqpOutbound(AmqpTemplate amqpTemplate,
#Qualifier("batch_launch_channel") MessageChannel batchLaunchChannel) {
return IntegrationFlows.from(batchLaunchChannel)
.<JobParameters, byte[]>transform(SerializationUtils::serialize)
.handle(Amqp.outboundAdapter(amqpTemplate).routingKey(routingKey)).get();
}
#Bean
public IntegrationFlow amqpLaunchSpringBatchJobFlow(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, routingKey))
.handle(message -> {
String jobName = (String) message.getHeaders().get("job_name");
byte[] bytes = (byte[]) message.getPayload();
JobParameters jobParameters = SerializationUtils.deserialize(bytes);
batchLauncher.launchJob(jobName, jobParameters);
}).get();
}
}
#Configuration
#Slf4j
public class AsyncAdsInsightsConfiguration {
#Value("${batch.core.pool.size}")
public Integer batchCorePoolSize;
#Value("${ixam_drive.facebook.api.ads-insights.async-poll-interval}")
public String asyncPollInterval;
#Autowired
ObjectMapper objectMapper;
#Autowired
private DataSource dataSource;
#Bean(name = "async_ads_insights")
public MessageChannel adsInsightsAsyncJobRequestChannel() {
return MessageChannels.direct().get();
}
#Bean(name = "ad_report_run_polling_channel")
public MessageChannel adReportRunPollingChannel() {
return MessageChannels.executor(Executors.newFixedThreadPool(batchCorePoolSize)).get();
}
#Bean
public IntegrationFlow adReportRunPollingLoopFlow(FacebookMarketingServiceProvider serviceProvider) {
return IntegrationFlows.from(adReportRunPollingChannel())
.<String>handle((payload, header) -> {
ObjectNode jsonNode = deserialize(payload);
String accessToken = jsonNode.get("accessToken").asText();
String reportRunId = jsonNode.get("report_run_id").asText();
try {
AdReportRun adReportRun = serviceProvider.getApi(accessToken)
.fetchObject(reportRunId, AdReportRun.class);
log.debug("ad_report_run: {}", adReportRun);
return jsonNode.set("ad_report_run", objectMapper.valueToTree(adReportRun));
} catch (Exception e) {
log.error("failed while polling for ad_report_run.id: {}", reportRunId);
throw new RuntimeException(e);
}
}).<JsonNode, Boolean>route(payload -> {
JsonNode adReportRun = payload.get("ad_report_run");
return adReportRun.get("async_percent_completion").asInt() == 100 &&
"Job Completed".equals(adReportRun.get("async_status").asText());
}, rs -> rs.subFlowMapping(true,
f -> f.transform(JsonNode.class,
source -> {
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder
.addString("accessToken", source.get("accessToken").asText());
jobParametersBuilder.addString("id", source.get("id").asText());
jobParametersBuilder
.addString("classifier", source.get("classifier").asText());
jobParametersBuilder
.addLong("report_run_id", source.get("report_run_id").asLong());
jobParametersBuilder
.addLong("job_request_id", source.get("job_request_id").asLong());
return jobParametersBuilder.toJobParameters();
}).channel("batch_launch_channel"))
.subFlowMapping(false,
f -> f.transform(JsonNode.class, this::serialize)
.<String>delay("delay", asyncPollInterval, c -> c.transactional()
.messageStore(jdbcMessageStore()))
.channel(adReportRunPollingChannel()))).get();
}
#SneakyThrows
private String serialize(JsonNode jsonNode) {
return objectMapper.writeValueAsString(jsonNode);
}
#SneakyThrows
private ObjectNode deserialize(String payload) {
return objectMapper.readerFor(ObjectNode.class).readValue(payload);
}
#Bean
public JdbcMessageStore jdbcMessageStore() {
JdbcMessageStore jdbcMessageStore = new JdbcMessageStore(dataSource);
return jdbcMessageStore;
}
#Bean
public JobParametersToApiParametersTransformer jobParametersToApiParametersTransformer() {
return new JobParametersToApiParametersTransformer() {
#Override
protected ApiParameters transform(JobParameters jobParameters) {
ApiParameters.ApiParametersBuilder builder = ApiParameters.builder();
MultiValueMap<String, String> multiValueMap = new LinkedMultiValueMap<>();
String level = jobParameters.getString("level");
if (!StringUtils.isEmpty(level)) {
multiValueMap.set("level", level);
}
String fields = jobParameters.getString("fields");
if (!StringUtils.isEmpty(fields)) {
multiValueMap.set("fields", fields);
}
String filter = jobParameters.getString("filter");
if (filter != null) {
try {
JsonNode jsonNode = objectMapper.readTree(filter);
if (jsonNode != null && jsonNode.isArray()) {
List<ApiFilteringParameters> filteringParametersList = new ArrayList<>();
List<ApiSingleValueFilteringParameters> singleValueFilteringParameters = new ArrayList<>();
ArrayNode arrayNode = (ArrayNode) jsonNode;
arrayNode.forEach(node -> {
String field = node.get("field").asText();
String operator = node.get("operator").asText();
if (!StringUtils.isEmpty(field) && !StringUtils.isEmpty(operator)) {
String values = node.get("values").asText();
String[] valuesArray = !StringUtils.isEmpty(values) ? values.split(",") : null;
if (valuesArray != null) {
if (valuesArray.length > 1) {
filteringParametersList.add(ApiFilteringParameters
.of(field, Operator.valueOf(operator), valuesArray));
} else {
singleValueFilteringParameters.add(ApiSingleValueFilteringParameters
.of(field, Operator.valueOf(operator), valuesArray[0]));
}
}
}
});
if (!filteringParametersList.isEmpty()) {
builder.filterings(filteringParametersList);
}
if (!singleValueFilteringParameters.isEmpty()) {
builder.filterings(singleValueFilteringParameters);
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
String start = jobParameters.getString("time_ranges.start");
String end = jobParameters.getString("time_ranges.end");
String since = jobParameters.getString("time_range.since");
String until = jobParameters.getString("time_range.until");
if (!StringUtils.isEmpty(start) && !StringUtils.isEmpty(end)) {
builder.timeRanges(ApiParameters.timeRanges(start, end));
} else if (!StringUtils.isEmpty(since) && !StringUtils.isEmpty(until)) {
builder.timeRange(new TimeRange(since, until));
}
String actionBreakdowns = jobParameters.getString("action_breakdowns");
if (!StringUtils.isEmpty(actionBreakdowns)) {
multiValueMap.set("action_breakdowns", actionBreakdowns);
}
String attributionWindows = jobParameters.getString("action_attribution_windows");
if (attributionWindows != null) {
try {
multiValueMap
.set("action_attribution_windows",
objectMapper.writeValueAsString(attributionWindows.split(",")));
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
builder.multiValueMap(multiValueMap);
String pageSize = jobParameters.getString("pageSize");
if (!StringUtils.isEmpty(pageSize)) {
builder.limit(pageSize);
}
return builder.build();
}
};
}
}
Here is how message flows:
1. channel[async_ads_insights] ->IntegrationFlow[amqpOutboundAsyncAdsInsights]->[AMQP]->IntegrationFlow[amqpAdsInsightsAsyncJobRequestFlow]->channel[ad_report_run_polling_channel]->IntegrationFlow[adReportRunPollingLoopFlow]-IF END LOOP->channel[batch_launch_channel] ELSE -> channel[ad_report_run_polling_channel]
2. channel[batch_launch_channel] -> IntegrationFlow[amqpOutbound]-> IntegrationFlow[amqpLaunchSpringBatchJobFlow]
3. Spring Batch Job is launched.
The exception isn't thrown immediately after both instances are started, but after a while. Launching Spring Batch Jobs do succeeds but then start to fail with "A job instance already exists and is complete for..."
The job is for retrieving facebook ads results.
I would appreciate your insights into what is causing the error above.
I also have this configuration which does not use AMQP and works without any problem, but it is only for one instance.
#Configuration
#Conditional(SimpleBatchLaunchCondition.class)
#Slf4j
public class SimpleBatchLaunchIntegrationFlows {
#Autowired
SpringBatchLauncher batchLauncher;
#Autowired
DataSource dataSource;
#Bean(name = "batch_launch_channel")
public MessageChannel batchLaunchChannel() {
return MessageChannels.queue(jdbcChannelMessageStore(), "batch_launch_channel").get();
}
#Bean
public ChannelMessageStoreQueryProvider channelMessageStoreQueryProvider() {
return new MySqlChannelMessageStoreQueryProvider();
}
#Bean
public JdbcChannelMessageStore jdbcChannelMessageStore() {
JdbcChannelMessageStore channelMessageStore = new JdbcChannelMessageStore(dataSource);
channelMessageStore.setChannelMessageStoreQueryProvider(channelMessageStoreQueryProvider());
channelMessageStore.setUsingIdCache(true);
channelMessageStore.setPriorityEnabled(true);
return channelMessageStore;
}
#Bean
public IntegrationFlow launchSpringBatchJobFlow(#Qualifier("batch_launch_channel")
MessageChannel batchLaunchChannel) {
return IntegrationFlows.from(batchLaunchChannel)
.handle(message -> {
String jobName = (String) message.getHeaders().get("job_name");
JobParameters jobParameters = (JobParameters) message.getPayload();
batchLauncher.launchJob(jobName, jobParameters);
}, e->e.poller(Pollers.fixedRate(500).receiveTimeout(500))).get();
}
}
Refer to the Spring Batch documentation. When launching a new instance of a job, the job parameters must be unique.
A common solution is to add a dummy parameter with a UUID or similar but batch provides a strategy, e.g to increment a numeric parameter each time.
EDIT
There is a certain class of exceptions where the members of which are considered irrecoverable (fatal) and it makes no sense to attempt redelivery.
Examples include MessageConversionException - if we can't convert it the first time, we probably can't convert on a redelivery. The ConditionalRejectingErrorHandler is the mechanism by which we detect such exceptions, and cause them to be permanently rejected (and not redelivered).
Other exceptions cause the message to be redelivered by default - there is another property defaultRequeuRejected which can be set to false to permanently reject all failures (not recommended).
You can customize the error handler by subclassing its DefaultExceptionStrategy - override isUserCauseFatal(Throwable cause) to scan the cause tree to look for a JobInstanceAlreadyCompleteException and return true (cause.getCause().getCause() instanceof ...)
I think it was triggered by the error thrown by the "SpringBatch job running already" exception.
That still indicates you have somehow received a second message with the same parameters; it's a different error because the original job is still running; that message is rejected (and requeued) but on subsequent deliveries you get the already completed exception.
So, I still say the root cause of your problem is duplicate requests, but you can avoid the behavior with a customized error handler in the channel adapter's listener container.
I suggest you log the duplicate message so you can figure out why you are getting them.
I have a Spring Integration configuration that utilizes a priority channel. When an item is read from that channel, local resources are checked at that point in time, and if the resources are not available to process the item, I would like to requeue the message so that another machine picks it up. Originally, I wrongly threw an exception thinking that a requeue would occur, but as was answered in my other question this is not going to work since the priority channel executes in another thread than the listener container.
I thought about placing a filter right after the inbound channel adapter, and throwing an exception if resources are not available at that time, but at that instance in time an accurate assessment of resources cannot be made because resource availability at that time does match what will be available when the message is selected based upon priority.
My next thought is to place a filter after the priority channel and before the service activator and direct messages that cannot be handled by current resources to the discard-channel which is defined as an outbound channel adapter that sends the message back to the original queue. Are there pitfalls to this approach?
EDIT 20150917:
Per Gary's advice, I have moved to RabbitMQ 3.5.x in order to take of the built-in priority queues. I now have a problem tracking the number of attempts as it appears my original message is placed back on the queue, rather than my modified message. I have updated the code blocks to reflect the current setup.
EDIT 20150922:
I am updating this post to reflect the final proof of concept code base that I created. I am not a Spring-Integration expert by any means, so please keep that in mind as well as the fact that this test code is not production ready. My original intent was to have messages resubmitted and retried a certain amount of times if a particular exception was thrown. This can be accomplished using the StatefulRetryOperationsInterceptor. But to experiment further, I wanted to be able to set/increment a header on failure and then have something in my flow that could react to that value. That was accomplished by using an extension of the RepublishMessageRecoverer that overrides additionalHeaders(). This object then is used to configure the RetryOperationsInterceptor.
One other minor thing: I wanted to reduce some of the default Spring Integration logging when my signal exception was thrown, so I needed to make sure I named my error channel "errorChannel" in order to replace the Spring Integration default. I also needed to create a custom ErrorHandler which to assign to the ListenerContainer default which logs everything to ERROR level.
Here is my current setup:
Spring Integration 4.2.0.RELEASE
Spring AMQP 1.5.0.RELEASE
RabbitMQ 3.5.x
Configuration
#Autowired
public void setSpringIntegrationConfigHelper (SpringIntegrationHelper springIntegrationConfigHelper) {
this.springIntegrationConfigHelper = springIntegrationConfigHelper;
}
#Bean
public String priorityPOCQueueName() {
return "poc.priority";
}
#Bean
public Queue priorityPOCQueue(RabbitAdmin rabbitAdmin) {
boolean durable = true;
boolean exclusive = false;
boolean autoDelete = false;
//Adding the x-max-priority argument is what signals RabbitMQ that this is a priority queue. Must be Rabbit 3.5.x
Map<String,Object> arguments = new HashMap<String, Object>();
arguments.put("x-max-priority", 5);
Queue queue = new Queue(priorityPOCQueueName(),
durable,
exclusive,
autoDelete,
arguments);
rabbitAdmin.declareQueue(queue);
return queue;
}
#Bean
public Binding priorityPOCQueueBinding(RabbitAdmin rabbitAdmin) {
Binding binding = new Binding(priorityPOCQueueName(),
DestinationType.QUEUE,
"amq.direct",
priorityPOCQueue(rabbitAdmin).getName(),
null);
rabbitAdmin.declareBinding(binding);
return binding;
}
#Bean
public AmqpTemplate priorityPOCMessageTemplate(ConnectionFactory amqpConnectionFactory,
#Qualifier("priorityPOCQueueName") String queueName,
#Qualifier("jsonMessageConverter") MessageConverter messageConverter) {
RabbitTemplate template = new RabbitTemplate(amqpConnectionFactory);
template.setChannelTransacted(false);
template.setExchange("amq.direct");
template.setQueue(queueName);
template.setRoutingKey(queueName);
template.setMessageConverter(messageConverter);
return template;
}
#Autowired
#Qualifier("priorityPOCQueue")
public void setPriorityPOCQueue(Queue priorityPOCQueue) {
this.priorityPOCQueue = priorityPOCQueue;
}
#Bean
public MessageRecoverer miTestMessageRecoverer(final AmqpTemplate priorityPOCMessageTemplate) {
return new MessageRecoverer() {
#Override
public void recover(org.springframework.amqp.core.Message msg, Throwable t) {
StringBuilder sb = new StringBuilder();
sb.append("Firing Test Recoverer: ").append(t.getClass().getName()).append(" Message Count: ")
.append(msg.getMessageProperties().getMessageCount())
.append(" ID: ").append(msg.getMessageProperties().getMessageId())
.append(" DeliveryTag: ").append(msg.getMessageProperties().getDeliveryTag())
.append(" Redilivered: ").append(msg.getMessageProperties().isRedelivered());
logger.debug(sb.toString());
PriorityMessage m = new PriorityMessage(5);
m.setId(randomGenerator.nextLong(10L, 1000000L));
priorityPOCMessageTemplate.convertAndSend(m , new SimulateErrorHeaderPostProcessor(Boolean.FALSE, m.getPriority()));
}
};
}
#Bean
public RepublishMessageRecoverer miRepublishRecoverer(final AmqpTemplate priorityPOCMessageTemplate) {
class MiRecoverer extends RepublishMessageRecoverer {
public MiRecoverer(AmqpTemplate errorTemplate) {
super(errorTemplate);
this.setErrorRoutingKeyPrefix("");
}
#Override
protected Map<? extends String, ? extends Object> additionalHeaders(
org.springframework.amqp.core.Message message, Throwable cause) {
Map<String, Object> map = new HashMap<>();
if (message.getMessageProperties().getHeaders().containsKey("jmattempts") == false) {
map.put("jmattempts", 0);
} else {
Integer count = Integer.valueOf(message.getMessageProperties().getHeaders().get("jmattempts").toString());
map.put("jmattempts", ++count);
}
return map;
}
} ;
return new MiRecoverer(priorityPOCMessageTemplate);
}
#Bean
public StatefulRetryOperationsInterceptor inadequateResourceInterceptor(#Qualifier("priorityPOCMessageTemplate") AmqpTemplate priorityPOCMessageTemplate
, #Qualifier("priorityMessageKeyGenerator") PriorityMessageKeyGenerator priorityMessageKeyGenerator
, #Qualifier("miTestMessageRecoverer") MessageRecoverer messageRecoverer
, #Qualifier("miRepublishRecoverer") RepublishMessageRecoverer miRepublishRecoverer) {
StatefulRetryInterceptorBuilder b = RetryInterceptorBuilder.stateful();
return b.maxAttempts(2)
.backOffOptions(2000L, 1.0D, 4000L)
.messageKeyGenerator(priorityMessageKeyGenerator)
.recoverer(miRepublishRecoverer)
.build();
}
#Bean(name="exec.priorityPOC")
TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor e = new ThreadPoolTaskExecutor();
e.setCorePoolSize(1);
e.setQueueCapacity(1);
return e;
}
/* #Bean(name="poc.priorityChannel")
public MessageChannel pocPriorityChannel() {
PriorityChannel c = new PriorityChannel(new PriorityComparator());
c.setComponentName("poc.priorityChannel");
c.setBeanName("poc.priorityChannel");
return c;
}
*/
#Bean(name="poc.inputChannel")
public MessageChannel pocPriorityChannel() {
DirectChannel c = new DirectChannel();
c.setComponentName("poc.inputChannel");
c.setBeanName("poc.inputChannel");
return c;
}
#Bean(name="poc.inboundChannelAdapter") //make this a unique name
public AmqpInboundChannelAdapter amqpInboundChannelAdapter(#Qualifier("exec.priorityPOC") TaskExecutor taskExecutor
, #Qualifier("errorChannel") MessageChannel pocErrorChannel
, #Qualifier("inadequateResourceInterceptor") StatefulRetryOperationsInterceptor inadequateResourceInterceptor) {
org.aopalliance.aop.Advice[] adviceChain = new org.aopalliance.aop.Advice[]{inadequateResourceInterceptor};
int concurrentConsumers = 1;
AmqpInboundChannelAdapter a = springIntegrationConfigHelper.createInboundChannelAdapter(taskExecutor
, pocPriorityChannel(), new Queue[]{priorityPOCQueue}, concurrentConsumers, adviceChain
, new PocErrorHandler());
a.setErrorChannel(pocErrorChannel);
return a;
}
#Transformer(inputChannel = "poc.inputChannel", outputChannel = "poc.procesPoc")
public Message<PriorityMessage> incrementAttempts(Message<PriorityMessage> msg) {
//I stopped using this in the POC.
return msg;
}
#ServiceActivator(inputChannel="poc.procesPoc")
public void procesPoc(#Header(SimulateErrorHeaderPostProcessor.ERROR_SIMULATE_HEADER_KEY) Boolean simulateError
, #Headers Map<String, Object> headerMap
, PriorityMessage priorityMessage) throws InterruptedException {
if (isFirstMessageReceived == false) {
//Thread.sleep(15000); //Cause a bit of a backup so we can see prioritizing in action.
isFirstMessageReceived = true;
}
Integer retryAttempts = 0;
if (headerMap.containsKey("jmattempts")) {
retryAttempts = Integer.valueOf(headerMap.get("jmattempts").toString());
}
logger.debug("Received message with priority: " + priorityMessage.getPriority() + ", simulateError: " + simulateError + ", Current attempts count is "
+ retryAttempts);
if (simulateError && retryAttempts < PriorityMessage.MAX_MESSAGE_RETRY_COUNT) {
logger.debug(" Simulating an error and re-queue'ng. Current attempt count is " + retryAttempts);
throw new AnalyzerNonAdequateResourceException();
} else if (simulateError && retryAttempts > PriorityMessage.MAX_MESSAGE_RETRY_COUNT) {
logger.debug(" Max attempt count exceeded");
}
}
/**************************************************************************************************
*
* Error Channel
*
**************************************************************************************************/
//Note that we want to override default Spring error channel, so the name of the bean must be errorChannel
#Bean(name="errorChannel")
public MessageChannel pocErrorChannel() {
DirectChannel c = new DirectChannel();
c.setComponentName("errorChannel");
c.setBeanName("errorChannel");
return c;
}
#ServiceActivator(inputChannel="errorChannel")
public void pocHandleError(Message<MessagingException> message) throws Throwable {
MessagingException me = message.getPayload();
logger.error("pocHandleError: error encountered: " + me.getCause().getClass().getName());
SortedMap<String, Object> sorted= new TreeMap<>();
sorted.putAll(me.getFailedMessage().getHeaders());
if (me.getCause() instanceof AnalyzerNonAdequateResourceException) {
logger.debug("Headers: " + sorted.toString());
//Let this message get requeued
throw me.getCause();
}
Message<?> failedMsg = me.getFailedMessage();
Object o = failedMsg.getPayload();
StringBuilder sb = new StringBuilder();
if (o != null) {
sb.append("AnalyzerErrorHandler: Failed Message Type: ")
.append(o.getClass().getCanonicalName()).append(". toString: ").append(o.toString());
logger.error(sb.toString());
}
//The first level sometimes brings back either MessagingHandlingException or
//MessagingTransformationException which may contain a subcause
Exception e = (Exception)me.getCause();
int i = 0;
sb.delete(0, sb.length());
sb.append("AnalyzerErrorHandler nested messages: ");
while (e != null && i++ < 10) {
sb.append(System.lineSeparator()).append(" ")
.append(e.getClass().getCanonicalName()).append(": ")
.append(e.getMessage());
}
if (i > 0) {
logger.error(sb.toString());
}
//Don't want a message to recycle
throw new AmqpRejectAndDontRequeueException(e);
}
/**
* This gets set on the ListenerContainer. The default handler on the listener
* container logs everything with full stack trace. We don't want to do that
* for our known resource exception
*/
public static class PocErrorHandler implements ErrorHandler {
#Override
public void handleError(Throwable t) {
Throwable cause = t.getCause();
if (cause != null) {
while (cause.getCause() != null) {
cause = cause.getCause();
}
} else {
cause = t;
}
if (cause instanceof AnalyzerNonAdequateResourceException) {
logger.info(AnalyzerNonAdequateResourceException.class.getName() + ": not enough resources to process the item.");
return;
}
else {
logger.error("POC Listener Exception", t);
}
}
}
SpringIntegrationHelper
protected ConnectionFactory connectionFactory;
protected MessageConverter messageConverter;
#Autowired
public void setConnectionFactory (ConnectionFactory connectionFactory) {
this.connectionFactory = connectionFactory;
}
#Autowired
public void setMessageConverter(#Qualifier("jsonMessageConverter") MessageConverter messageConverter) {
this.messageConverter = messageConverter;
}
public AmqpInboundChannelAdapter createInboundChannelAdapter(TaskExecutor taskExecutor
, MessageChannel outputChannel, Queue[] queues, int concurrentConsumers
, org.aopalliance.aop.Advice[] adviceChain,
ErrorHandler errorHandler) {
SimpleMessageListenerContainer listenerContainer =
new SimpleMessageListenerContainer(connectionFactory);
//AUTO is default, but setting it anyhow.
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
listenerContainer.setAutoStartup(true);
listenerContainer.setConcurrentConsumers(concurrentConsumers);
listenerContainer.setMessageConverter(messageConverter);
listenerContainer.setQueues(queues);
//listenerContainer.setChannelTransacted(false);
listenerContainer.setErrorHandler(errorHandler);
listenerContainer.setPrefetchCount(1);
listenerContainer.setTaskExecutor(taskExecutor);
listenerContainer.setDefaultRequeueRejected(true);
if (adviceChain != null && adviceChain.length > 0) {
listenerContainer.setAdviceChain(adviceChain);
}
AmqpInboundChannelAdapter a = new AmqpInboundChannelAdapter(listenerContainer);
a.setMessageConverter(messageConverter);
a.setAutoStartup(true);
a.setHeaderMapper(MyAmqpHeaderMapper.createPassAllHeaders());
a.setOutputChannel(outputChannel);
return a;
}
It's not clear why you want to use a PriorityChannel in this context; why not use a priority queue in RabbitMQ? That way, you can run your flow on the container thread.
Sending the queue to the back of the queue yourself would work, but there is a risk of message loss.
We are launching a website that will have a very heavy volume for a short period of time. It is basically giving tickets. The code is written in Java, Spring & Hibernate. I want to mimic the high volume by spawning multiple threads and trying to get the ticket using JUnit test case. The problem is that in my DAO class the code just simply dies after I begin transaction. I mean there is no error trace in the log file or anything like that. Let me give some idea about the way my code is.
DAO code:
#Repository("customerTicketDAO")
public class CustomerTicketDAO extends BaseDAOImpl {// BaseDAOImpl extends HibernateDaoSupport
public void saveCustomerTicketUsingJDBC(String customerId) {
try{
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
// A select query
Query query1 = getSession().createSQLQuery("my query omitted on purpose");
.
.
// An update query
Query query2 = getSession().createSQLQuery("my query omitted on purpose");
getSession().getTransaction().commite();
} catch (Exception e) {
}
}
Runnable code:
public class InsertCustomerTicketRunnable implements Runnable {
#Autowired
private CustomerTicketDAO customerTicketDAO;
public InsertCustomerTicketRunnable(String customerId) {
this.customerId = customerId;
}
#Override
public void run() {
if (customerTicketDAO != null) {
customerTicketDAO.saveCustomerTicketUsingJDBC(customerId);
}
}
}
JUnit method:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations={"file:src/test/resources/applicationContext-test.xml"})
public class DatabaseTest {
#Before
public void init() {
sessionFactory = (SessionFactory)applicationContext.getBean("sessionFactory");
Session session = SessionFactoryUtils.getSession(sessionFactory, true);
TransactionSynchronizationManager.bindResource(sessionFactory, new SessionHolder(session));
customerTicketDAO = (CustomerTicketDAO)applicationContext.getBean("customerTicketDAO");
}
#After
public void end() throws Exception {
SessionHolder sessionHolder = (SessionHolder) TransactionSynchronizationManager.unbindResource(sessionFactory);
SessionFactoryUtils.closeSession(session);
}
#Test
public void saveCustomerTicketInMultipleThreads () throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (int i=0; i<1000; i++) {
executor.submit(new InsertCustomerTicketRunnable(i));
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
// Wait until all threads are finish
executor.awaitTermination(1, TimeUnit.SECONDS);
}
I see no data being inserted into the database. Can someone please point me as to where I am going wrong?
Thanks
Raj
SessionFactory is thread safe but Session is not. So my guess is that you need to call SessionFactoryUtils.getSession() from within each thread, so that each thread gets its own instance. You are currently calling it from the main thread, so all children threads try to share the same instance.
Naughty, naughty!
public void saveCustomerTicketUsingJDBC(String customerId) {
try {
getSession().getTransaction().begin(); //NOTHING HAPPENS AFTER THIS LINE OF CODE
.
.
} catch (Exception e) {
}
}
You should never (well, hardly ever) have an empty catch block, if there is a problem you will find that your code 'just simply dies' with no log messages. Oh look, that's what's happening ;)
At the very minimum you should log the exception, that will go a long way towards you helping you find what the problem is (and from there, the solution).
I wanted to debug the Seed() method in my Entity Framework database configuration class when I run Update-Database from the Package Manager Console but didn't know how to do it. I wanted to share the solution with others in case they have the same issue.
Here is similar question with a solution that works really well.
It does NOT require Thread.Sleep.
Just Launches the debugger using this code.
Clipped from the answer
if (!System.Diagnostics.Debugger.IsAttached)
System.Diagnostics.Debugger.Launch();
The way I solved this was to open a new instance of Visual Studio and then open the same solution in this new instance of Visual Studio. I then attached the debugger in this new instance to the old instance (devenv.exe) while running the update-database command. This allowed me to debug the Seed method.
Just to make sure I didn't miss the breakpoint by not attaching in time I added a Thread.Sleep before the breakpoint.
I hope this helps someone.
If you need to get a specific variable's value, a quick hack is to throw an exception:
throw new Exception(variable);
A cleaner solution (I guess this requires EF 6) would IMHO be to call update-database from code:
var configuration = new DbMigrationsConfiguration<TContext>();
var databaseMigrator = new DbMigrator(configuration);
databaseMigrator.Update();
This allows you to debug the Seed method.
You may take this one step further and construct a unit test (or, more precisely, an integration test) that creates an empty test database, applies all EF migrations, runs the Seed method, and drops the test database again:
var configuration = new DbMigrationsConfiguration<TContext>();
Database.Delete("TestDatabaseNameOrConnectionString");
var databaseMigrator = new DbMigrator(configuration);
databaseMigrator.Update();
Database.Delete("TestDatabaseNameOrConnectionString");
But be careful not to run this against your development database!
I know this is an old question, but if all you want is messages, and you don't care to include references to WinForms in your project, I made some simple debug window where I can send Trace events.
For more serious and step-by-step debugging, I'll open another Visual Studio instance, but it's not necessary for simple stuff.
This is the whole code:
SeedApplicationContext.cs
using System;
using System.Data.Entity;
using System.Diagnostics;
using System.Drawing;
using System.Windows.Forms;
namespace Data.Persistence.Migrations.SeedDebug
{
public class SeedApplicationContext<T> : ApplicationContext
where T : DbContext
{
private class SeedTraceListener : TraceListener
{
private readonly SeedApplicationContext<T> _appContext;
public SeedTraceListener(SeedApplicationContext<T> appContext)
{
_appContext = appContext;
}
public override void Write(string message)
{
_appContext.WriteDebugText(message);
}
public override void WriteLine(string message)
{
_appContext.WriteDebugLine(message);
}
}
private Form _debugForm;
private TextBox _debugTextBox;
private TraceListener _traceListener;
private readonly Action<T> _seedAction;
private readonly T _dbcontext;
public Exception Exception { get; private set; }
public bool WaitBeforeExit { get; private set; }
public SeedApplicationContext(Action<T> seedAction, T dbcontext, bool waitBeforeExit = false)
{
_dbcontext = dbcontext;
_seedAction = seedAction;
WaitBeforeExit = waitBeforeExit;
_traceListener = new SeedTraceListener(this);
CreateDebugForm();
MainForm = _debugForm;
Trace.Listeners.Add(_traceListener);
}
private void CreateDebugForm()
{
var textbox = new TextBox {Multiline = true, Dock = DockStyle.Fill, ScrollBars = ScrollBars.Both, WordWrap = false};
var form = new Form {Font = new Font(#"Lucida Console", 8), Text = "Seed Trace"};
form.Controls.Add(tb);
form.Shown += OnFormShown;
_debugForm = form;
_debugTextBox = textbox;
}
private void OnFormShown(object sender, EventArgs eventArgs)
{
WriteDebugLine("Initializing seed...");
try
{
_seedAction(_dbcontext);
if(!WaitBeforeExit)
_debugForm.Close();
else
WriteDebugLine("Finished seed. Close this window to continue");
}
catch (Exception e)
{
Exception = e;
var einner = e;
while (einner != null)
{
WriteDebugLine(string.Format("[Exception {0}] {1}", einner.GetType(), einner.Message));
WriteDebugLine(einner.StackTrace);
einner = einner.InnerException;
if (einner != null)
WriteDebugLine("------- Inner Exception -------");
}
}
}
protected override void Dispose(bool disposing)
{
if (disposing && _traceListener != null)
{
Trace.Listeners.Remove(_traceListener);
_traceListener.Dispose();
_traceListener = null;
}
base.Dispose(disposing);
}
private void WriteDebugText(string message)
{
_debugTextBox.Text += message;
Application.DoEvents();
}
private void WriteDebugLine(string message)
{
WriteDebugText(message + Environment.NewLine);
}
}
}
And on your standard Configuration.cs
// ...
using System.Windows.Forms;
using Data.Persistence.Migrations.SeedDebug;
// ...
namespace Data.Persistence.Migrations
{
internal sealed class Configuration : DbMigrationsConfiguration<MyContext>
{
public Configuration()
{
// Migrations configuration here
}
protected override void Seed(MyContext context)
{
// Create our application context which will host our debug window and message loop
var appContext = new SeedApplicationContext<MyContext>(SeedInternal, context, false);
Application.Run(appContext);
var e = appContext.Exception;
Application.Exit();
// Rethrow the exception to the package manager console
if (e != null)
throw e;
}
// Our original Seed method, now with Trace support!
private void SeedInternal(MyContext context)
{
// ...
Trace.WriteLine("I'm seeding!")
// ...
}
}
}
Uh Debugging is one thing but don't forget to call:
context.Update()
Also don't wrap in try catch without a good inner exceptions spill to the console.
https://coderwall.com/p/fbcyaw/debug-into-entity-framework-code-first
with catch (DbEntityValidationException ex)
I have 2 workarounds (without Debugger.Launch() since it doesn't work for me):
To print message in Package Manager Console use exception:
throw new Exception("Your message");
Another way is to print message in file by creating a cmd process:
// Logs to file {solution folder}\seed.log data from Seed method (for DEBUG only)
private void Log(string msg)
{
string echoCmd = $"/C echo {DateTime.Now} - {msg} >> seed.log";
System.Diagnostics.Process.Start("cmd.exe", echoCmd);
}
I have a service that needs to invoke a runnable class.
Here are the lines of code that are being used in my service.
#Autowired
private LinkBrc2MemberProfile brcTask;
// Background Task.
SimpleAsyncTaskExecutor sate = new SimpleAsyncTaskExecutor();
sate.createThread(new LinkBrc2MemberProfile(user));
Here is my Runnable class
#Service
public class LinkBrc2MemberProfile implements Runnable {
private final Logger log = LoggerFactory.getLogger(LinkBrc2MemberProfile.class);
#Autowired
private LoyaltyDao dao;
private Member member;
public LinkBrc2MemberProfile() {
super();
}
public LinkBrc2MemberProfile(Member member) {
this.member = member;
}
public void run() {
log.debug("*** Member User Name: " + member.getString("USER_NAME"));
String emailAddress = member.getString("USER_NAME");
Map<String, Object> map = dao.findBrcByEmailAddress( emailAddress );
log.debug("==========================================================");
if( ! map.isEmpty() ) {
try {
//a.CUSTOMER_ID, a.EMAIL_ADDRESS, b.card_no
String customerId = (String) map.get("CUSTOMER_ID");
String brcCardNumber = (String) map.get("CARD_NO");
log.debug("\ncustomerId: " + customerId + " brcCardNumber: " + brcCardNumber);
if(!brcCardNumber.equals("")) {
// Add the Be Rewarded Card.
HashMap<String, String> userAttributes = new HashMap<String, String>();
String brcNumber = member.getString("BREWARDED_CARD_NO");
if (brcNumber.equals("")) {
userAttributes.put("BREWARDED_CARD_NO", brcCardNumber);
try {
member.putAll(userAttributes);
} catch (Exception e) {
String errorMessage = "Unable to save user's BRC information due to: " + e.getMessage();
log.error("{}", errorMessage);
}
}
}
} catch (Exception e) {
log.error(e.getMessage());
e.printStackTrace();
}
}
}
I'm not seeing any errors in the log but at the same time it does not appear to be invoking the Runnable class. Am I missing an annotation somewhere? Are there any good examples that you can point me to, the only ones I have found use XML files to configure the runnable class I would like to use annotations. Thanks in Advance.
I've updated my service to do the following.
Please help, my DAO is NULL so it looks like my #Autowired in my Runnable class is not wiring it in.
I've added the following bean to my bean-config.xml file.
<bean id="brcType" class="com.ws.ocp.service.LinkBrc2MemberProfile" scope="prototype"/>
I removed my #Autowired annotation and added the following to my service class.
ClassPathResource rsrc = new ClassPathResource("bean-config.xml");
XmlBeanFactory factory = new XmlBeanFactory(rsrc);
LinkBrc2MemberProfile brcTask = (LinkBrc2MemberProfile) factory.getBean("brcType");
SimpleAsyncTaskExecutor sate = new SimpleAsyncTaskExecutor();
// Set Member attribute
brcTask.setMember(user);
// Executer
sate.execute(brcTask);
Why is my dao still null?
The runnable will throw a NullPointerException, since you create it yourself (using the new operator), instead of letting Spring create it. This obviously means that the autowired DAO attribute won't be autowired, which will lead to a NPE when calling dao.findBrcByEmailAddress(...).
You should get your Runnable instance from the bean factory (as a prototype), set its member attribute, and then submit it to the executor.
To answer your question of how to properly use a Prototype-Bean, this is my favorite way:
#Component
abstract class MyBean {
/* Factory method that will be installed by Spring */
#Lookup
protected abstract YourPrototypeBean createBean();
void someCode() {
YourPrototypeBean bean = createBean();
}
}
Since it's a factory method, you can create as many instances as you like.