if the configured channel in int-jdbc:inbound-channel-adapter is full,after that polling happens or not ?
Can I use a queue channel in int-jdbc:inbound-channel-adapter?
My requirement is that if the channel is full,no DB call should make ,until channel is free.
You need to use a QueueChannel with restricted capacity. As far as any Inbound Channel Adapter uses a MessagingTemplate with the sendTimeout = -1, the QueueChannel performs a put() operation on the underlying LinkedBlockingQueue:
/**
* Inserts the specified element at the tail of this queue, waiting if
* necessary for space to become available.
*
* #throws InterruptedException {#inheritDoc}
* #throws NullPointerException {#inheritDoc}
*/
public void put(E e) throws InterruptedException {
To ensure that the poller is blocked until space is available you should use a fixed-delay option do not allow any parallel calls to DB.
So, in other words so far so good as long as you use default options.
Related
As a result of Spring integration MessageQueue without polling, I have a poller that consumes messages from a queue instantly, using a custom TaskScheduler:
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setThreadNamePrefix("resultProcessor-");
IntegrationFlows
.from("inbound")
.channel(MessageChannels.priority().get())
.bridge(bridge -> bridge
.taskScheduler(taskScheduler)
.poller(Pollers.fixedDelay(0).receiveTimeout(Long.MAX_VALUE)))
.fixedSubscriberChannel()
.route(inboundRouter())
.get()
Now I'd like to have multiple threads consuming concurrently, so I tried:
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setThreadNamePrefix("resultProcessor-");
scheduler.setPoolSize(4);
However, since in AbstractPollingEndpoint the task scheduler schedules a synchronous poller (it's a bit complicated), only 1 thread is created. If I set the TaskExecutor to anything but SyncTaskExecutor (default), I run into a flood of scheduled tasks (see Spring integration MessageQueue without polling).
How can I concurrently consume from a queue in Spring Integration? This seems pretty basic but I couldn't find a solution.
Instead of the queue I could use an ExecutorChannel, however, (AFAIK) I then lose queue features like priority, queue size, and metrics on which I rely.
See PollerSpec.taskExecutor():
/**
* Specify an {#link Executor} to perform the {#code pollingTask}.
* #param taskExecutor the {#link Executor} to use.
* #return the spec.
*/
public PollerSpec taskExecutor(Executor taskExecutor) {
This way after scheduling the task periodically according your taskScheduler and delay, the real task is performed on a thread from that provided executor. By default it indeed performs the task on a scheduler's thread.
UPDATE
I'm not sure if this meets your requirements, but this is only way to keep your queue logic and process whatever is pull in parallel:
.bridge(bridge -> bridge
.taskScheduler(taskScheduler)
.poller(Pollers.fixedDelay(0).receiveTimeout(Long.MAX_VALUE)))
.channel(channels -> channel.executor(threadPoolExecutor()))
.fixedSubscriberChannel()
I was able to solve it like this:
A single-threaded task scheduler that performs the polling
A thread pool executor with a synchronous queue
This way, the task scheduler can give each executor 1 task and blocks when no executor is free, thus not draining the source queue or spamming tasks.
#Bean
public IntegrationFlow extractTaskResultFlow() {
return IntegrationFlows
.from(ChannelNames.TASK_RESULT_QUEUE)
.bridge(bridge -> bridge
.taskScheduler(taskResultTaskScheduler())
.poller(Pollers
.fixedDelay(0)
.taskExecutor(taskResultExecutor())
.receiveTimeout(Long.MAX_VALUE)))
.handle(resultProcessor)
.channel(ChannelNames.TASK_FINALIZER_CHANNEL)
.get();
}
#Bean
public TaskExecutor taskResultExecutor() {
ThreadPoolExecutor executor = new ThreadPoolExecutor(
1, // corePoolSize
8, // maximumPoolSize
1L, // keepAliveTime
TimeUnit.MINUTES,
new SynchronousQueue<>(),
new CustomizableThreadFactory("resultProcessor-")
);
executor.setRejectedExecutionHandler(new CallerBlocksPolicy(Long.MAX_VALUE));
return new ErrorHandlingTaskExecutor(executor, errorHandler);
}
#Bean
public TaskScheduler taskResultTaskScheduler() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setThreadNamePrefix("resultPoller-");
return scheduler;
}
(The initial example was copied from the linked question, this one now resembles my actual solution)
I have read that redis is single threaded per user/client.
Is it possible to use jedis connection pool to achieve multithreaded behaviour?
Basically different jedis client request would be using different connection obatined from jedis connection pool , to fire their commands to one redis server.
Since client requests are using different connections hence one redis should be server multiple requests parallely right ?
Is it possible to use jedis connection pool to achieve multithreaded behaviour?
Though you use multiple connection or threading or processing to communicating with redis, These commands are all put on one queue, and redis will get one by one and execute one by one.And every single command is atomic. So this is still a single thread behaviour at the aspect of the redis.
Since client requests are using different connections hence one redis should be server multiple requests parallely right ?
Not always, the commands from different clients may arrive at redis at arbitrary order, so if these command has some data relation, dangerous things will happen.
consider this simple scenario, if redis has a counter S, if all client do the INCR command, then it is ok, because every command is executed atomicly, But if some client's code is like something that:
s = get S
t = s + 10
set S t
Then things would be wrong. And you should use the Multi/exec command to ensure that multi commands would be atomic no just only one command. More things about transactions in redis, you could refer this page. It is detailed.
I answered similar question here, you may check it for details.
TL/DR: Redis itself will still be single threaded, but in the grand scheme of things time used by Redis to answer is order of magnitude less than time used by networking. You will get benefit from multithreading, yes.
EDIT: Along the request pipeline you will have 3 points where requests are processed sequentially. Its when the data packets are sent through a wire (you only have one wire, but the packets from different requests will be mixed) forward and backward, and when Redis serves your request. But Redis itself is no dumb either. Your requests will come from a network socket and will be pre-processed and written into client buffers. This part is processed in parallel. Then Redis's main loop will pick your commands from a queue, process and write response into client buffers. This part is done sequentially. Then responses from client buffers will be packed and sent through a wire back to you. This is also being done in parallel.
This works fine in my (web based) multi-threaded environment.
Make sure that RedisInterface instance be static in your class.
public class RedisInterface implements Closeable
{
/*
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version>
</dependency>
*/
private JedisPool jedisPool = null;
private synchronized void initializePool()
{
if(jedisPool!=null) return;
JedisPoolConfig poolConfig = new JedisPoolConfig();
poolConfig.setMaxTotal(Integer.parseInt(AppConfig.REDIS_MAX_CONN)); //max conn count
poolConfig.setMaxIdle(Integer.parseInt(AppConfig.REDIS_MAX_IDLE_CONN)); //max idle conn count
poolConfig.setMaxWaitMillis(Long.parseLong(AppConfig.REDIS_MAX_WAIT_TIME)); // max wait time for new connection (before throwing exception)
jedisPool = new JedisPool(poolConfig,
AppConfig.REDIS_IP,
Integer.parseInt(AppConfig.REDIS_PORT),
Integer.parseInt(AppConfig.REDIS_CONN_TIMEOUT));
}
//not synchronized after testing thread safety of jedisPool.getResource()
protected Jedis getJedis()
{
if(jedisPool==null)
initializePool();
Jedis jedis = jedisPool.getResource();
return jedis;
}
public Long set(final byte[] key, final byte[] field, final byte[] value)
{
Jedis redis = null;
Long ret =0L;
try
{
redis = getJedis();
ret = redis.hset(key, field, value);
redis.expire(key, Integer.parseInt(AppConfig.REDIS_EXPIRE_MIN)*60); //
}
finally
{
if(redis!=null)
redis.close();
}
return ret;
}
public byte[] get(final byte[] key, final byte[] field) {
Jedis redis = null ;
byte[] valueBytes = null;
try
{
redis = getJedis();
valueBytes = redis.hget(key, field);
}
finally
{
if(redis!=null)
redis.close();
}
return valueBytes;
}
#Override
public void close() throws IOException {
if(jedisPool!=null)
jedisPool.close();
}
}
The function pm_runtime_put_sync() is called in spi-omap2-mcspi.c
Can somebody please explain what actually this function call does.
Thank you!
It calls __pm_runtime_idle(dev, RPM_GET_PUT) internally which is documented as
int __pm_runtime_idle(struct device *dev, int rpmflags)
Entry point for runtime idle operations.
* #dev: Device to send idle notification for.
* #rpmflags: Flag bits.
*
* If the RPM_GET_PUT flag is set, decrement the device's usage count and
* return immediately if it is larger than zero. Then carry out an idle
* notification, either synchronous or asynchronous.
*
* This routine may be called in atomic context if the RPM_ASYNC flag is set,
* or if pm_runtime_irq_safe() has been called.
here are the source and Documentation
I have a spring batch job which takes-in flat file, processes the records and writes-out the output to another flat file.
I have used FlatFileItemReader and FlatFileItemWriter as reader and writer respectively.
However, when I try to implement multi-threaded steps, my job does not work properly. I get following warnings in my log file
WARN ChunkMonitor:109 - No ItemReader set (must be concurrent step), so ignoring offset data.
WARN ChunkMonitor:141 - ItemStream was opened in a different thread. Restart data could be compromised.
Can you please help me out in implementing multi-threaded step?
This is because FlatFileItemReader is not thread-safe as the doc says :
/**
* Abstract superclass for {#link ItemReader}s that supports restart by storing
* item count in the {#link ExecutionContext} (therefore requires item ordering
* to be preserved between runs).
*
* Subclasses are inherently *not* thread-safe.
*
* #author Robert Kasanicky
*/
To implements a multi-thread reader, you will have to write a custom reader that synchronize
the calls to open and update from the ItemStream interface. If not, your job won't be safe to be restartable.
Hope it helps
Regards
I'm working on a Java EE Web application.
I've developed a Scheduled Batch using #WebListener annotation:
#WebListener
public class BatchServlet implements ServletContextListener {
#Override
public void contextInitialized(ServletContextEvent sce) {
context = sce.getServletContext();
schedulerEngine = Executors.newSingleThreadScheduledExecutor();
schedulerEngine.scheduleAtFixedRate(new MyThread(), 0, 300, TimeUnit.SECONDS);
}
public class MyThread implements Runnable {
#Override
public void run() {
...... //my business logic
}
and it works correctly.
But, for my purpose, I need to force the MyThread() batch on demand, for example, when I received a particular request within a Servlet. Obviosly I can call the MyBatch thread separately, but I was wondering that there are some best practices to perform this operation.
There are a couple of options: create a regular thread pool (with multiple threads) and just put tasks on the executor queue whenever they come in.
If you need more fine-grained control, you can share something like a concurrent queue between the servlet and the one thread that you have. Let this thread read from the queue, it will block automatically when the queue is empty.
Then your servlet can put work on this queue, if the thread happened to be blocked on getting an item it will automatically wake up.
You might wanna look up the "producer-consumer" pattern. There is a lot of literature about this.