My scheduler's application context defines this trgger:
<bean id="myTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="myJob"/>
<property name="cronExpression" value="0 0 ∗ ∗ ∗ ?"/>
</bean>
Does it fire every day at 00:00? Or every hour?
I'd say the latter, but the documentation of this project says otherwise...
Can you help me out?
Are there different kind of expressions?
Spring is using quartz for scheduling jobs so it is definitely every hour.
Please check:
http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/tutorial-lesson-06
Related
I am reading a message , transforming it and outputting on the JMS channel. The JMS channel uses a WorkManager Task Executor to read the messages and processes it.
Even though we configured the WorkManager in application server to have 10 threads, only one thread is being used.
<si:chain id="prenotifchain" input-channel="preNotificationChannel" output-channel="notificationJMSChannel">
<si:transformer id="prenotif" method="transformRequest" ref="notificationTransformer"/>
</si:chain>
<si-jms:channel id="notificationJMSChannel" queue="notificationQueue" connection-factory="queueConnectionFactory" transaction-manager="txManager" task-executor="notificationTaskExecutor" />
<jee:jndi-lookup id="notificationQueue" jndi-name="jms/notifqueue"/>
<bean id="notificationTaskExecutor"
class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
<property name="workManagerName" value="notifWM" />
<property name="resourceRef" value="true" />
</bean>
Are we missing any configuration or is there another way to read multiple ?
Please, use concurrency attribute:
<xsd:attribute name="concurrency" type="xsd:string">
<xsd:annotation>
<xsd:documentation><![CDATA[
The number of concurrent sessions/consumers to start for each listener.
Can either be a simple number indicating the maximum number (e.g. "5")
or a range indicating the lower as well as the upper limit (e.g. "3-5").
Note that a specified minimum is just a hint and might be ignored at runtime.
Default is 1; keep concurrency limited to 1 in case of a topic listener
or if message ordering is important; consider raising it for general queues.
]]></xsd:documentation>
</xsd:annotation>
</xsd:attribute>
We have an application which can consume around 300 JMS messages per minute. We need to increase the speed to 3000 messages per minute.
I created a simple test program which reads the messages from the queue and logs the messages. No processing is involved, so I expected a high speed. However, the logging is still happening at a speed of around 400 messages per minute.
Below are the excerpts of my program
<int-jms:message-driven-channel-adapter id="testJmsInboundAdapter"
auto-startup="true"
destination="testQueueDestination"
connection-factory="testConnectionFactory"
channel="messageTransformerChannel" />
<int:channel id="messageTransformerChannel" />
<int:service-activator
id="loggerActivator"
input-channel="messageTransformerChannel"
method="log"
ref="logger" />
The logger method simply logs the message
public void log(final GenericMessage<Object> object) {
LOGGER.info("Logging message" + object);
}
Any advise where should I look at the bottleneck. Is there any limitation on the number of messages that can be consumed per minute using spring integration's message-driven-channel-adapter
Pay attention to these options:
<xsd:attribute name="concurrent-consumers" type="xsd:string">
<xsd:annotation>
<xsd:documentation>
Specify the number of concurrent consumers to create. Default is 1.
Specifying a higher value for this setting will increase the standard
level of scheduled concurrent consumers at runtime: This is effectively
the minimum number of concurrent consumers which will be scheduled
at any given time. This is a static setting; for dynamic scaling,
consider specifying the "maxConcurrentConsumers" setting instead.
Raising the number of concurrent consumers is recommendable in order
to scale the consumption of messages coming in from a queue. However,
note that any ordering guarantees are lost once multiple consumers are
registered
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
<xsd:attribute name="max-concurrent-consumers" type="xsd:string">
<xsd:annotation>
<xsd:documentation>
Specify the maximum number of concurrent consumers to create. Default is 1.
If this setting is higher than "concurrentConsumers", the listener container
will dynamically schedule new consumers at runtime, provided that enough
incoming messages are encountered. Once the load goes down again, the number of
consumers will be reduced to the standard level ("concurrentConsumers") again.
Raising the number of concurrent consumers is recommendable in order
to scale the consumption of messages coming in from a queue. However,
note that any ordering guarantees are lost once multiple consumers are
registered.
</xsd:documentation>
</xsd:annotation>
</xsd:attribute>
I have a scenario below and is currently leverage Spring integration as the technology to achieve.
I have around 18000 staff Id data
for each staff, a process needs to kick off to do 1 HTTP call to retrieve staff profile information from mail calender server, then 1 HTTP call to retrieve some other information, then may need to send out 3-5 more HTTP calls in a single task
I need to finish this process for above 50000 staff in 15 mins.
I will need this whole batch process to run every 15mins again and again.
Assume each job takes 5 seconds to finish.. i still need 30 mins to finish
=================
Inital Thinking
I can use spring integration to have something like:
- create one job for each staff - 18000 jobs. The job request likely only contains a staff ID so request is very light weight.
- add all the jobs to the int:queue at once so it triggers the input channel - calenderSynRequestChannel
- have a poller - 100 concurrent workers to clean up the job in 15 mins.
Questions:
it is a good way to do this kind of batch processing? some concerns i have is the size of the queue to support 18000 jobs at once
should I use file base approach to store all the staff id in multiple files and get picked up later by the poller? however, this will also complicate the design as there could have concurrent issue for read/write/delete the files by the workers.
Current solution:
<int:service-activator ref="synCalenderService" method="synCalender" input-channel="calenderSynRequestChannel">
<int:poller fixed-delay="50" time-unit="MILLISECONDS" task-executor="taskExecutor" receive-timeout="0" />
</int:service-activator>
<task:executor id="taskExecutor" pool-size="50" keep-alive="120" queue-capacity="500"/>
Anyone encounters similar problem might give a bit of insight on how to address using Spring Integration
Why not do a spring batch job that:
Reader that reads the staff data
Processor that make the HTTP calls
Writer that writes the result to a logfile (for example)
Then utilize the TaskScheduler (spring batch framework) to schedule execution for every 15 minutes, or maybe even better with a fixed delay.
If you want to do it more in parallel, utilize the org.springframework.batch.integration.async.AsyncItemProcessor (and writer).
How to configure Spring sftp:inbound-channel-adapter to run between specific timing lets say 8am-7pm.
Currently i just have the below configuration and i need to poll only between 8am-7pm
<int:poller fixed-rate="300000" max-messages-per-poll="1" />.Heard that Spring batch would help. any suggestions ?
Please, be more specific. Describe your requirements in human words.
E.g. you need to poll every 5 min starting from 8am and ending on 7pm every day.
In this case the cron will look like:
0 0/5 8-19 * *
Iam trying to configure nutch for running multi-threaded crawling.
However , Iam facing an issue. I am not able to run crawl with multiple threads , I have modified the nutch-site.xml to use 25 threads but still I can see only 1 Threads running.
<property>
<name>fetcher.threads.fetch</name>
<value>25</value>
<description>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection).</description>
</property>
<property>
<name>fetcher.threads.per.host</name>
<value>25</value>
<description>This number is the maximum number of threads that
should be allowed to access a host at one time.</description>
</property>
I always get the value of
activeThreads=25, spinWaiting=24, fetchQueues.totalSize=some value.
Whats the meaning of this, can you please explain whats the issue and how can I solve it.
I will highly appreciate your help.
Thanks,
Sumit
I think your issue is related to a known bug w/the new Nutch fetcher. See NUTCH-721.
You can try using OldFetcher (if you have Nutch 1.0) to see if that solves your problem.
-- Ken