I have jdbc:inbound-channel-adapter polling 50 records.
I am trying to improve the performance by scaling pollerExecutor pool-size to 1-10, so that multiple thread can process 50 records each:
<int-jdbc:inbound-channel-adapter
id="initial.ContactType.poller"
query="${poller.ContactType.get}"
max-rows="${poller.deliveryContactType.maxRow:50}"
row-mapper="ContactTypePollerRowMapper"
data-source="dataSource" channel="ContactTypeChannel">
<int:poller fixed-rate="3000" time-unit="MILLISECONDS" task-executor="pollerExecutor">
<int:advice-chain>
<ref bean="pollerLoggingAdvice"/>
<ref bean="txAdvice" />
</int:advice-chain>
</int:poller>
</int-jdbc:inbound-channel-adapter>
<task:executor id="pollerExecutor" pool-size="1-10"
queue-capacity="0" rejection-policy="CALLER_RUNS" />
I tested the time taken to process 100,000 records is same irrespective of the pool-size.
I did three rounds of tests with pool-size=1, pool-size=1-3 and pool-size=1-10 respectively,
in all three tests 100,000 records took 1 hr each time.
I confirmed by checking the logs that pollerExecutor threads are not working parallely.
pollerExecutor-1 process all 50 records before pollerExecutor-2 starts processing.
Why container/pollerExecutor is not working parallely?
I think your problem is here:
/**
* Set the capacity for the ThreadPoolExecutor's BlockingQueue.
* Default is {#code Integer.MAX_VALUE}.
* <p>Any positive value will lead to a LinkedBlockingQueue instance;
* any other value will lead to a SynchronousQueue instance.
* #see java.util.concurrent.LinkedBlockingQueue
* #see java.util.concurrent.SynchronousQueue
*/
public void setQueueCapacity(int queueCapacity) {
So, if you specify queue-capacity="0", then you end up with a SynchronousQueue, which cannot accept new parallel tasks because there is already one busy for processing those 50 records.
Try to have some reasonable queue-capacity to observe possible parallelism.
Related
I'm trying to use FreeRTOS to discuss real time concepts with students, using the POSIX-Linux simulator framework. To accomplish this, I have to find a way of wasting processing time in a controlled way (simulating task in "Running" status by a predetermined period of processing time).
Delays are not good because they change the task status to "Block" and, with a preemptive scheduler, it means the scheduler can give the processor to other tasks. Using Linux native time control approaches (e.g. using clock_gettime() to build the logic) are not good because I don't have the control of the exact running time of a single task, specially with preemptiveness. Regular iterations (for, while) don't have the control I need for processing time (my computer and students computers will have different processing times depending on their architectures).
During my researches on FreeRTOS documentation, I found both the TaskStatus_t struct and the vTaskGetInfo() function which were supposed to help me out. And my problem is when I implement something like:
// Creating the task
if (xTaskCreate(
app_task,
task1_info.name,
configMINIMAL_STACK_SIZE,
(void *) &task1_info,
task1_info.priority,
NULL
) != pdPASS) printf("Task create error %s\n", task1_info.name);
// ...
starting_time_ticks = xTaskGetTickCount(); // starting time in ticks
vTaskStartScheduler();
// ...
static void app_task( void *pvParameters )
// The task itself
{ // ...
for( ;; )
{ // ...
app_proc_ticks(
pdMS_TO_TICKS( task_info.proc_time_ms ),
task_info.name
); // consuming processing time...
// ... }
// ... }
static void app_proc_ticks( TickType_t proc_time_ticks, uint8_t name[APP_MAX_MSG_SIZE])
// Consuming the number of ticks in order to attain a certain processing time
{
TaskHandle_t xHandle;
TaskStatus_t xTaskDetails;
xHandle = xTaskGetHandle( name );
configASSERT( xHandle );
vTaskGetInfo( xHandle, &xTaskDetails, pdTRUE, eInvalid );
TickType_t begin = xTaskDetails.ulRunTimeCounter;
while((xTaskDetails.ulRunTimeCounter - begin) < proc_time_ticks)
{
vTaskGetInfo( xHandle, &xTaskDetails, pdTRUE, eInvalid );
}
}
For a task_info.proc_time_ms equal to 25 ms, my code shows up as the task is consuming around 250 ms worth of ticks, a error factor of 10x. The way I count this is with the following "timestamp" strategy:
static TickType_t get_timestamp_ticks() {
return xTaskGetTickCount() - starting_time_ticks;
}
As far as I can see, I'm having problems to understand and properly convert xTaskDetails.ulRunTimeCounter time unit (ticks, ms, or probably something else). Also probably some tick to ms constant I'm not aware of. Right now, to convert from "ms" to "ticks" I'm using pdMS_TO_TICKS() macro and to convert from "ticks" to "ms" I'm multiplying the number of ticks by portTICK_RATE_MS.
Also, after make, I'm using taskset -c 0 ./build/posix_demo to run and ensure the use of a single processor by my executable.
I'm not trying to hold on to this solution, though. If anyone could share how to do a time controlled delay with "real consumption of processing time" for tasks in FreeRTOS, I would appreciate it as well.
I want to do very basic cron job on Azure Function.
I'm writing functions on Visual Studio and the functions will be dockerized job.
There are two different azure function on my solution.
Runs for every month. (Refreshes whole table).
Runs for every five minutes. (Refhreshes just feature records. Not historical records.)
My expectation is that the functions shouldn't block eachother. So, they shouldn't work at the same time.
My cron expressions for the functions are;
Function1: 0 0 0 1 * * (Runs every month on 1st.)
Function2: 0 */5 * * * * (Runs every five minutes)
I don't want to run function2 on every month like 01/01/2021 00:00:00.
How can I exclude the time from function2?
There is no direct way to do that.
As a workaround, you can add if else code block in Function2. For example, in the if block, you can determine if it's the time like 01/01/2021 00:00:00. if yes, then do nothing. If not, then go to the else block to execute your logic.
Like below:
public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log)
{
var the_time = DateTime.Now;
//if the date is the first day of a month, and time is 00:00:00
if(the_time.Day==1 && the_time.ToLongTimeString().Contains("00:00:00"))
{
//do nothing, don't write any code here.
}
else
{
//write your code logic here
}
}
I am having trouble converting a query parameter in CSV format to a java.util.Set in my Spring Integration inbound gateway.
My outbound gateway adds the parameter like ?ids=[ID_1, ID_2].
My inbound gateway is reading the whole parameter as a single element into a Set like ["ID_1, ID_2"].
I have created my own static helper method to convert the String to a Set but wonder if there is a way to do the conversion implicitly in Spring Integration?
Thanks for your help!
My code is below.
Outbound Gateway:
<int-http:outbound-gateway
url="MY_URL?ids={ids}&foo={foo}"
request-channel="myChannel"
http-method="GET"
message-converters="myConverter"
header-mapper="myMapper"
expected-response-type="MyDto">
<int-http:uri-variable name="ids" expression="payload"/>
<int-http:uri-variable name="foo" expression="headers.foo"/>
</int-http:outbound-gateway>
Inbound Gateway:
<int:service-activator
input-channel="myChannel"
expression="#'myService'.getStuff(payload.ids, headers.foo)"/>
<int-http:inbound-gateway
id="myGateway"
request-channel="myChannel"
path="MY_URL"
message-converters="myConverter"
header-mapper="myMapper"
supported-methods="GET">
<int-http:header name="foo" expression="#requestParams.foo"/>
</int-http:inbound-gateway>
EDIT
This looks like it will solve my problem: https://docs.spring.io/spring-integration/docs/4.3.12.RELEASE/reference/html/messaging-endpoints-chapter.html#payload-type-conversion
Consider to use org.springframework.util.StringUtils.commaDelimitedListToSet():
/**
* Convert a comma delimited list (e.g., a row from a CSV file) into a set.
* <p>Note that this will suppress duplicates, and as of 4.2, the elements in
* the returned set will preserve the original order in a {#link LinkedHashSet}.
* #param str the input {#code String}
* #return a set of {#code String} entries in the list
* #see #removeDuplicateStrings(String[])
*/
public static Set<String> commaDelimitedListToSet(#Nullable String str) {
I am implementing a booking system, and I need to record the concept of the time between the job being accepted and the job starting, and then the duration of the job. I need to move from 'State A' to 'State B' when the start time of the job is reached, and then from 'State B' to 'State C' once the end time of the job has been reached.
I know I could implement this using a queue system like Bull or Bee-Queue, but it seems wrong, as I only need to change the 'State' of my job, not perform any background processing.
How can I simply say "At 3pm, change this Task to 'State B', and at 5pm, change this Task to 'State C'".
Could I poll my DB every minute? Is this the 'correct' approach?
How about using node-cron module? You can use node-cron to schedule jobs based on time, frequency etc. see the official link for more examples
https://www.npmjs.com/package/node-cron
I pasted one for reference
var cron = require('node-cron');
cron.schedule('* * * * *', function(){
console.log('running a task every minute');
});
An approach that doesn't involve scheduling would be to have the value of your state property calculated at request time.
Instead of updating the value of state at exactly 3pm, you check the current time when someone requests the booking resource. If it's >3pm and <5pm return the state as 'State B'. If its >5pm, return state as 'State C'.
You can use setTimeout(function, timeout) to schedule task at certain period of time.
var now = new Date(); //creates date object at current time
var date = new Date(...); //create first state-changing time
//Execute task after (date - now) milliseconds
setTimeout(function(){
//execute tasks
}, date - now);
Now you can do setTimeout() inside setTimeout() with the help of loops to complete your task!
I am using with a row mapper expecting it wuold return me the list of my objects but instead it is returning me LinkedHasMap. I want resultset to be List of my mapped objects.
Any idea how can i achieve this ?
Config
<int-jdbc:stored-proc-inbound-channel-adapter
channel="eventObj"
stored-procedure-name="p_get_completed_data"
data-source="dataSource"
auto-startup="true"
id="completedEventAdpt"
ignore-column-meta-data="true"
is-function="false"
return-value-required="false">
<int:poller max-messages-per-poll="1" fixed-rate="180000"/>
<int-jdbc:returning-resultset name="mapper" row-mapper="mapper" />
</int-jdbc:stored-proc-inbound-channel-adapter>
<bean id="mapper" class="com.db.mapper.MyMapper"/>
Stored procedures can return multiple result sets, keyed by the parameter name.
In your case, you only have one result set, but it's still returned in a map.
Simply add a <transformer/> after the inbound adapter...
<transformer ... expression="payload['mapper']" />