I am new to gatling, I am trying to parallely execute 2 scenarios, but I want 2nd scenario to be executed after some time. Is there a way that I can implement this? I tried something as below but didn't work.
val scn1 = scenario("file upload").exec(getFileUpload())
val scn2 = scenario("get datasetId").exec(getDataSetId()).pause(4)
setUp(scn1.inject(atOnceUsers(1)),scn2.inject(atOnceUsers(1))).protocols(httpConf)
Any suggestions ?
Use nothingFor to delay the injection profile of scn2, see doc.
Related
I use spring batch to load data from a file to a database.The job contains only one step.I use a ThreadPoolTaskExecutor to execute step concurrently.The step is similar to this one.
public Step MyStep(){
return StepBuilderFactory.get("MyStep")
.chunk(10000)
.reader(flatFileItemWriter)
.writer(jdbcBatchItemWriter)
.faultTolerant()
.skip(NumberFormatException.class)
.skip(FlatFileParseException.class)
.skipLimit(3)
.throttleLImit(10)
.taskExecutor(taskExecutor)
.build();
}
There are 3 "numberformat" errors in my file,so I set skipLimit 3,but I find that when I execute the job,it will start 10 threads and each thread has 3 skips,so I have 3 * 10 = 30 skips in total,while I only need 3.
So the question is will this cause any problems?And is there any other way to skip exactly 3 times while executing a step concurrently?
github issue
Robert Kasanicky opened BATCH-926 and commented
When chunks execute concurrently each chunk works with its own local copy of skipCount (from StepContribution). Given skipLimit=1 and 10 chunks execute concurrenlty we can end up with successful job execution and 10 skips. As the number of concurrent chunks increases the skipLimit becomes more a limit per chunk rather than job.
Dave Syer commented
I vote for documenting the current behaviour.
Robert Kasanicky commented
documented current behavior in the factory bean
However, this seems to be a correct thing for a very old version of spring batch.
I have a different problem in my code, but the skipLimit seems to be aggregated when I use multiple threads. Albeit the job sometimes hangs without properly failing when SkipLimitException is thrown.
I am planning for a nodeJS Application where the server has to handle time-based jobs. For example:
Job 1: Every 2 hours
Job 2: Every 5 minutes
Job 3: Every 10 days
...
These jobs are saved in a database and there will be a lot of jobs - it could be a millions of these.
So now is my question which is the right way in nodeJS to handle this ? There are a lot cron packages for cron like processes but I thought its maybe "to much" for this.
I was thinking about .... Timeouts! Right - timeouts.
This is the schema I was thinking of:
var job = getJob();
var executeInMs = getMSUntilExecute(job);
setTimeout(function(){handle();}, executeInMs);
With this schema it really could be that there are a million timeouts running at the same time in my application.
The question now is: Is this a Server-Killer or is this the right way to handle this ?
I have multiple Spring batch jobs that I am required to schedule such that they are triggered at the same time.So far I am not having luck around this and I also do not under stand why.I read some where that having multiple jobs to trigger on same CRON expression(e.g. after every 10 mins) is not possible with Quartz and I have to increase the SIZE of the thread pool some how to make it possible.I just want to understand is this correct?If yes then is there an alternate way of achieving this sort of concurrency and if not then how can I increase the thread pool size in the xml configuration?Many Thanks in advance
I am working on an iterative algorithm using Apache Spark, which claims to be perfect for just that. The examples I have found so far creates a single job with a hardcoded number of iterations. I need the algorithm to run until a certain condition is met.
My current implementation launches a new job for each iteration something like this:
var data = sc.textFile(...).map().cache()
while(data.filter(...).isEmpty()) {
// Run the Algorithm (also handles caching)
val data = performStep(data)
}
This is pretty inefficient. Between each iteration I wait a long time for the next job to start. For four servers I wait around 10 seconds in between each job, for 32 servers is almost 100 seconds. In total I end up spending at least half of the runtime waiting in between jobs.
I find conditional iterations quite common in certain types of algorithms, for example early stopping criteria in machine learning. So I am hoping this can be improved.
Is there a more efficient way of doing this? For example away to run this conditional repetition in a single job? Thanks!
We have implemented spring batch and which is handling large data set and because of that it is taking too much time to finish it ( around 2 to 4 hours). We have ItemReade, ItemProcesser and ItemWriter.
We are trying to improve performance , code is fine so we are planning to implement multithreading.
Can anybody give suggestion , how to implement multithreading in Spring Batch
I think we should also take care of member variables which is inject as AutoWired.
Please give your suggestion
We have used Executor to achieve this , we did following things,
Find out the code or portion of the code which took more time to execute.
Did analysis that can we use multithreading.
Use Executor with our oven framework which make sure that threads are returning in sequence manner how it is inserted.
Convert all Member variables to local variables which are having dependency to execute batch as that can cause multithreading problem.
We had able to reduce total time from 12 hours to 4 hours of all the spring batch job.
H