Using the import to models option, I am importing an XLS file with about 15,000 rows.
With the microtime_float function, the script times and echos out how long it takes. At 29.6 secs, this happens, showing it took less than 30 seconds. At that time, I can see the database has all 15k+ records as expected, no issues there.
Problem is, the browser is kept busy and at 1 min 22 secs, 1 min 55 secs and 2 min 26 secs it prompts me to either wait or kill the process. I keep clicking wait and finally it ends at 2 mins 49 secs.
This is a terrible user experience, how can I cut off this extra wait time?
It's a very basic setup: the route calls importcontroller#import with http get and the code is as follows:
public function import()
{
ini_set('memory_limit', '1024M');
$start = $this->microtime_float();
Excel::import(new myImport, 'myfile.xls' , null, \Maatwebsite\Excel\Excel::XLS);
$end = $this->microtime_float();
$t = $end - $start;
return "Time: $t";
}
The class uses certain concerns as follows:
class myImport implements ToModel, WithBatchInserts, WithChunkReading, WithStartRow
Related
For LivyOperator we set the following parameters:
polling_interval=60
retries_num_timeout=100
We set it up according to this documentation: https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/stable/_api/airflow/providers/apache/livy/operators/livy/index.html
But, in this configuration after 100 * 60 seconds = 6000 seconds = 1 hour 40 minutes Livy-session is interrupted, operator becomes failed, loading is interrupted. Is there any way to resove such inconsistency on Airflow/Livy side?
I need a way to execute my python script every 15 mins starting from 9:30:42 till 15:00:42 every day from Mon-Fri.
I have explored APScheduler with cron syntax but can't figure out how to code the above condition. I tried below but doesn't work (execute is my function name)
sched.add_cron_job(execute, day_of_week='mon-fri', hour='9:30:42-15:00:42', minute='*/15')
Any pointer is deeply appreciated.
As far as I can tell you won't be able to do what you want with a single job.
This is the closest I could get with one:30-59/15 9-14 * * 1-5 which equates to Every 15 minutes, minutes 30 through 59 past the hour, between 09:00 AM and 02:59 PM, Monday through Friday.
Although it isn't exactly what you wanted I hope this helps as a base.
I wrote custom code to solve my problem. Posting here in case it helps someone. Any optimisations suggestions are welcome.
The first infinite loop starts the job when the start time is hit. The 2nd infinite loop wakes up every x minutes to check if next run time has approached. If yes, it executes else goes back to sleep. If the end time for execution has reached, then it breaks out
def execute_schedule_custom():
start_time_of_day = datetime.combine(date.today(), time(9, 30, 42))
next_run_time = start_time_of_day
end_time_of_day = datetime.combine(date.today(), time(15, 0, 42))
interval = 15
sleep_secs = 60 * 5 #sleep for 5 mins
while True:
if datetime.now() >= start_time_of_day:
execute()
next_run_time = start_time_of_day + timedelta(minutes=interval)
break
while True:
if datetime.now() >= end_time_of_day:
break
elif datetime.now() >= next_run_time:
execute()
next_run_time = next_run_time + timedelta(minutes=interval)
t.sleep(sleep_secs)
I have the following scenario to be load tested for a service and it does not seem to work as expected. My scenario is as follows.
Test with rampUsers(100) over 15 minutes duration
Hold the users for about 10 minutes holdFor(10 minutes)
Then again rampUsers(200) over 15 minutes duration
Hold the users for about 10 minutes holdFor(10 minutes)
Then again rampUsers(200) over 15 minutes duration
I am trying to use throttle option for this but it does not seem to work as expected
here is my code snippets combinations that I have tried so far
//NUM_USERS = 300
//DURATION = 15 minutes
//CONSTANT_DURATION = 5 minutes
// Tried with different combinations of NUM_USERS and DURATION but not helpful
scn.inject(
rampUsers(NUM_USERS*1) during DURATION,
constantUsersPerSec(1) during CONSTANT_DURATION,
rampUsers(NUM_USERS*2) during DURATION,
constantUsersPerSec(2) during CONSTANT_DURATION,
rampUsers(NUM_USERS*3) during DURATION,
constantUsersPerSec(3) during CONSTANT_DURATION
)
scn.inject(
rampUsers(NUM_USERS) during DURATION
).throttle(
reachRps(NUM_USERS/4) in (CONSTANT_DURATION),
holdFor(CONSTANT_DURATION),
jumpToRps(NUM_USERS/3),
holdFor(CONSTANT_DURATION),
jumpToRps(NUM_USERS/2),
holdFor(CONSTANT_DURATION)
)
scn.inject(
rampUsers(NUM_USERS) during DURATION
).throttle(
holdFor(CONSTANT_DURATION),
reachRps(NUM_USERS+NUM_USERS) in (DURATION+DURATION),
holdFor(CONSTANT_DURATION)
)
Can any one help on this which one works in this case. I would like to have a graph like this
To target injection rate as you stated you want in the comments, you need something like this
scn.inject(
rampUsersPerSec(0) to (300) during DURATION,
constantUsersPerSec(300) during CONSTANT_DURATION,
rampUsersPerSec(300) to (600) during DURATION,
constantUsersPerSec(600) during CONSTANT_DURATION,
...
)
I am having issues using rampUser() method in my gatling script. The request is getting stuck after the following entry which had passed half way through.
Version : 3.3
================================================================================
2019-12-18 09:51:44 45s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=2 KO=0 )
> graphql / request_0 (OK=1 KO=0 )
> rest / request_0 (OK=1 KO=0 )
---- xxxSimulation ---------------------------------------------------
[##################################### ] 50%
waiting: 1 / active: 0 / done: 1
================================================================================
I am seeing the following in the log which gets repeated for ever and the log size increases
09:35:46.495 [GatlingSystem-akka.actor.default-dispatcher-2] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
09:35:47.494 [GatlingSystem-akka.actor.default-dispatcher-6] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
The above issue is happening only with rampUser and not happening with
atOnceUsers()
rampUsersPerSec()
rampConcurrentUsers()
constantConcurrentUsers()
constantUsersPerSec()
incrementUsersPerSec()
Is there a way to mimic rampUser() in some other way or is there a solution for this.
My code is very minimal
setUp(
scenarioBuilder.inject(
rampUsers(2).during(1 minutes)
)
).protocols(protocolBuilder)
I am stuck with this for some time and my earlier post with more information can be found here
Can any of the gatling experts help me on this?
Thanks for looking into it.
It seems you have slightly incorrect syntax for a rampUsers. You should try remove a . before during.
I have in my own script this code and it works fine:
setUp(userScenario.inject(
// atOnceUsers(4),
rampUsers(24) during (1 seconds))
).protocols(httpProtocol)
Also, in Gatling documentation example is also without a dot Open model:
scn.inject(
nothingFor(4 seconds), // 1
atOnceUsers(10), // 2
rampUsers(10) during (5 seconds), // HERE
constantUsersPerSec(20) during (15 seconds), // 4
constantUsersPerSec(20) during (15 seconds) randomized, // 5
rampUsersPerSec(10) to 20 during (10 minutes), // 6
rampUsersPerSec(10) to 20 during (10 minutes) randomized, // 7
heavisideUsers(1000) during (20 seconds) // 8
).protocols(httpProtocol)
)
My guess is that syntax can't be parsed, so instead 0 is substituted. (Here is example of rounding. Not applicable, but as reference: gatling-user-injection-constantuserspersec)
Also, you mentioned that others method work, could you paste working code as well?
cross-post: http://perlmonks.org/?node_id=1191821
Consider app.psgi:
#!perl
use 5.024;
use strictures;
use Time::HiRes qw(sleep);
sub mock_connect {
my $how_long_it_takes = 3 + rand;
sleep $how_long_it_takes;
return $how_long_it_takes;
}
sub main {
state $db_handle = mock_connect($dsn);
return sub { [200, [], ["connect took $db_handle seconds\n"]] };
}
my $dsn = 'dbi:blahblah'; # from config file
my $app = main($dsn);
Measuring plackup (HTTP::Server::PSGI: Accepting connections at http://0:5000/):
› perl -MBenchmark=timeit,timestr,:hireswallclock -E"say timestr timeit 10, sub { system q(curl http://localhost:5000) }"
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
connect took 3.0299610154043 seconds
2.93921 wallclock secs ( 0.03 usr + 0.06 sys = 0.09 CPU) # 107.53/s (n=10)
Measuring thrall (Starting Thrall/0.0305 (MSWin32) http server listening at port 5000):
› perl -MBenchmark=timeit,timestr,:hireswallclock -E"say timestr timeit 10, sub { system q(curl http://localhost:5000) }"
connect took 3.77111188120125 seconds
connect took 3.15455510265111 seconds
connect took 3.77111188120125 seconds
connect took 3.15455510265111 seconds
connect took 3.77111188120125 seconds
connect took 3.64333342488772 seconds
connect took 3.15455510265111 seconds
connect took 3.77111188120125 seconds
connect took 3.85268922343767 seconds
connect took 3.64333342488772 seconds
17.4764 wallclock secs ( 0.02 usr + 0.09 sys = 0.11 CPU) # 90.91/s (n=10)
This performance is not acceptable because the initialisation happens several times, despite the state variable. How do you make it so it happens only once?
For whatever reason, the program thrall hard-coded a "loader" parameter in its configuration section:
my $runner = Plack::Runner->new(
server => 'Thrall',
env => 'deployment',
loader => 'Delayed',
version_cb => \&version,
);
$runner->parse_options(#ARGV);
That string "Delayed" refers to the module Plack::Loader::Delayed, which delays the loading of .psgi files until first request comes. That would match your benchmarking result. (If you re-run the benchmark again without killing thrall, you'll see identical output).
You may try running thrall -L +Plack::Loader app.psgi, which reverts the "loader" parameter to the default value hard-coded in Plack::Runner.
Isn't this what the --preload-app option to Starman does?