I'm running a Java program and I need to get how much time each it spent garbage collecting.
I found these 2 JVM flags:
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime
but I'm not being able to find information about it.
I suppose that PrintGCApplicationStoppedTime prints for how long the application time was in a STW, however I am not sure about -XX:+PrintGCApplicationConcurrentTime. Does it print for how long the application was executing concurrently with collection threads?
Thanks in advance.
The names of these flags are not very accurate.
PrintGCApplicationStoppedTime shows how much time the application was stopped at safepoint. Most often safepoints are caused by stop-the-world phases of garbage collection, however, many other reasons exist.
PrintGCApplicationConcurrentTime is how much time the application worked without stopping, i.e. the time between two successive safepoints.
You using those flags to know how long your application
runs between garbage collections.
Eventually that can be calculated from the
GC logs but a convenient way to see that information is with
the command line flags -XX:+PrintGCApplicationStoppedTime and
-XX:+PrintGCApplicationConcurrentTime.
Adding these to your command line produces output such as this:
Application time: 0.2863875 seconds
Total time for which application threads were stopped: 0.0225087 seconds
Application time: 0.1476791 seconds
Total time for which application threads were stopped: 0.0255697 seconds
The application ran (reported in the first line) for about 287 milliseconds and then was
stopped for about 22 milliseconds (reported in the second line).
The flags can be used separately or together.
Related
I configured Java based Selenium WebDriver test in Apache JMeter with the following setup:
Number of Threads (Users): 10
Ramp-up period (Second): 120
Loop Count: 1
I ticked the Delay Thread Creation until needed to save resources.
My expectation regarding the functionality:
I expected that if I have 10 users with 120 seconds ramp up time, then every user activity will start each other and the Jmeter will wait at least 12 seconds to start the next thread.
The issue is:
The threads start sometimes within 11 seconds, sometimes 12 seconds.
I don't know why does it happen because I would like to see the threads start after each other exactly in 12 seconds.
The question is
Are there any solution that to tell the JMeter to wait exactly 12 seconds for next thread start?
Here is the picture about started jobs with date time stamp:
I don't think you will be able to achieve this level of precision using ramp-up period approach of the normal Thread Group, a better idea would be going for the Ultimate Thread Group (can be installed using JMeter Plugins Manager) which allows absolute flexibility in terms of definition of ramp-up, ramp-down and time to hold the load.
Example setup:
Example output:
In order to get only one execution of the "job" per each virtual user you can use Throughput Controller configured like:
You can add Flow Control Action for pausing exact time
it allows pauses to be included without needing to generate a sample. For variable delays, set the pause time to zero, and add a Timer as a child.
I'm using JMeter 3.1.1 to run a load test. My test plan is with 40 threads and each thread executes 6 HTTP Requests. It is running fine for the first few hours with a latency of around 20ms.
After few hours, latency grows up to 500ms. I verified that server is processing fine. Also, I have no 'Listeners' in my test plan and I run it in NonUI mode.
Also it seems that the thread group is executing only one thread at time. Coz I see hardly one or two requests being executed by thread group per second.
Im really clueless what to suspect. Any help would be greatly appreciated.
BTW., memory and CPU consumption are normal.
About my TestPlan:
Total Thread Groups:4
1. Setup Thread Group
2. Load test thread group with 40 threads
(Action To be taken after error :Continue
Ramp-Up period: 0
Number of Threads: 40
Loop Count: Forever)
2.1 Counter
2.2 Random Variable
2.3 User Defined Variables
2.4 If Condition = true
- 2.4.1 HTTP Request1
- 2.4.2 HTTP Request2
- 2.4.2 Loop for 5 times
-- 2.4.2.1 HTTP Request1
-- 2.4.2.2 HTTP Request2
3. Introspection thread group with 1 thread
4. Tear Down thread group
Please let me know, if more details are needed
Another observation is:
Server is having TIME_WAITs : 4418 (I check 'Use keep-alive' option for HTTP Request., still so many TIME_WAITs)
Latest Observations(Thanks to one and all for your valuable comments)
Actually, the memory must be an issue., I have given already like this.,
-Xms512m -Xmx2048m -XX:NewSize=512m -XX:MaxNewSize=2048m
But I really wonder why JVM was not going 512 MB. So i tried with both Xms ans Xmx with 2g each. Now its kind of running for more longer. However it' performance is slowing down still. May be my Beanshell Post Processors are consuming all the memory. I really wonder why they are not releasing the memory. If you see., per hour how the performance is degraded.
Hour #Requests sent
---- --------------
Hour 1: 1471917
Hour 2: 1084182 (Seems all 2g heap is used up by this time)
Hour 3: 705471
Hour 4: 442912
Hour 5: 255826
Hour 6: 136292
I read that Beanshell hogs memory, but I have no choice but to use it as I have to use a third party jar with in the sampler to make few java calls. I'm not sure if I can do the same using JSR223(Groovy) or any other better performing sampler (pre / post processor)
as you have seen the Heap settings I did., here are the Memory and CPU Utilization of Jmeter. Im running 100 threads now. What should I do in my test plan to reduce the CPU utilization. I have a 100ms sleep after every 4 HTTP Requests.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11850 xxx 20 0 7776m 2.1g 4744 S 678.2 27.4 6421:50 java
%CPU: 678.2 (Fluctuating between 99% - 700%)
MEM: 2.1g (Xmx = 2g)
1) Do you run the JMeter with standard script (jmeter/jmeter.bat)?
Then mind that the default size for JVM heap in there is capped at 512M. Consider increasing it, at least at the maximum end (means, change default -Xmx512m).
Next thing to consider is the -XX:NewSize=128m -XX:MaxNewSize=128m default values.
Here's what Oracle suggesting:
In heavy throughput environments, you should consider using this
option to increase the size of the JVM young generation. By default,
the young generation is quite small, and high throughput scenarios can
result in a large amount of generated garbage. This garbage
collection, in turn, causes the JVM to inadvertently promote
short-lived objects into the old generation.
So try to play with this parameters, that may help.
P.S. Aren't you, by chance, running it at AWS EC2 instance? If yes - what's the instance type?
Thanks to all, who all tried to help me.
However., I could resolve this.,
Culprit is : If Controller, which is evaluating the condition for every iteration of every thread. Sounds quite normal isn't it? The problem is., condition is evaluation is JavaScript based. So all threads are eating CPU and Memory for invoking JavaScript.
Now Im getting consistent requests to server and JMeter is also almost stable at 1.9g of memory for 100 threads.
Im posting this just in case any one can get benefited without wasting day and nights figuring out the issue :)
I want to run a small clean up process every few hours on an Erlang server.
I know of the timer module. I saw an example in a tutorial used chained timer:sleep commands to wait for an event that would occur multiple days later, which I found strange. I understand that Erlang process are unique compared to those in other languages, but the idea of a process/thread sleeping for days, weeks, and even months at a time seemed odd.
So I set out to find out the details of what sleeping actually does. The closest I found was a blog post mentioning that sleep is implemented with a receive timeout, but that still left the question:
What do these sleep/sleep-like functions actually do?
Is my process taking up resources as it sleeps? Would having thousands of sleeping process use as many resources, as say, thousands of process servicing a recursive call that did nothing? Is there any performance penalty from repeatedly sleeping within processes, or sleeping for long periods of time? Is the VM constantly expending resources to see if the conditions to end the processes' sleep are up?
And as a side note, I'd appreciate if someone could comment on if there is a better way than sleeping to pause for hours or days at a time?
That is the Karma of any erlang process: it waits or dies :o)
when a process is spawned, it start executing until the last execution line, and die, returning the last evaluation.
To keep a process alive, there is no other solution to recursively loop in a never ending succession of calls.
of course there are several conditions that make it stop or sleep:
end of the loop: the process received a message which tell him to
stop recursion
a receive bloc: the process will wait until a message
matching one entry in the receive bloc is posted in the message
queue.
The VM scheduler stop it temporarily to let access to the CPU
to other processes
in the 2 last cases the execution will restart under the responsibility of the VM scheduler.
while waiting it uses no CPU bandwidth, but keeps the exact same memory layout it had when it started waiting. The Erlang OTP offers some means to reduce this memory layout to the minimum using the hibernate option (see the documentation of gen_serevr or gen_fsm, but it is for advanced usage only in my mind).
a simple way to create a "signal" that will fire a process at regular (or almost regular) interval is effectively to use receive block with timout (The timeout is limited to 65535 ms), for example:
on_tick_sec(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,1000,Period,0).
on_tick_mn(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,60000,Period,0).
on_tick_hr(Module,Function,Arglist,Period) ->
on_tick(Module,Function,Arglist,60000,Period*60,0).
on_tick(Module,Function,Arglist,TimeBase,Period,Period) ->
apply(Module,Function,Arglist),
on_tick(Module,Function,Arglist,TimeBase,Period,0);
on_tick(Module,Function,Arglist,TimeBase,Period,CountTimeBase) ->
receive
stop -> stopped
after TimeBase ->
on_tick(Module,Function,Arglist,TimeBase,Period,CountTimeBase+1)
end.
and usage:
1> Pid = spawn(util,on_tick_sec,[io,format,["hello~n"],5]).
<0.40.0>
hello
hello
hello
hello
2> Pid ! stop.
stop
3>
[edit]
The timer module is a standard gen_server running in a separate process. All the function in the timer module are public interfaces that execute a hidden gen_server:call or gen_server:cast to the timer server. This is a common usage to hide the internal of a server and allow further evolutions without impact on existing applications.
The server uses internally a table (ets) to store all the actions it has to do along with each timer reference and it uses its own function to be awaken when needed (at the end, the VM must take care of this ?).
So you can hibernate a process without any effect on the timer server behavior. The hibernation mechanism is
tricky, see documentation at hibernate/3 definition, you will see that yo have to "rebuild" the context by yourself since everything was removed from the process context, and a tuple(Module,Function,Arguments} is stored by the system to restart your process when needed.
cost some time in garbage collecting and process restart
It is why I said that it is really an advance feature that need good reason to be used.
There is also erlang:hibernate/3 that puts a process in "deep sleep", minimizing memory usage for it.
In my code i run a cron job which is run for every five seconds, and I've been getting the same WARNING ever since.
this is the api that i used:
sched.add_cron_job(test_3, second="*/5")
And I get a warning:
WARNING:apscheduler.scheduler:Execution of job "test_3 (trigger: cron[second='*/5'], next run at: 2013-11-28 15:56:30)" skipped: maximum number of running instances reached (1)
I tried giving time gap of 2 minutes it doesn't solve the issue.....
Help me in overcoming this issue..
I used the proc.terminate() to stop the execution of my method. So that the instance of the 1st thread is terminated before a new thread could start again.
Also provide a timing mechanism to complete your process well within the scheduled time say within a minute, hour or day etc. In my application i used *sleep(in_seconds)* for providing the timing mechanism.
I had a similar problem, and it turned out it was just your job 'test_3' lasting too long, more then 5 secs (or 2 minutes as you tried).
APScheduler is trying to re-execute you job, but the previous one is still running.
Running a single script with only two users as a single scenario without any pacing, just think time set to 3 seconds and random (50%-150%) I experience that the web app server runs of of memory after 10 minutes every time (I have run the test several times, and it happens at the same time every time).
First I thouhgt this was a memory leak in the application, but after some thought I figured it might have to do with the scenario design.
The entire script having just one action including log in and log out within the only action block takes about 50 seconds to run and I have the default as soon as the previous iteration ends set not the with delay after the previous iteration ends or fixed/random intervalls set.
Could not using fixed/random intervalls cause this "memory leak" to happen? I guess non of the settings mentioned would actually start a new iteration before the one before ends, this obvioulsy leading to accumulation of memory on the server resulting in this "memory leak". But with no pacing set is there a risk for this to happen?
And having no iterations in my script, could I still be using pacing?
To answer your last question: NO.
Pacing is explicitly used when a new iteration starts. The iteration start is delayed according to pacing settings.
Speculation/Conclusions:
If the web-server really runs out of memory after 10 minutes, and you only have 2 vu's, you have a problem on the web-server side. One could manually achieve this 2vu load and crash the web-server. The pacing in the scripts, or manual user speeds are irrelevant. If the web-server can be crashed from remote, it has bugs that need fixing.
Suggestion:
Try running the scenario with 4 users. Do you get OUT OF MEMORY on the web-server after 5 mins?
If there really is a leak, your script/scenario shouldn't be causing it, but I would think that you could potentially cause it to appear to be a problem sooner depending on how you run it.
For example, let's say with 5 users and reasonable pacing and think times, the server doesn't die for 16 hours. But with 50 users it dies in 2 hours. You haven't caused the problem, just exposed it sooner.
i hope its web server problem.pacing is nothing but a time gap in between iterations,it's not effect actions or transactions in your script