Imagine you write application, alternative to some existing version and you want to compare if it's more effective or not,
you can simply use time like
time yourcommand
time oldcommand
and compare the execution time to check some difference, but this isn't very detailed
Is there similar command to check more data? Such as memory usage, cpu utilization, cpu peak, memory peak etc...
A good implementation of time actually tells you a lot more than wallclock time. Most Linux systems have one, but Bash tends to obscure it in favor of its built-in time, so you have to call it as /usr/bin/time:
$ /usr/bin/time python -c "import numpy as np; np.empty(100000)"
0.12user 0.00system 0:00.13elapsed 96%CPU (0avgtext+0avgdata 12860maxresident)k
0inputs+0outputs (0major+3777minor)pagefaults 0swaps
That's CPU use, memory usage and several other statistics for a simple Python command. See the manpage time(1) for what time can do.
There is no single best way to do what you're talking about, as it depends a lot on your application, as well as what you wish to profile.
But this post offers some suggestions on ways to profile Linux or a specific application, which may help you along the right direction.
You will likely find better answers if you can tell us more specifically what you're hoping to profile, which language(s) you're using, etc.
Related
How do I figure out how much walltime (mem? vmem?) a PBS job (PBS Pro) actually ended up using, if it's not presented in the stodut/sterr logs?
In Torque, this information is visible in the accounting log and in the qstat -f output for the job. In qstat -f, you wanted to look at the resources_used information.
This may have diverged somewhat in PBS Pro, but my guess is they have something similar.
Wall time is always measured outside of the system. That's why it refers to the "clock on the wall".
This is important because it often encompasses elements that some systems fail to measure, or measure poorly. To illustrate, before a system can capture the time, some code must run to allocate the memory to capture the time, and then some code must run to assign that memory. Everything before that happens is misreported to not have "cost" any time at all.
While I may have described the essence of wall time, do look to dbeer's excellent answer for capturing a time close to wall clock time (and hopefully solving your metric gathering problem).
Background
I've written a tool to capture CPU usage on a per/thread basis. The output of the tools is a binary file, that I can pump into my parsing utility that I wrote. And the output of the parsing utility is a CSV file that I can import into Excel to chart pretty graphs of process/thread CPU usage.
This CPU usage capture tool is running on an embedded ARM platform running a Linux kernel based on 2.6.35.3. That being said, I was concerned about making the tool light weight. I didn't want it to store directly to a CSV file, in order to minimize the processing time and the file size of the captured data.
Question
The tool works, but I'm wondering if I took the long way around the problem? Is there already a tool out there that does this (or something like it)?
You're probably wondering why I care if I already made a tool that works. Well, it's not as light weight as I'd like. It's taking up about 10% of CPU usage. As a benchmark, top only takes up about 1% (max).
Update
I've decided to continue using my tool for now. At least until a better solution becomes available. I was able to shave off a couple percentage points by using open() instead of fopen() on /proc/stat. I'm also using read() instead of fgets().
IBM has a tool called nmon which does the same(for AIX & Linux): According to IBM's documentation, it takes ~2% CPU. You may want to look at that.
Comparing nmon with your tool could give you a fair idea about your program's performance and how you may improve your csv capture.
This might be a bit of a steep learning curve, but you might want look into SystemTap: http://sourceware.org/systemtap/
Is there a way to benchmark a bash script's performance? the script downloads a remote file, and then makes calls to multiple commandline programs to manipulate. I would like to know (or as much as possible):
Total time
Time spent downloading
Time spent on each command called
-=[ I think these could be wrapped in "time" calls right? ]=-
Average download speed
uses wget
Total Memory used
Total CPU usage
CPU usage per command called
I'm able to make edits to the bash script to insert any benchmark commands needed at specific points (ie, between app calls). Not sure if some "top" ninja-ry could solve this or not. Not able to find anything useful (at least to limited understanding) in man file.
Will be running the benchmarks on OSX Terminal as well as Ubuntu (if either matter).
strace -o trace -c -Ttt ./scrip
-c is to trace the time spent by cpu on specific call.
-Ttt will tell you time in microseconds at time of each system call running.
-o will save output in file "trace".
You should be able to achieve this a number of ways. One way is to use time built-in function for each command of interest and capture the results. You may have to be careful about any pipes and redirects;
You may also consider trapping SIGCHLD, DEBUG, RETURN, ERR and EXIT signals and putting timing information in there, but you may not get some results.
This concept of CPU usage of each command won't give you any thing useful, all commands use 100% of cpu. Memory usage is something you can pull out but you should look at
If you want to get deep process statistics then you would want to use strace... See strace(1) man page for details. I doubt that -Ttt as it is suggest elsewhere is useful all that tells you are system call times and you want other process trace info.
You may also want to see ltrace and dstat tools.
A similar question is answered here Linux benchmarking tools
I have a program in which significant amount of time is spent loading and saving data. Now I want to know how much time each function is taking in terms of percentage of the total running time. However, I want to exclude the time taken by loading and saving functions from the total time considered by the profiler. Is there any way to do so using gprof or any other popular profiler?
Similarly you can use
valgrind --tool=callgrind --collect-atstart=no --toggle-collect=<function>
Other options to look at:
--instr-atstart # to avoid runtime overhead while not profiling
To get instructionlevel stats:
--collect-jumps=yes
--dump-instr=yes
Alternatively you can 'remote control' it on the fly: callgrind_control or annotate your source code (IIRC also with branch predictions stats): callgrind_annotate.
The excellent tool kcachegrind is a marvellous visualization/navigation tool. I can hardly recommend it enough:
I would consider using something more modern than gprof, such as OProfile. When generating a report using opreport you can use the --exclude-symbols option to exclude functions you are not interested in.
See the OProfile webpage for more details; however for a quick start guide see the OProfile docs page.
Zoom from RotateRight offers a system-wide time profile for Linux. If your code spends a lot of time in i/o, then that time won't show up in a time profile of the CPUs. Alternatively, if you want to account for time spent in i/o, try the "thread time profile".
for a simple, basic solution, you might want log data to a csv file.
e.g. Format [functionKey,timeStamp\n]
... then load that up in Excel. Get the deltas, and then include or exclude based on if functions. Nothing fancy. On the upside, you could get some visualisations fairly cheaply.
I'm trying to find the best way to use 'top' as semi-permanent instrumentation in the development of a box running embedded Linux. (The instrumentation will be removed from the final-test and production releases.)
My first pass is to simply add this to init.d:
top -b -d 15 >/tmp/toploop.out &
This runs top in "batch" mode every 15 seconds. Let's assume that /tmp has plenty of space…
Questions:
Is 15 seconds a good value to choose for general-purpose monitoring?
Other than disk space, how seriously is this perturbing the state of the system?
What other (perhaps better) tools could be used like this?
Look at collectd. It's a very light weight system monitoring framework coded for performance.
We use sysstat to monitor things like this.
You might find that vmstat and iostat with a delay and no repeat counter is a better option.
I suspect 15 seconds would be more than adequate unless you actually want to watch what's happening in real time, but that doesn't appear to be the case here.
As far as load, on an idling PIII 900Mhz w/ 768MB of RAM running Ubuntu (not sure which version, but not more than a year old) I have top updating every 0.5 seconds and it's about 2% CPU utilization. At 15s updates, I'm seeing 0.1% CPU utilization.
depending upon what exactly you want, you could use the output of uptime, free, and ps to get most, if not all, of top's information.
If you are looking for overall load, uptime is probably sufficient. However, if you want specific information about processes, you are adventurous, and have the /proc filessystem enabled, you may want to write your own tools. The primary benefit in this environment is that you can focus on exactly what you want and minimize the load introduced to the system.
The proc file system gives your application read access to the kernel memory that keeps track of many of the interesting variables. Reading from /proc is one of the lightest ways to get this information. Additionally, you may be able to get more information than provided by top. I've done this in the past to get amount of time spent in user and system by this process. Additionally, you can use this to get information about the number of file descriptors open by the process. You might also use this to get detailed information about how the network system is working.
Much of this information is pre-processed by other applications which can be used if you get the information you need. However, it is rather straight-forward to read the raw information. Do a man proc for more information.
Pity you haven't said what you are monitoring for.
You should decide whether 15 seconds is ok or not. Feel free to drop it way lower if you wish (and have a fast HDD)
No worries unless you are running a soft real-time system
Have a look at tools suggested in other answers. I'll add another sugestion: "iotop", for answering a "who is thrashing the HDD" questions.
At work for system monitoring during stress tests we use a tool called nmon.
What I love about nmon is it has the ability to export to XLS and generate beautiful graphs for you.
It generates statistics for:
Memory Usage
CPU Usage
Network Usage
Disk I/O
Good luck :)