Linux has ioctl SIOCOUTQ described in man-page tcp(7) that returns amount of unsent data in socket buffers. If I understand kernel code right, all the non-ACKed data is counted as "unsent". The ioctl is available at least since 2.4.x.
Is there anything alike for {Free,Net,Open,*}BSD, Solaris, Windows?
There are (at least) two different pieces of information you might want: the amount of data that hasn't been sent yet, and the amount of data that's been sent-but-not-ACK-ed.
On Linux: SIOCOUTQ is documented to give the amount of unsent data, but actually gives the sum of (unsent data + sent-but-not-ACK-ed data). A recent patch (Feb 2016) made it possible to get the actual unsent data from the tcpi_notsent_bytes field in the TCP_INFO struct.
On macOS and iOS: getsockopt(fd, SOL_SOCKET, SO_NWRITE, ...) is just like SIOCOUTQ: it's documented to give the amount of unsent data, but actually gives the sum of (unsent data + sent-but-not-ACK-ed data). I don't know any way to get more fine-grained information.
On Windows: GetPerTcpConnectionEStats with the TcpConnectionEstatsSendBuff option gives you both unsent data and sent-but-not-ACK-ed data as two separate numbers.
I don't know how to get this information on other operating systems.
Since TCP/IP is implemented as a stream device, it might be possible to take a kernel dive and get the queue->q_count (number of bytes on the queue).
These flags are defined in kernel/sched/sched.h and are used when enabling/disabling migration for a task in core.c. I haven't been able to determine what SCA is short for from looking at the code or patch notes.
I spent two hours trying to dig it up from kernel's Git history and it seems the first apparition of this prefix starts from commit 9cfc3e18adb0362533e911bf3ce6ec8c821cfccc, which says :
sched: Massage set_cpus_allowed()
Thread a u32 flags word through the set_cpus_allowed () callchain.
This will allow adding behavioural tweaks for future users.
I believe this is what it means.
Sampling rate can be set for perf record command using -F. I want to know what is the sampling rate for intel_pt event i.e., for command
perf record -e intel_pt// -- ./a.out
With -F in user mode max sampling rate allowed is 8000. While it is possible that perf record stores the trace few thousand times per second, but the trace event that are recorded using perf record -e intel_pt// have much higher frequency.
In other words with intel_pt event a trace of an application execution is collected. Is it the case that perf record work differently while recording using intel_pt event, i.e., in some non-sampling mode?
Yes, intel_pt mode of perf record is different and is not same sampling (statistical) profiling with software (cpu-clock) or hardware (cycles) events. Sampling has 4000 of current EIP samples per second and gives you basic inexact view over code execution. intel_pt is hardware-based tracing technique which generates a lot of data about every control flow instruction (in default perf intel_pt mode) allowing to reconstruct full control flow, but it has bigger overhead. So, frequency of Intel PT is same as how many calls, branches and returns are executed per second by program code (100s of millions).
With sampling on hardware events, perf record will ask hardware PMU to count some events like CPU cycles, and to generate an overflow interrupt after for example 2 million of such events. On such interrupt perf_events subsystem in kernel will record current OS timestamp, pid/tid of current thread, EIP instruction pointer to ring buffer and reset the PMU counter for new value. perf subsystem does limit maximum frequency of interrupts by autotuning the value, and -F option can be used to change desired frequency of interrupts. When the ring buffer (around several megabytes in size) is filled, perf user-space tool will dump it contents into perf.data file, and you can view raw data with perf script or perf script -D. Or just to make histograms with perf report (sort EIPs by how often there was an interrupt on that EIP instruction address, which is proportional to time taken by that code). This mode has around 4 thousand events per second of thread execution (perf report --header | grep sample_freq), with 48 bytes per sample, or 192 kilobyte per second. Overhead is basically low enough, but the sampling is not exact.
perf wiki has separate page for intel processor trace (intel_pt) - https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
Control flow tracing is different from other kinds of performance analysis and debugging. It provides fine-grained information on branches taken in a program, but that means there can be a vast amount of trace data. Such an enormous amount of trace data creates a number of challenges, but it raises the central question: how to reduce the amount of trace data that needs to be captured. That inverts the way performance analysis is normally done. Instead of taking a test case and creating a trace of it, you need first to create a test case that is suitable for tracing.
So, intel_pt is tracing (logging) module integrated into CPU hardware, and when armed it will generate "hundreds of megabytes of trace data per CPU per second", according to used settings. With some settings it may event generate tracing data (packet log) faster than it can be written to disk or even to RAM ("overflow packets"). According to https://lwn.net/Articles/648154/ article, perf_events (kernel-mode) in intel_pt mode will just save full packet log into separate (bigger?) ring buffer and perf tool (user-space) will just periodically save data from ring buffer into file for offline filtering, parsing and decode. (Period of saving aux or ring mmap into the file is not the same as overflow interrupt frequency option -F) PT decoder then will be used to reconstruct PT packet log into perf-compatible samples. Log data volume is huge, overhead is 1% - 5% - 10% or more depending on branch frequency in code executed.
Documentation of intel_pt is manpage man perf-intel-pt and long text stored inside linux kernel source code at
https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt
Intel PT is first supported in Intel Core M and 5th generation Intel Core
processors that are based on the Intel micro-architecture code name Broadwell.
Trace data is collected by 'perf record' and stored within the perf.data file. ... Trace data must be 'decoded' which involves walking the object code and matching the trace data packets. ... Decoding is done on-the-fly. The decoder outputs samples in the same format as
samples output by perf hardware events, for example as though the "instructions"
or "branches" events had been recorded. Presently 3 tools support this:
'perf script', 'perf report' and 'perf inject'. ... The main distinguishing feature of Intel PT is that the decoder can determine
the exact flow of software execution. Intel PT can be used to understand why
and how did software get to a certain point, or behave a certain way. ... A limitation of Intel PT is that it produces huge amounts of trace data
(hundreds of megabytes per second per core) which takes a long time to decode
By default perf record -e intel_pt// is same as -e intel_pt/tsc=1,noretcomp=0/. config terms section of manpage man perf-intel-pt says what is default settings:
tsc Always supported.
Produces TSC timestamp packets to provide timing information. In some cases it is possible to decode without timing information, for example a per-thread context that does not overlap executable memory maps.
noretcomp Always supported. Disables "return compression" so a TIP
packet is produced when a function returns. Causes more packets to be
produced but might make decoding more reliable.
pt Specifies pass-through which enables the branch config term.
branch Enable branch tracing. Branch tracing is enabled by default
To represent software control flow, "branches" samples are produced.
By default a branch sample is synthesized for every single branch.
As it says, intel_pt in default mode is used to produce control flow log, by asking hardware to generate log packets for every control flow instruction like call, branch, return, and to add timestamps to synchronize pt log with some service perf samples (like exec or mmap to find actual code being loaded into memory). It tries to generate not too much, for example [single bit is used per conditional branch (tnt)](https://conference.hitb.org/hitbsecconf2017ams/materials/D1T1 - Richard Johnson - Harnessing Intel Processor Trace on Windows for Vulnerability Discovery.pdf#page=12) and several bytes per indirect branch, but there are hundreds of millions branches per second for many programs.
Some useful and short slides on perf + intel_pt:
Andi Kleen, 2015 https://halobates.de/pt-tracing-summit15.pdf (PT modes current: Full trace mode, Snapshot mode; Upcoming: Sampling mode, Core dump, System crash mode)
Andi Kleen's posts on PT: https://halobates.de/blog/p/category/pt
Suchakrapani Datt Sharma, POLYTECHNIQUE MONTREAL, 2015 https://hsdm.dorsal.polymtl.ca/system/files/10Dec2015_0.pdf (trace packets overview - PSB (Packet Stream Boundary), TNT (Taken Not-Taken), TIP (Target IP) at branches, non-default CYC Packets : Cycle counter data for IPC, MTC (Mini Timestamp Counter), ...)
Jack Henschel, 2017 about design and use-cases https://blog.cubieserver.de/publications/Henschel_Intel-PT_2017.pdf
[https://events.static.linuxfound.org/sites/events/files/slides/lcna13_kleen.pdf Efficient and Large Scale Program Flow Tracing in Linux, Alexander Shishkin], Intel, 2013 ("What is it good for? •Profiling / performance measurement •Functional debugging •Code coverage analysis")
About generic difference between sampling and (software) tracing: https://danluu.com/perf-tracing/
Update: While intel pt trace log has full trace (there are packets inside for every branch/call/return), perf report does run conversion from pt log into sample set like in classic perf.data, and there is sampling rate in sample set. This is configured with --itrace option of perf report (iNNTT, where NN is amount and TT is type - i/t/us/ns, as described in man page of perf-report:
--itrace
Options for decoding instruction tracing data. The options are:
i synthesize instructions events
g synthesize a call chain (use with i or x)
The default is all events i.e. the same as --itrace=ibxwpe,
In addition, the period (default 100000, ...)
for instructions events can be specified in units of:
i instructions
t ticks
ms milliseconds
us microseconds
ns nanoseconds (default)
So it seems like by default perf report will convert full trace log into instruction samples at sampling rate of 100000 instructions (1 perf sample generated per 100 thousands instructions). It can be changed to higher rate, but processing time will increase.
Manpage of perf-intel-pt gives more examples of itrace option usage:
Because samples are synthesized after-the-fact, the sampling period
can be selected for reporting. e.g. sample every microsecond
sudo perf report pt_ls --itrace=i1usge
See the sections below for more information about the --itrace
option.
Beware the smaller the period, the more samples that are produced,
and the longer it takes to process them.
Also note that the coarseness of Intel PT timing information will
start to distort the statistical value of the sampling as the
sampling period becomes smaller.
To see every possible IPC value, "instructions" events can be used
e.g. --itrace=i0ns
--itrace=i10us
sets the period to 10us i.e. one instruction sample is synthesized
for each 10 microseconds of trace. Alternatives to "us" are "ms"
(milliseconds), "ns" (nanoseconds), "t" (TSC ticks) or "i"
(instructions).
For Intel PT, the default period is 100us.
Setting it to a zero period means "as often as possible".
In the case of Intel PT that is the same as a period of 1 and a unit
of instructions (i.e. --itrace=i1i).
http://halobates.de/blog/p/410 has some additional examples of complex conversions:
perf script --ns --itrace=cr
Record program execution and display function call graph.
perf script by defaults “samples” the data (only dumps a sample every
100us). This can be configured using the --itrace option (see
reference below)
perf script --itrace=i0ns --ns -F time,pid,comm,sym,symoff,insn,ip | xed -F insn: -S /proc/kallsyms -64
Show every assembly instruction executed with disassembler.
perf report --itrace=g32l64i100us --branch-history
Print hot paths every 100us as call graph histograms
perf script --itrace=i100usg | stackcollapse-perf.pl > workload.folded
flamegraph.pl workloaded.folded > workload.svg
google-chrome workload.svg
Generate flame graph from execution, sampled every 100us
I am usign the Intel Advisor 2018 (build 523188) on Linux CentOS 7.4 to profile a collection of benchmarks (I want to plot them all in a single Roofline plot) and I am using the command line tool advixe-cl to collect the survey, tripcounts and flops information for each benchmark.
However, I cannot find a way to report the measured performance in FLOPs (for each loop or function or even whole program) using the command line interface. The documentation I am looking at is found here https://software.intel.com/en-us/advisor-help-lin-command-line-interface-reference, but I think that it is not complete e.g. the options -flops-and-masks and -no-tip-counts are not mentioned anywhere.
Do you know if there is any way to report the measured flops via the command line interface? Or do you know where I can find a complete documentation of advixe-cl?
You have to first "collect" (means "profile") FLOPS data and, secondly, report it via user interface.
To collect the data: follow https://software.intel.com/en-us/articles/intel-advisor-roofline (better and newer article) or https://software.intel.com/en-us/intel-advisor-2017-user-guide-linux-running-roofline-analysis (older syntax).
To report/investigate the FLOP values/data it is recommended to launch Intel Advisor GUI: $advixe-gui <project-dir>
Alternatively you could report/explore FLOP values using command line reports ($advixe-cl -report survey <project-dir>).
See advixe-cl documentation at :
https://software.intel.com/en-us/intel-advisor-2017-user-guide-linux-using-intel-advisor-command-line-interface or https://software.intel.com/en-us/intel-advisor-2017-user-guide-linux-report
I have an SNMP monitoring box and want to monitor interface utilisation on a clustered database server. I'm trying to work out the correct OID to monitor - I just need SNMP to return the total interface throughput at a given time.
The SNMP box is already configured and will correctly graph it. All howtos I can find talk about setting up Catci or MRTG which is all well and good, but what I need seems simpler, yet I can't seem to find what I'm looking for. The SNMP box is already configured with the correct community name etc so this should be a really easy one in theory.
Any help very gratefully received
Thanks
When you say "interface utilisation", I assume you mean Ethernet interface utilization. If that assumption is correct, there are a couple OIDs to investigate:
1.3.6.1.2.1.2.2.1.10 - ifInOctets returns the total number of octets received on the interface, including framing characters.
1.3.6.1.2.1.2.2.1.16 - ifOutOctets returns the total number of octets transmitted out of the interface, including framing characters.
1.3.6.1.2.1.31.1.1.1.6 - ifHCInOctets returns the total number of octets received on the interface, including framing characters (this is the 64-bit version of ifInOctets).
1.3.6.1.2.1.31.1.1.1.10 - ifHCInOctets returns the total number of octets transmitted out of the interface, including framing characters (this is the 64-bit version of ifOutOctets).
Each OID is part of a table and will have an associated index that links it to an interface description (e.g., eth0 or br1).
These OIDs provide a count of octets received and transmitted so they require a little massaging to get into the utilization rates you desire. In the past when I've monitored these OIDs I've queried for two values a few seconds apart and then calculated the rate.
(QueryResult2 - QueryResult1) / (SecondsElapsed)
I would guess that Cacti (which I assume you're using since you tagged your question with it) has some way to calculate rates from SNMP values, however, I've never used it so I am not positive.
One other important note is that the default snmpd.conf included with CentOS may not have these OIDs enabled. If you run snmpwalk on 1.3.6.1.2.1.2 and 1.3.6.1.2.1.31 and receive empty results, edit /etc/snmpd.conf to configure the SNMP daemon to respond to those OIDs. I can't remember the exact syntax but I think adding a line like,
view all included .1
will enable all available OIDs on the server.
http://namhuy.net/908/how-to-install-iftop-bandwidth-monitoring-tool-in-rhel-centos-fedora.html
Requirements:
libpcap: module provides a user-level network packet capture information and statistics.
libncurses: is a API programming library that enables programmers to provide text-based interfaces in a terminal.
gcc: GNU Compiler Collection (GCC) is a compiler system produced by the GNU Project supporting various programming languages.
Install libpcap, libnurses, gcc via yum
yum -y install libpcap libpcap-devel ncurses ncurses-devel gcc
Download and Install iftop
wget http://www.ex-parrot.com/pdw/iftop/download/iftop-0.17.tar.gz
./configure
make
make install