How does HTOP gather information like CPU consumption, memory consumption etc by every process and the whole system? I am not familiar with any operating system specific api (like socket api) that is provided by linux in order to gather those statistics.
Well, I'd point you to the htop sources, first.
Then, operating systems like Linux have a lot of interfaces to get those information. Some of these metrics are available via the classical /proc pseudo-filesystem, or via /sys/; others might be only available via ioctls or specific system calls (e.g. getgid).
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I'm studying Operating Systems. I read Window have lots of system calls for manage windows and GUI components. I have read you can change the GUI manager of your Linux Operating System. Then does Linux have system calls for GUI managements? How GUI works in Linux?
I'll take x86 as an example as I am more aware of x86 stuff than ARM stuff. Also, I may get some information wrong as I've been doing some research on this question while answering. Feel free to correct me if I am wrong.
System booting
Some time ago, Linux used to boot with a legacy bootloader (GRUB legacy version). The GRUB bootloader would be started by the BIOS at 0x7c00 in RAM and then would read the kernel from the hard-disk. It would follow the multiboot specification. The multiboot specification mentions the state that the computer needs to be in before jumping to the kernel's entry point. The kernel would then launch a first process (init) that every process would be a child of.
Today, most Linux distributions boot with UEFI (with the option of legacy booting also available). A UEFI app is placed on the boot partition partionned as a GPT ESP (EFI System Partition). This EFI app is launched and then follows the Linux Boot Protocol to launch Linux. The init process was also replaced by systemd. Linux will thus launch systemd as the first process of the computer. Actually, as stated on the manpage for systemd:
systemd is usually not invoked directly by the user, but is
installed as the /sbin/init symlink and started during early
boot.
The process that will be started is thus /sbin/init but it is a symlink to systemd. The systemd process will then read several configuration files on the hard-disk called units. These units are often targets which specify several units to read. Targets are thus units which specify several units to read. At first systemd will read default.target which specifies several other units. Some of these other units will start some processes among which is the Display Manager (fancy terminology which means login prompt). Recently, Ubuntu starts the Gnome Display Manager (GDM) as the first displaying program (gdm.service unit). This program will start the X server before presenting the user login screen (https://en.wikipedia.org/wiki/X_display_manager).
When the display manager runs on the user's computer, it starts the X server before presenting the user the login screen, optionally repeating when the user logs out.
Once logged in, GDM will start several other binaries responsible to let you interact with the system (the actual desktop, a binary to gather input for this desktop, etc). All of these components depend on the X server to work properly.
The DRM
The X server is a user program which makes extensive use of the Direct Rendering Manager (DRM) of the Linux kernel. The DRM is a system call interface which is used to interact with graphics cards. When the DRM detects a graphics card, it exposes a file like /dev/dri/card0 which is a character device (http://manpages.ubuntu.com/manpages/bionic/man7/drm.7.html).
In earlier days, the kernel framework was solely used to provide raw hardware access to
priviledged user-space processes which implement all the hardware abstraction layers. But
more and more tasks were moved into the kernel. All these interfaces are based on ioctl(2)
commands on the DRM character device. The libdrm library provides wrappers for these
system-calls and many helpers to simplify the API.
When a GPU is detected, the DRM system loads a driver for the detected hardware type. Each
connected GPU is then presented to user-space via a character-device that is usually
available as /dev/dri/card0 and can be accessed with open(2) and close(2). However, it
still depends on the grapics driver which interfaces are available on these devices. If an
interface is not available, the syscalls will fail with EINVAL.
The ioctl call allows to have any number of operations on the /dev/dri/card0 file since it is a general call which includes a request argument which is simply an unsigned long. It also takes a variable amount of arguments (see https://man7.org/linux/man-pages/man2/ioctl.2.html).
The ioctl call thus allows hardware vendors (like NVIDIA, AMD, etc) to provide drivers for their cards with the general ioctl call used as a general interface between user mode and kernel mode.
OpenGL
There exists several 3D rendering APIs available (OpenGL, Direct3D). OpenGL is mostly a set of C headers and a convention. The convention says what a certain call should do. It is up to the hardware vendor to implement the convention for their own card. Mesa3D has been an attempt to create an open source implementation of OpenGL for certain graphics cards. It worked quite well for integrated Intel HD Graphics (since documentation is open) and AMD (since they cooperated and offered some insight into the workings of their cards), but not for NVIDIA (the Nouveau driver is mostly not working or slow).
When you program some OpenGL, you include the OpenGL headers and link with libraries provided by hardware vendors which provide the definitions of the functions in the headers. These definitions make use of the DRM and cooperate with the X server to show content on the screen.
I'm studying Operating Systems. I read Window have lots of system calls for manage windows and GUI components. I have read you can change the GUI manager of your Linux Operating System. Then does Linux have system calls for GUI managements? How GUI works in Linux?
System calls (provided by the kernel) are often buried (e.g. in some cases deliberately undocumented and proprietary) and should not be used. Almost everything you see are actually normal functions in dynamically linked libraries/shared libraries. This allows the kernel's system calls to be radically changed without breaking everything (because everything only depends on the dynamically linked libraries/shared libraries); and reduces the functionality needed in the kernel itself.
For an example; most of the "system calls for managing windows and GUI components" you think Windows has could (internally, inside the relevant DLL) just end up using a single "send_message()" system call (to tell a different process, the GUI, that you want to create a window or change its position or ...).
For Linux it's roughly similar. The kernel's system calls (which actually are documented, for no sane reason - it goes against the spirit of SYS-V specs and means badly written "linux executables" aren't compatible with other Unix clones like FreeBSD or Solaris or OSX) exist to use things like low level memory management and raw file IO and sockets; but (like Windows) the kernel's system calls are buried under layers of shared libraries, and those shared libraries (e.g. like Xlib, GLib, KWindowSystem, Qt, ...) just use "something" (file IO, pipes, sockets, ...) provided by kernel to talk to another process (display server, GUI, ..).
Linux and Windows fall under separate categories; Linux is just a kernel, i.e. the piece under the hood that gives us the basic functionality we expect to run programs, like threads, memory and process management, etc. Windows is a full operating system, including the user facing components and numerous system libraries. An apter comparison would be a specific Linux distro and Windows.
On that note, distros, as independent operating systems, obviously can have different implementations of any OS component. Some distros, like Arch, don't come with a GUI by default at all. That said, essentially the entire Linux ecosystem uses Xorg and/or Wayland; I would recommend looking into the implementation details of those two.
A Linux GUI has quite a few differences compared to Windows GUI. For example, the GUI is not considered to be a part of the operating system, but rather an external part of it; that means no syscalls (not embedded whatsoever in the OS). After all, like the previous answer says, Linux is a kernel, that means it's only something really basic (allows execution of programs, memory/threads management, processes management, but not really much more). Whatever comes next (GUI, for example) are added features using packages.
This allows, for example, installing a GUI on top of a minimal installation of any Linux distro (CentOS, for example), and that GUI can be the one you want (Gnome, KDE...).
I'm trying to get hold of CPU architecture information under Linux.
I understand the information is available via the sysfs filesystem.
I have CentOS 5 running in a Xen VM. The sysfs filesystem is mounted. However, the /sys/devices/system/cpu/cpu0/ directory is almost empty. The only entry is a single file, "online", with a value of "1".
What gives? where's all my CPU information?
The actual cpu information is still in /proc/cpuinfo.
The sysfs-files are used to control things like scheduling and frequency settings, not to get information on the cpus themselves.
Okay, I've just had a chat with a sysadmin at work.
Looking at some machines, it looks like this information simply is not pushed by VMs. The VMs think they have a virtual CPU - rather than a CPU of the type of the real underlying CPU - and the cache information simply is not published.
It is published (and it's nice to finally see it!) on real machines with reasonably modern kernels.
I would like to profile my embedded linux device for various actions (e.g. touch input on screen, flip a page in a news reading app etc). But the requirement is that the profiler should generate information for the entire software stack on the device both user space and kernel space. It would be best if the profiler does not include a significant overhead for logging purposes as memory and resources are very limited on the embedded linux device.
Any suggestions ?
Have you tried OProfile?
It supports several hardware architectures and I believe that it is included in the latest mainline kernels. It can also profile both the userspace and the kernel itself.
I'm trying to do some tuning for Oracle on Linux boxes living on SAN based infrastructure. I'm looking specifically for tools that would allow us to profile IO per process (or per process tree would be even better). My questions are?
What are the tools that would be recommended for this kind of task?
What other useful metrics should I seek to measure on a SAN based infrastructure?
I have used "iotop" with great results. It gets specific info per process with IO usage.
It works like "top"
http://guichaz.free.fr/iotop/
I am not sure though if it would be reasonable to use from a Linux box that has the SAN mounted or if you wanted a tool that could run within the SAN.
Once you start to get this specialized, I've found that the easiest thing to do is to write some custom scripts that pull information from files under /proc.
If you're analysis for which you don't already have a tool that gives you the exact report you need, you're probably going to end up doing some scripting anyway, and most of the tools you'd use under Linux are just going to /proc to get their information anyway and then reformatting it for you.
If you're more into the databasey side of things, pulling info from /proc on a regular basis, adding timestamps, and recording it in a way that it can be imported into an RDBMS can be very useful. This can be particularly good if you put all of your server and process performance information into a single RDBMS, because then you can compare, arbitrary things such as the performance of the same application on different servers.
Keep in mind that if you go further with this, you my start adding information from different sources, such as IPMI monitoring of hosts, so don't do things that you'll have to undo once you're using more than /proc.
You can use sysstat utilities which are a collection of performance monitoring tools for
Linux.
From the website (perso.orange.fr/sebastien.godard/)
* Can monitor a huge number of different metrics:
1. Input / Output and transfer rate statistics (global, per device, per partition, per network filesystem and per Linux task / PID)
2. CPU statistics (global, per CPU and per Linux task / PID), including support for virtualization architectures
3. Memory and swap space utilization statistics
4. Virtual memory, paging and fault statistics
5. Per-task (per-PID) memory and page fault statistics
6. Global CPU and page fault statistics for tasks and all their children
7. Process creation activity
8. Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources)
9. Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols.
10. NFS server and client activity
11. Socket statistics
12. Run queue and system load statistics
13. Kernel internal tables utilization statistics
14. System and per Linux task switching activity
15. Swapping statistics
16. TTY device activity
17. Power management statistics
I usually use atop to monitor the load on my systems. Some features require that you patch the kernel, but it gives precise information about I/O as well as other info.
What other useful metrics should I seek to measure on a SAN based infrastructure?
CPU load. It's Main metrics for oracle database.
Depending on how low-level you want to get, System Tap could be very useful for you. It is similar to DTrace on Solaris.
We implemented a server application available on Windows only. Now we like to port it to Linux, HP-UX and AIX, too. This application provides internal statistics through performance counters into the Windows Performance Monitor.
To be more precise: The application is a data base, and we like to provide information like number of connected users or number of requests executed to the administrator. So these are "new" information, proprietary to our application. But we like to make them available in the same environment where the operating system delivers information like the CPU, etc. The goal is to make them easily readable for the administrator.
What is the appropriate and commonly used performance monitor under Linux, HP-UX and AIX?
I would say: that depends on which performance you want to monitor. Used CPU time? Free RAM? Disk IO? Number of beers in your freezer...
But regardless of this you can look at any files below /proc. I'm not sure for HP, but at least Linux and AIX should have that tree (if it's not deactivated at kernel compile time).
Management is where most OSes depart from one another. For this reason there are not many tools that are common between all the OSes.
Additionally, Unix tools follow the single process single responsibility idiom where one tool gets cpu info, another gets memory etc.
The only tool i have seen in the Unix world that gets all this info in one place is top. Almost all sys admins are familiar with this tool and works on all the flavors of OSes you are interested in. It also has the additional advantage of being open source. You could simply extend this tool to expose the counters you are interested in and ship it along with your application.
Another way to do this might be to expose your counters through SNMP and leave it to some third party SNMP tool like HP open view that can collect and present a consistent view along with other management info. This might be a more enterprisy solution, which might appeal to the marketing folks.
I would also say its a good idea to write a standalone console tool that admins can use from their custom home grown scripts (there are many firsm out there with super human admins / over paid it staff that does this).
All together would be a healthy solution for your requirement i think.
The most standard unix tools for such data are the *stat (iostat, vmstat, netstat) tools and sar. On Linux you'll find all this information in /proc, but most Unixes don't have /proc nicely filled with what you are looking for. The mentioned tools are quite standardized and can be used to gather the data you need.