So I have finished reading an article here:
https://msdn.microsoft.com/en-us/library/ms809983.aspx
about why we have COM and how it lets us share code without worrying about name mangling of compilers or unicode/ascii issues or memory management in a language independent manner.
I have elsewhere read that COM isn't supposed by LINUX because COM basically uses the OS as the moderator for acquisition of these standardized objects. Shouldn't there be something similar in Linux? and if so, what is it?
On Linux you can run any program that accepts its input on standard input, and connect it, via a pipe, to any other program that generates its results on its standard output.
The simple, file and pipe-based input/output in POSIX predates MS-Windows by decades. And, as long as both sides of the pipe agree on the format of the data being interchanged, it doesn't matter which compiler was used to create each program (although, on Linux, there's pretty one de-factor compiler, so it's a moot point).
And by using a socket-pair, the pipe becomes bi-directional, so both processes can swap data with each other.
This is, generally, how processes interoperate on Linux:
1) A pipe, or a network socket, that connects the two processes together
2) An agreed, established standard for the format of the data exchanged between the two processes.
It is important to understand that there is no practical standard for all processes to use the same exact format for exchanging messages. The closest that would come to such a standard, I suppose, would be the remote procedure call, or RPC, standard that's used in some low-level protocols, like NFS, but, mostly, individual applications define and use a particular format that's tailored for them.
For example, the X Window System Protocol: http://www.x.org/releases/X11R7.7/doc/xproto/x11protocol.html -- this is a format definition of a protocol for communicating between an X server and an X client. Applications that are written to use this protocol (they'll typically use an intermediate library or a toolkit, actually) can establish a connection and use any X server that talks the same protocol, over a network connection or a local pipe.
Related
My question lies in a paragraph, the paragraph are shown as follow, I can't understand the the bold sentence. If it doesn't need to invoke message passing, how does it complete communication between process?
Modules
Perhaps the best current methodology for operating-system design involves
using loadable kernel modules (LKMs). Here, the kernel has a set of core
components and can link in additional services via modules, either at boot time
or during run time. This type of design is common in modern implementations
of UNIX, such as Linux, macOS, and Solaris, as well as Windows.
The idea of the design is for the kernel to provide core services, while
other services are implemented dynamically, as the kernel is running. Linking
services dynamically is preferable to adding new features directly to the kernel,
which would require recompiling the kernel every time a change was made.
Thus, for example, we might build CPU scheduling and memory management
algorithms directly into the kernel and then add support for different file
systems by way of loadable modules.
The overall result resembles a layered system in that each kernel section
has defined, protected interfaces; but it is more flexible than a layered system,
because any module can call any other module. The approach is also similar to
the microkernel approach in that the primary module has only core functions
and knowledge of how to load and communicate with other modules; but it
is more efficient, because modules do not need to invoke message passing in
order to communicate.
Linux uses loadable kernel modules, primarily for supporting device
drivers and file systems. LKMs can be “inserted” into the kernel as the system is started (or booted) or during run time, such as when a USB device is
plugged into a running machine. If the Linux kernel does not have the necessary driver, it can be dynamically loaded. LKMs can be removed from the
kernel during run time as well. For Linux, LKMs allow a dynamic and modular
kernel, while maintaining the performance benefits of a monolithic system. We
cover creating LKMs in Linux in several programming exercises at the end of
this chapter.
In OS, why loadable kernel modules (LKMs) don't need to invoke message passing in order to communicate?
The simple answer is that because they're loaded into kernel space and dynamically linked; the kernel can use "mostly normal" functions calls instead of anything more expensive (message passing, remote procedure calls, ...) to communicate with it.
Note: Typically (especially for *nix systems) a driver will provide a set of function pointers to the kernel (e.g. maybe one for open(), one for read(), one for ioctl(), etc) in some kind of "device context" structure; allowing the kernel to call the driver's functions via. the function pointers (e.g. like "result = deviceContext->open( ..);).
"The approach is also similar to the microkernel approach in that the primary module has only core functions and knowledge of how to load and communicate with other modules; but it is more efficient, because modules do not need to invoke message passing in order to communicate."
This paragraph has the potential to give you a false impression. For extensibility alone, modular monolithic kernels are similar to micro-kernels (and both are a lot more extensible than a "literally monolithic (one piece, like stone)" kernel). For other things (e.g. security) modular monolithic kernels are extremely dissimilar to micro-kernels.
For Linux specifically; you can think of it as almost 30 million lines (growing at a rate of over 1 million lines per year) of potential security vulnerabilities running at the highest privilege level with full access to every scrap of data, with an average of about 150 discovered critical vulnerabilities per year (and who knows how many undiscovered critical vulnerabilities).
One of the main goals of micro-kernels is to place isolation barriers between the "kernel core" and everything else; so that you might end up with several thousand lines of kernel that doesn't grow (and a significant improvement in security). It's those isolation barriers that require less efficient communication (e.g. message passing).
"...but it is more efficient, because modules do not need to invoke message passing in order to communicate."
This could be rephrased more correctly as "...but it is more efficient, because modules do not need to pass through an isolation barrier."
Note that message passing is merely one way to pass through an isolation barrier - there's shared memory, signals, pipes, sockets, remote procedure calls, etc. Nothing says a micro-kernel has to use message passing and you could design a micro-kernel that does not use message passing at all.
Erlang supplies two ways for managing OS processes the simple os module and the better API built around erlang:open_port(). I've also found an interesting project on GitHub partially written in C++ called erlexec.
Anyway all of this doesn't fit the need of controlling a running process. I'd like to:
Get OS PID from process name.
Send signals to the process.
(Optional: Send data to the process).
Check process statistics: at least CPU Usage % and Used Memory (same data of commands like top).
Being able to support at least Linux and Mac OS X platforms.
After doing some research I've concluded that, in Linux platform for example, I need to use a mix of C calls and reading from /proc filesystem.
Am I on the right path or there's another way (excluding calling shell commands to get data or perform operations)?
Is there out a library I wasn't able to find (an Erlang one or a C library well suited to be called from Erlang?
I'm very familiar with Linux (I've been using it for 2 years, no Windows for 1 1/2 years), and I'm finally digging deeper into kernel programming and I'm working a project. So my questions are:
Will a kernel module run faster than a traditional c program.
How can I communicate with a module (is that even possible), for example call a function in it.
1.Will a kernel module run faster than a traditional c program.
It Depends™
Running as a kernel module means you get to play by different rules, you potentially get to avoid some context switches depending on what you are doing. You get access to some powerful tools that can be leveraged to optimize your code, but don't expect your code to run magically faster just by throwing everything in kernelspace.
2.How can I communicate with a module (is that even possible), for example call a function in it.
There are various ways:
You can use the various file system interfaces: procfs, sysfs, debugfs, sysctl, ...
You could register a char device
You can make use of the Netlink interface
You could create new syscalls, although that's heavily discouraged
And you can always come up with your own scheme, or use some less common APIs
Will a kernel module run faster than a traditional c program.
The kernel is already a C program, which is most likely be compiled with same compiler you use. So generic algorithms or some processor intensive computations will be executed with almost same speed.
But most userspace programs (like bash) have to ask kernel to perform some operations on system resources, i.e. print prompt onto monitor. It will require entering the kernel with system call, sending data over tty interfaces and passing to a video-driver, it may introduce some latency. If you'd implemented bash in-kernel, you may directly call video-driver, which is definitely faster.
That approach however, have drawbacks. First of all, bash should be able to print prompt on ssh-session or serial console, and that will complicate logic. Also, if your bash will hang, you cannot just kill, you have to reboot system.
How can I communicate with a module (is that even possible), for example call a function in it.
In addition to excellent list provided by #tux3, I would suggest to start with char devices.
In Linux (2.6.35-22-generic), man pipe states that
pipe() creates a pipe, a unidirectional data channel that can be used for interprocess communication."
In FreeBSD (6.3-RELEASE-p5), man pipe states that
The pipe() system call creates a pipe, which is an object allowing bidirectional data flow, and allocates a pair of file descriptors."
One is unidirectional, the other is bidirectional. I hope this isn't a silly question, but which method is the standard way of doing this? Are they both POSIX compliant?
To make my intentions clear, I lost some points on an exam for believing pipe() was one way and am looking for some ammo to get any points back ;p
I started this as a comment on Greg's answer at first, but it occurs to me that it more closely answers your specific question:
pipe()s documentation in the POSIX standard explicitly states that the behavior in question is "unspecified" -- that is, pipe() is not required to be bidirectional, though it's not forbidden. Linux's is unidirectional, FreeBSD's is bidirectional. Both are compliant, one just implements additional behavior that is not required (but doesn't break apps built to work on compliant systems).
Data can be written to the file
descriptor fildes[1] and read from the
file descriptor fildes[0]. A read on
the file descriptor fildes[0] shall
access data written to the file
descriptor fildes[1] on a
first-in-first-out basis. It is
unspecified whether fildes[0] is also
open for writing and whether fildes[1]
is also open for reading.
I wouldn't count on getting the points back (though you should). Professors have a tendency to ignore the real world in favor of whatever they've decided is correct.
The FreeBSD man page for pipe is pretty clear on this point:
The bidirectional nature of this implementation of pipes is not portable to older systems, so it is recommended to use the convention for using the endpoints in the traditional manner when using a pipe in one direction.
Both "netstat -p" and "lsof -n -i -P" seems to readlinking all processes fd's, like stat /proc/*/fd/*.
How to do it more efficiently?
My program wants to know what process is connecting to it. Traversing all processes again and again seems too ineffective.
Ways suggesting iptables things or kernel patches are welcome too.
Take a look at this answer, where various methods and programs that perform socket to process mappings are mentioned. You might also try several additional techniques to improve performance:
Caching the file descriptors in /proc, and the information in /proc/net. This is done by the programs mentioned in the linked answer, but is only viable if your process lasts more than a few seconds.
You might try getpeername(), but this relies you knowing of the possible endpoints and what processes they map to. Your questions suggests that you are connecting sockets locally, you might try using Unix sockets which allow you to receive the credentials of a peer when exchanging messages by passing SO_PASSCRED to setsockopt(). Take a look at these examples (they're pretty nasty but the best I could find).
http://www.lst.de/~okir/blackhats/node121.html
http://www.zanshu.com/ebook/44_secure-programming-cookbook-for-c-and-cpp/0596003943_secureprgckbk-chp-9-sect-8.html
Take a look at fs/proc/base.c in the Linux kernel. This is the heart of the information given by the result of a readlink on a file descriptor in /proc/PID/fd/FD. A significant part of the overhead is the passing of the requests up and down the VFS layer, the numerous locking that occurs on all the kernel data structures that provide the information given, and the stringyfying and destringyfying at the kernel and your end respectively. You might adapt some of the code in this file to generate this information without many of the intermediate layers, in particular minimizing the locking to once per process, or simply once per scan of the entire data set you're after.
My personal recommendation is to just brute force it for now, ideally traverse the processes in /proc in reverse numerical order, as the more recent and interesting processes will have higher PIDs, and return as soon as you've located the results you're after. Doing this once per incoming connection is relatively cheap, it really depends on how performance critical your application is. You'll definitely find it worthwhile to bypass calling netstat and directly parse the new connection from /proc/net/PROTO, then locate the socket in /proc/PID/fd. If all your traffic is localhost, just switch to Unix sockets and get the credentials directly. Writing a new syscall or proc module that dumps huge amounts of data regarding file descriptors I'd save for last.