I am hoping that someone can point me toward Linux software similar to the Microsoft tools Application Verifier and Driver Verifier. (They are stress testers for Windows applications and drivers, respectively.)
Do such things exist for Linux?
I'm not familiar with Application Verifier and Driver Verifier at all...
For applications, Valgrind is very useful as a tool to check for leaks, use-after-free, double free, buffer overflow, use of unitialized data, unsafe concurrent data access, and much more.
There also exist many fuzzers (zzuf, fusil, etc.) which test a program's resiliance to invalid input.
GCC itself has -fstackprotector, which enables SSP (stack-smashing protector, aka ProPolice); -fmudflap, which detecs some other bad memory usage; and (in conjunction with glibc) -D_FORTIFY_SOURCE=n, which puts extra checking on various string and memory functions.
In the Linux kernel, there are many configuration switches under the "Kernel hacking" menu:
CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, etc., which ensure that memory is allocated, used, and freed sanely
CONFIG_DEBUG_OBJECTS, which checks that objects are used and freed orderly
kmemcheck, "Valgrind for the kernel"
CONFIG_PROVE_LOCKING, which analyzes for all possible deadlocks
CONFIG_DEBUG_PREEMPT, CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_SPINLOCK_SLEEP, etc., which warn on improper use of locking
CONFIG_FAULT_INJECTION & co., which probabilistically cause failures of memory allocation and I/O
Related
This question already has answers here:
Userspace vs kernel space driver
(2 answers)
Closed 5 years ago.
I have been reading "Linux Device Drivers" by Jonathan Corbet. I have some questions that I want to know:
What are the main differences between a user-space driver and a kernel driver?
What are the limitations of both of them?
Why user-space drivers are commonly used and preferred nowadays over kernel drivers?
What are the main differences between a user-space driver and a kernel driver?
User space drivers run in user space. Kernel drivers run in kernel space.
What are the limitations of both of them?
The kernel driver can do anything the kernel can, so you could say it has no limitations. But kernel drivers are much harder to "prove correct" and debug. It's all-to-easy to introduce race conditions, or use a kernel function in the wrong context or with the wrong locking. Things will appear to work for a while, but cause problems (including crashing the whole system) down the road. Drivers must also be wary when reading all user input (both from the device and from userspace) because invalid data can sometimes cause crashes.
A user-space driver usually needs a small shim in the kernel to do it's bidding. Usually, that 'shim' provides a simpler API. For example, the FUSE layer lets people write file systems in any language. They can be mounted, read/written, then unmounted. The shim must also protect the kernel against all invalid input.
User-space drivers have lots of limitations. For example, the kernel reserves some memory for use during emergencies, but that is not available for users-space. During memory pressure, the kernel will kill random user-space programs, but never kill kernel threads. User-space programs may be swapped out, which could lead to your device being unavailable for several seconds. (Kernel code can not be swapped out.) Running code in user-space requires several context switches. These waste a "lot" of CPU time. If your device is a 300 baud modem, nobody will notice. But if it's a gigabit Ethernet card, and every packet has to go to your userspace driver before it gets to the real user, the system will have major bottlenecks.
User space programs are also "harder" to use because you have to install that user-space software, which often has many library dependencies. Kernel modules "just work".
Why user-space drivers are commonly used and preferred nowadays over kernel drivers?
The question is "Does this complexity really need to be in the kernel?"
I used to work for a company that made USB dongles that talked a particular protocol. We could have written a full kernel driver, but instead just wrote our program on top of libUSB.
The advantages: The program was portable between Linux, Mac, Win. No worrying about our code vs the GPL.
The disadvantages: If the device needed to data to the PC and get a response quickly, there is no guarantee that would happen. For example, if we needed a real-time control loop on the PC, it would be harder to have bounded response times. (Maybe not entirely impossible on Linux.)
If there is a way to do it in userspace, I would try that first. Only if there are significant performance bottlenecks, or significant complexity in keeping it in userspace would you move it. Even then, consider the "shim" approach, and/or the "emulator" approach (where your kernel module makes your device look like a serial port or a block device.)
On the other hand, if there are already several kernel modules similar to what you want, then start there.
The documentation for PortAudio (a C audio library) states:
Pa_Terminate() MUST be called before exiting a program which uses PortAudio. Failure to do so may result in serious resource leaks, such as audio devices not being available until the next reboot.
(emphasis mine)
However, I've always understood that no modern operating system permits you to leak resources beyond the program's exit, so while it's "good practice" to clean up your allocated memory after use, it certainly won't harm anything if you don't (related questions: 1 2 etc).
But I'm also aware that older graphics card drivers would fail to free graphics memory in some situations, so leaks in that situation could cause system crashes.
My question is - does this actually happen with modern audio software and hardware?
If so, what exactly is the cause and mode of failure, in software or hardware?
If not, what's the foundation for the quote from the PortAudio documentation?
Which operating systems are vulnerable and which are not?
What can be done to prevent applications causing such problems?
Is there any special usage/advantage over each other (kmemleak and kmemcheck) ? Can I enable these tools on Android operating system (not Linux OS) please guide me how.
Ref: https://www.kernel.org/doc/Documentation/kmemcheck.txt
https://www.kernel.org/doc/Documentation/kmemleak.txt
Kmemleak and Kmemcheck perform different tasks, none is better than the other.
1.
Kmemleak checks if some memory blocks were allocated by the kernel but were not freed (that is, checks for memory leaks in the kernel, hence the name). The performance overhead is usually acceptable.
2.
Kmemcheck checks if some kernel code accesses uninitialized memory. Example: the kernel code allocates a structure, does not fill it with values and then reads something from that structure. Kmemcheck should detect that.
Kmemcheck does not check for memory leaks, by the way.
Kmemcheck often slows down the system so much that graphical environments are impossible to use. The boot process may also become very slow (and may fail).
3.
If I am not mistaken, Kmemleak works at least for x86 and ARM. Kmemcheck is x86 only.
4.
Unfortunately, I cannot say how to enable Kmemleak on Android, I only used it on desktop Linux systems.
5.
Depending on what you are trying to accomplish, there could be tools that suit your needs better. For example, the Linux kernel has a variety of debugging features that can be enabled and built in. Again, I have no experience with Android kernel in this regard.
How do operating systems detect stack overflows of user-space programs [and then send SIGTERM or SIGSEGV to those userspace programs] ?
Guard pages. When the OS creates the stack for the program it will allocate a little bit more than is specified. The memory is allocated in pages (usually 4KB each), and the extra page will have settings such that any attempt to access it will result in an exception being thrown.
The answer will depend on the target architecture and the particular OS. Since the question is tagged Linux, you have rather biased the question which on the face of it seems more general.
In a sophisticated OS or RTOS such as Linux or QNX Neutrino, with MMU protection support, memory protection mechanisms may be used such as the guard pages already mentioned. Such OSs require a target with an MMU of course.
Simpler OSs and typical RTOS scheduling kernels without MMU support may use a number of methods. The simplest is to place a guard signature at the top of the stack, which is checked for modification when the scheduler runs. This is a bit hit-and-miss, it requires that the stack-overflow actually modifies the signature, and that the resulting corruption does not cause a crash before the scheduler next runs. Some systems with on-chip debug resources may be able to place an access break-point on the signature word and cause an exception when it is hit.
In development a common technique is to initially fill each thread stack with a signature and to have a thread periodically check for the "high-tide" and issue a warning if it exceeds a certain percentage level.
As well as guard pages mentioned in another answer, some smaller (MMU-less) embedded microcontrollers have specific exceptions for stack overflow (and underflow).
I thought this was expected behavior?
From: http://classic.chem.msu.su/cgi-bin/ceilidh.exe/gran/gamess/forum/?C35e9ea936bHW-7675-1380-00.htm
Paraphrased summary: "Working on the Linux port we found that cudaHostAlloc/cuMemHostAlloc CUDA API calls return un-initialized pinned memory. This hole may potentially allow one to examine regions of memory previously used by other programs and Linux kernel. We recommend everybody to stop running CUDA drivers on any multiuser system."
My understanding was that "Normal" malloc returns un-initialized memory, so I don't see what the difference here is...
The way I understand how memory allocation works would allow the following to happen:
-userA runs a program on a system that crunches a bunch of sensitive information. When the calculations are done, the results are written to disk, the processes exits, and userA logs off.
-userB logs in next. userB runs a program that requests all available memory in the system, and writes the content of his un-initialized memory, which contains some of userA's sensitive information that was left in RAM, to disk.
I have to be missing something here. What is it? Is memory zero'd-out somewhere? Is kernel/pinned memory special in a relevant way?
Memory returned by malloc() may be nonzero, but only after being used and freed by other code in the same process. Never another process. The OS is supposed to rigorously enforce memory protections between processes, even after they have exited.
Kernel/pinned memory is only special in that it apparently gave a kernel mode driver the opportunity to break the OS's process protection guarantees.
So no, this is not expected behavior; yes, this was a bug. Kudos to NVIDIA for acting on it so quickly!
The only part that requires root priviledges to install CUDA is the NVIDIA driver. As a result all operations done using NVIDIA compiler and link can be done using regular system calls, and standard compiling (provided you have the proper information -lol-). If any security holes lies there, it remains, wether or not cudaHostAlloc/cuMemHostAlloc is modified.
I am dubious about the first answer seen on this post. The man page for malloc specifies that
the memory is not cleared. The man page for free does not mention any clearing of the memory.
The clearing of memory seems to be in the responsability of the coder of a sensitive section -lol-, that leave the problem of an unexpected (rare) exit. Apart from VMS (good but not widely used OS), I dont think any OS accept the performance cost of a systematic clearing. I am not clear about the way the system may track in the heap of a newly allocated memory what was previously in the process area, and what was not.
My conclusion is: if you need a strict level of privacy, do not use a multi-user system
(or use VMS).