I have a process running on Linux which creates a lot of pThreads (each of them have its own purpose). Let's say by some reason one of threads crashes. Sometimes, crash might be caused by some other thread and it would be good to know what threads were running before the crashed one.
So the question is:
Is there a way to ask Linux scheduler what last threads were scheduled?
Any help is really appreciated.
Thanks.
May be you are aware of the Linux "top" command which can show you all the threads open by your process:
top -H -p "pid of your process"
I may help to identify that how many threads are running which is stopped or crashed.
you will have to make changes in kernel code to gather the scheduling data at each context switch and keep writing in some place in memory, it is somewhat similar to Flight recorder functionality available in PNE kernel.
Related
I recently learned that some OS schedulers will use idle threads (one for each core in multi-core processors) during scheduling so that there will always be at least one runnable thread. I was wondering if macOS did something similar? If so, how can I see how much CPU time the idle thread has had? I've tried using the Activity Monitor to see if I could find anything related to this idle thread or scheduler but I can't see to find anything there. Any pointers would be greatly appreciated!
Original poster here, 2 years and 6 months later (now working on the Platform Kernel team at Apple), following up on this question of idle threads in macOS.
So macOS uses the XNU operating system kernel, which it shares with iOS/iPadOS/tvOS/etc., so the real question here is whether or not the XNU scheduler employs idle threads; the answer to that is yes. You can actually find the bona fide idle thread code in the open source drop of XNU here (search for the idle_thread() function). As was my hunch back then, there is in fact a per-processor idle thread (look here for the idle_thread field of struct processor). As far as I can tell, there's no way to use Activity Monitor to tell how much time various processors are spending executing the idle thread.
Say I'm running a program on a linux machine. I want to be able to give some guarantees about how it will perform. In order to do this, I need to know if the linux kernel gives any guarantees about my process. This is in the same area as timeouts from gc pauses, i.e. when the process stops responding for a while but later comes back online.
Does the linux kernel give any guarantees about for how long a process may be paused, i.e. not executing?
What if there are no processes with a higher priority than the program I'm interested in?
As far as i know Linux does not give any guarantees on anything,
But if it is about your system running out of resources and pausing a process it will depend on the hardware running it.
If your process is the highest is will run unless instructed overwise,
I have never tested this but i beleive you can keep a process paused for as long as you want if you choose to.
Hope this Helped :)
The other day, when doing testing on a Linux server, we observed that under some conditions, one process could die and then started again. After checking the code, we found it was caused by an infinite loop.
This aroused my curiosity how the process went dead and then got started? Is it the OS who detects and determines the abnormal process and get it restarted? If yes, how does that work?
Let's assume you won't be able to fix your code... And let's ignore all crazy options like attaching gdb via script or so.
You can either check CPU usage (most accidental infinite loops that I've done used 100% of CPU for hours :) ), or (more likely option) use strace to check what the software is doing right now and implement your own signature tracing (if those 20 APIs repeats 20 times let's assume infinite loop or so).
For example:
#!/bin/bash
strace -p`cat your_app.pid` | ./your_signature_evaluator
# Or
strace -p12345 | ./your_signature_evaluator
As for automatic system recognition... It seems normal that program crashes after calling things in loop uncontrollably (for example malloc() until you deplete memory, opening files...), but I've (and correct me in comment if I'm wrong) never seen system (kernel) restarting the app. I think you've either:
have conditions (signal handling, whatever) inside program that helps to recover
you're running a watchdog (check every 20 seconds that <pid> is running and if not start new instance)
you're running distribution that provides service/program configuration with restart if stopped
But I really doubt that Linux would be so nice to your application on it's own.
If it could the person that wrote that kernel will have solved the halting problem
PS: Vytor - Web servers are in an infinite loop and do not use 100% CPU.
Fixed:
Well this seems a bit silly. Turns out top was not displaying correctly and programs actually continue to run. Perhaps the CPU time became too large to display? Either way, the program seems to be working fine and this whole question was moot.
Thanks (and sorry for the silly question).
Original Q:
I am running a simulation on a computer running Ubuntu server 10.04.3. Short runs (<24 hours) run fine, but long runs eventually stall. By stall, I mean that the program no longer gets any CPU time, but it still holds all information in memory. In order to run these simulations, I SSH and nohup the program and pipe any output to a file.
Miscellaneous information:
The system is definitely not running out of RAM. The program does not need to read or write to the hard drive until completion; the computation is done completely in memory. The program is not killed, as it still has a PID after it stalls. I am using openmp, but have increased the max number of processes and the max time is unlimited. I am finding the largest eigenvalues of a matrix using the ARPACK fortran library.
Any thoughts on what is causing this behavior or how to resume my currently stalled program?
Thanks
I assume this is an OpenMP program from your tags, though you never actually state this. Is ARPACK threadsafe?
It sounds like you are hitting a deadlock (more common in MPI programs than OpenMP, but it's definitely possible). The first thing to do is to compile with debugging flags on, then the next time you find this problem, attach with a debugger and find out what the various threads are doing. For gdb, for instance, some instructions for switching between threads are shown here.
Next time your program "stalls", attach GDB to it and do thread apply all where.
If all your threads are blocked waiting for some mutex, you have a
deadlock.
If they are waiting for something else (e.g. read), then you need to figure out what prevents the operation from completing.
Generally on UNIX you don't need to rebuild with debug flags on to get a meaningful stack trace. You wouldn't get file/line numbers, but they may not be necessary to diagnose the problem.
A possible way of understanding what a running program (that is, a process) is doing is to attach a debugger to it with gdb program *pid* (which works well only when the program has been compiled with debugging enabled with -g), or to use strace on it, using strace -p *pid*. the strace command is an utility (technically, a specialized debugger built above the ptrace system call interface) which shows you all the system calls done by a program or a process.
There is also a variant, called ltrace that intercepts the call to functions in dynamic libraries.
To get a feeling of it, try for instance strace ls
Of course, strace won't help you much if the running program is not doing any system calls.
Regards.
Basile Starynkevitch
I'm trying to determine the reason for a stalled process on Linux. It's a telecom application, running under fairly heavy load. There is a separate process for each of 8 T1 spans. Every so often, one of the processes will get very unresponsive - up to maybe 50 seconds before an event is noted in the normally very busy process's log.
It is likely some system resource that runs short. The obvious thing - CPU usage - looks to be OK.
Which linux utilities might be best for catching and analyzing this sort of thing, and be as unobtrusive about it as possible, as this is a highly loaded system? It would need to be processes rather than system oriented, it would seem. Maybe ongoing monitoring of /proc/pid/XX? Top wouldn't seem to be too useful here.
If you are able to spot this "moment of unresponsiveness", then you might use strace to attach to the process in question during that time and try to figure out where it "sleeps":
strace -f -o LOG -p <pid>
More lightweight, but less reliable method:
When process hangs, use top/ps/gdp/strace/ltrace to find out the state of the process (e.g. whether it waits in "select" or consumes 100% cpu in some library call)
Knowing the general nature of the call in question, tailor the invocation of strace to log specific syscalls or groups of syscall. For example, to log only file access-related syscalls, use:
strace -e file -f -o LOG ....
If the strace is too heavy a tool for you, try monitoring:
Memory usage with "vmstat 1 > /some/log" - maybe process is being swapped in (or out) during that time
IO usage with vmstat/iotop - maybe some other process is thrashing the disks
/proc/interrupts - maybe driver for your T1 card is experiencing problems?
You can strace the program in question and see what system calls it's making.
Thanks - strace sounds useful. Catching the process at the right time will be part of the fun. I came up with a scheme to periodically write a time stamp into shared memory, then monitor with another process. Sending a SIGSTOP would then let me at least examine the application stack with gdb. I don't know if strace on a paused process will tell me much, but I could maybe then turn on strace and see what it will say. Or turn on strace and hit the process with a SIGCONT.