Related
I am confused about the following:
I am hoping to get a job in the field of embedded systems. However, every interview I've had seems to end up with a conversation about threads in C and how to do thread-safe programming
My question is how do I go about learning multithreading in embedded systems? Are they the same as POSIX threads? For example, the tasks in FreeRTOS. Are they same thing as pthreads?
Can someone give me some tips on what to do and where to start?
Every OS has it's own threads/task/processes characteristics.
Despite the differences, the methods to synchronize, guard and interchange data between those, are roughly the same.
If someone knows that you don't know a specific OS, invited you to an interview - he/she probably expects you to answer in general and not to be OS specific.
You can solve any problem with POSIX (or any other) tool-set in mind and to mention that migration of the solution to a non-POSIX environment will keep same logic with some minor adaptations.
Multithreading concept is almost same everywhere, whether in RTOS or Linux.
The difference is in the operational behavior.
My question is how do I go about learning multithreading in embedded
systems?
My suggestion is to first learn and understand the concepts of multithreading by referring some online material, you can practice by writing some simple codes on your desktop running any flavor of Linux.
The go for some advanced topics like synchronization mechanism using Semaphore and Mutexes, you will then get to learn about the basic concept of when to use a semaphore and when to use a mutex for thread synchronization.
Then move to some Embedded Targets and try out some code using uCOS-II/uCOS-III or FreeRTOS.
Are they the same as POSIX threads?
No, they are not exactly same, POSIX thread library is a bit advance and is highly portable on different OS. For e.g. a multithread code written on Linux using pthread can also be compiled and executed on Windows with little or no change.
On the other hand, a thread implementation on RTOS is different, threads in RTOS are treated as tasks and they start executing only when a call to start the scheduler is made.
From my own experience trying to find learning resources, I found the the FreeRTOS docs very useful. They have both a reference manual as well as the Mastering the FreeRTOS Kernal doc which includes code snippets and covers topics such as task management, software timers, resource management, and general thread safe programming techniques. I dont think this would be the best place to start out, but once you've familiarized yourself with basics the other answers and comments have mentioned, this could help with the next step of learning by doing.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I didn't find any useful information about programming languages for real time systems. All I found was Real Time Systems and Programming Languages: Ada 95, Real-Time Java and Real-Time C/POSIX (some pdf here), which seems to talk about extensions of Java and C for real times systems (I don't have the book to read). Also, the book was published in 2001, and the information may be obsolete now.
So, I'm dubious whether these languages are used in real world applications, or if real time systems in the real world are made in other languages, like DSLs.
If the second option is true for you, what are the outstanding characteristics of the language you use?
I am an avionics software engineer.
I was able to participate in several development projects.
The languages I used in those projects are: C, C++, and Real-time Java.
C is great.
C++ is not so bad but C/C++ require strict coding standards for the safety considerations such as DO-178B.
I think Real-time Java is the way to go but I don't see many avionics applications, yet.
Korean jet trainer T-50 will have a mission computer running RT Java application serving HUD and MFD displays, and all of the mission critical functions.
The Real-Time Specification for Java now has several commercial-grade implementations:
Sun's JavaRTS
IBM's WebSphere Real-Time
Aonix PERC
aicas JamaicaVM
Apogee Aphelion
These products span the continuum from compilation to native code (Aonix) to J2ME (aicas, apogee), to full J2SE (Sun, IBM). Most, if not all, have seen deployments in small numbers of safety- or mission-critical systems, but momentum is building. Examples include Eglin AFB's space surveillance radar modernization and the US Navy's use of RTSJ in the DDG-1000/Zumwalt destroyer. Sun also claims deployment in the financial transaction processing domain.
If you are interested in RTSJ, I suggest Peter Dibble's Real-Time Platform Programming, or Professor Wellings' Concurrent and Real-Time Programming in Java.
On a related note, there is also work underway to provide a Safety-Critical profile for the Java programming language, built as a subset of RTSJ. Also, an expert group has formed to explore a Distributed RTSJ DRTSJ, but the work is stalled.
The book covers use of Ada 95, the Java Real-Time System and realtime POSIX extensions (programmed in C). None of these is directly a domain specific language.
Ada 95 is a programming language commonly used in the late 90s and (AFAIK) still widely used for realtime programming in defence and aerospace industries. There is at least one DSL built on top of Ada - SparkAda - which is a system of annotations which describe system characteristics to a program verification tool.
This interview of April 6, 2006 indicates some of the classes and virtual machine changes which make up the Java Real-Time System. It doesn't mention any domain specific language extensions. I haven't come across use of Java in real-time systems, but I haven't been looking at the sorts of systems where I'd expect to find it (I work in aerospace simulation, where it's C++, Fortran and occasionally Ada for real-time in-the-loop systems).
Realtime POSIX is a set of extensions to the POSIX operating system facilities. As OS extensions, they don't require anything specific in the language. That said, I can think of one C based DSL for describing embedded systems - SystemC - but I've no idea if it's also used to generate the embedded systems.
Not mentioned in the book is Matlab, which in the last few years has gone from a simulation tool to a model driven development system for realtime systems.
Matlab/Simulink is, in effect, a DSL for linear programming, state machines and algorithms. Matlab can generate C or HDL for realtime and embedded systems. It's very rare to see an avionics, EW or other defence industry real-time job advertised which doesn't require some Matlab experience. (I don't work for Matlab, but it's hard to over emphasis how ubiquitous it really is in the industry)
Real time applications can be made in almost any language. The environment (operating system, runtime and runtime libraries) must however be compliant to real time constraints. In most cases real-time means that there's always a deterministic time in which something happens. Deterministic time being ussually a very low time value in the microseconds/milliseconds range.
Real time systems depend solely on this criteria, as the specificiations usually say something like 'Every x (period of time) (do something | check something)'. Usually this happens if the system interfaces with external sensors and controls life-saving or life-threatening systems.
I was working on an in-car navigation and infotainment system developed mostly in C/C++ with an operating system configured specifically to meet the real-time constraints to provide real-time navigation and media playback.
But this is not all to real-time systems: Usually the selection of algorithms in the entire system is made to have deterministic runtimes according to the Big-O notation, mostly using linear or constant time. Everything else is considered non-deterministic and thus not useable for real-time systems.
All of the real-time systems I have worked with were predominantly written in C with some bits of assembler, or written mostly in assembler with little bits of C. (Depending on whether we're talking the 90s and beyond, or the 80s, respectively.) However, some of the real-time systems I've worked with have used -- not exactly DSLs -- special homegrown code generators.
Real-time oriented language?
What is real-time
First we have to define what real-time mean.
Of course depending on how your tool will work against the physical environment pure real-time couldn't be effectively done, mostly because there will be a lot of third party dependencies.
If you are building embed stuff by using microcontrollers like arduino, the language to use will be limited by the hardware, but with more complex stuff like Raspberry Pi, the language choice is very wide.
Granularity
This is depending on what you are measuring, if you're working with:
weather temperatures, one read each 10 minute could be enough
people height or weight, one or maybe four read by day
server status, between 1 second for fine debugging to approx 1 hour for quiet unimportant secondary server.
atomic collision count: something finer...
Event based reading
The right (better) way for collecting data is based on value change event... whenever the device do permit it.
Your tool have to not poll values from device, but the device have to send values to your tool, when they change.
This could be done by using an hardware interrupt trigger or by using port protocole like RS-232 staying listening on some serial port, for sample.
Monitoring environment
The last thing to be warned is how legitimate user will interact with.
If you're building embed standalone device, like robot, you may use graphic libraries to interact with touch screen.
If you're building web based monitor, you may have to keep in mind that the client could be an old 800x600 monochrome screen, using poor internet connection and small processor... But depending on final goal if you may interact with clients, you could ensure strong hardware and strong internet connections. Anyway you have to watch for connexion loosing and event for communication delay between server and client. There is mostly third party dependencies.
Which programming language?
From there, the language choice is wide and clearly depend on
your knowledge.
granularity requested (by using event-based too, of course)
the amount of time you have to build the tool (money;)
delay, co-workers...
kind of device
kind of monitoring
some other political reasons
You could build real-time monitoring engine by using bash and sql only, I've seen sophisticated engines that was built under postgresql only... I've personally built a web based, solar energy monitor by using perl, mysql and javascript.
I cannot believe no one has mentioned LabVIEW programming language which is widely used for Real-time safety-critical systems. It has extensive libraries and well-known design patterns for architecturing and implementing for RT systems.
Also National Instruments makes various hardware (cRIO, PXI and etc) which are designed for real-time applications.
We use LabVIEW for Fracking (Hydraulic Fracturing) which is used in safety-critical environments.
PLCs run ladder and fbd code which is really a real-time dsl in the sense that your options are so limited that it is difficult to program in a way that would result in unpredictable runtime performance
A really purposeful application of the C language to real-time programming - and all related issues (such as parallel programming) - is offered by my Kickstarter
http://www.kickstarter.com/projects/767046121/crawl-space-computing-with-connel
It is called "Wide Programming" and I've been doing it most of my life. The rewards include a software library and a book - designed to be useful.
the company I've been working for since 2003 has been developing and deploying a Scada/Mes platform. Original implementation started in 1993, used Modula2 on OS/2. Later (1998) it was ported to Ada95 and Windows. Currently (2019) we use Ada compiler by AdaCore. Our system was ported and has been deployed to 32/64 Windows, HPUX, OpenVMS (and lately even to Raspberry). We have multiple installation in central Europe (gas industry, refineries, factories, power plants).
We feel Ada's features give our system a high degree of reliability and prevents a lot of errors that would easily occour if we used languages like C.
See also my blog
https://www.ipesoft.com/en/blog/what-language-is-the-d2000-written
Closed. This question is off-topic. It is not currently accepting answers.
Closed 9 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
How do emulators work? When I see NES/SNES or C64 emulators, it astounds me.
Do you have to emulate the processor of those machines by interpreting its particular assembly instructions? What else goes into it? How are they typically designed?
Can you give any advice for someone interested in writing an emulator (particularly a game system)?
Emulation is a multi-faceted area. Here are the basic ideas and functional components. I'm going to break it into pieces and then fill in the details via edits. Many of the things I'm going to describe will require knowledge of the inner workings of processors -- assembly knowledge is necessary. If I'm a bit too vague on certain things, please ask questions so I can continue to improve this answer.
Basic idea:
Emulation works by handling the behavior of the processor and the individual components. You build each individual piece of the system and then connect the pieces much like wires do in hardware.
Processor emulation:
There are three ways of handling processor emulation:
Interpretation
Dynamic recompilation
Static recompilation
With all of these paths, you have the same overall goal: execute a piece of code to modify processor state and interact with 'hardware'. Processor state is a conglomeration of the processor registers, interrupt handlers, etc for a given processor target. For the 6502, you'd have a number of 8-bit integers representing registers: A, X, Y, P, and S; you'd also have a 16-bit PC register.
With interpretation, you start at the IP (instruction pointer -- also called PC, program counter) and read the instruction from memory. Your code parses this instruction and uses this information to alter processor state as specified by your processor. The core problem with interpretation is that it's very slow; each time you handle a given instruction, you have to decode it and perform the requisite operation.
With dynamic recompilation, you iterate over the code much like interpretation, but instead of just executing opcodes, you build up a list of operations. Once you reach a branch instruction, you compile this list of operations to machine code for your host platform, then you cache this compiled code and execute it. Then when you hit a given instruction group again, you only have to execute the code from the cache. (BTW, most people don't actually make a list of instructions but compile them to machine code on the fly -- this makes it more difficult to optimize, but that's out of the scope of this answer, unless enough people are interested)
With static recompilation, you do the same as in dynamic recompilation, but you follow branches. You end up building a chunk of code that represents all of the code in the program, which can then be executed with no further interference. This would be a great mechanism if it weren't for the following problems:
Code that isn't in the program to begin with (e.g. compressed, encrypted, generated/modified at runtime, etc) won't be recompiled, so it won't run
It's been proven that finding all the code in a given binary is equivalent to the Halting problem
These combine to make static recompilation completely infeasible in 99% of cases. For more information, Michael Steil has done some great research into static recompilation -- the best I've seen.
The other side to processor emulation is the way in which you interact with hardware. This really has two sides:
Processor timing
Interrupt handling
Processor timing:
Certain platforms -- especially older consoles like the NES, SNES, etc -- require your emulator to have strict timing to be completely compatible. With the NES, you have the PPU (pixel processing unit) which requires that the CPU put pixels into its memory at precise moments. If you use interpretation, you can easily count cycles and emulate proper timing; with dynamic/static recompilation, things are a /lot/ more complex.
Interrupt handling:
Interrupts are the primary mechanism that the CPU communicates with hardware. Generally, your hardware components will tell the CPU what interrupts it cares about. This is pretty straightforward -- when your code throws a given interrupt, you look at the interrupt handler table and call the proper callback.
Hardware emulation:
There are two sides to emulating a given hardware device:
Emulating the functionality of the device
Emulating the actual device interfaces
Take the case of a hard-drive. The functionality is emulated by creating the backing storage, read/write/format routines, etc. This part is generally very straightforward.
The actual interface of the device is a bit more complex. This is generally some combination of memory mapped registers (e.g. parts of memory that the device watches for changes to do signaling) and interrupts. For a hard-drive, you may have a memory mapped area where you place read commands, writes, etc, then read this data back.
I'd go into more detail, but there are a million ways you can go with it. If you have any specific questions here, feel free to ask and I'll add the info.
Resources:
I think I've given a pretty good intro here, but there are a ton of additional areas. I'm more than happy to help with any questions; I've been very vague in most of this simply due to the immense complexity.
Obligatory Wikipedia links:
Emulator
Dynamic recompilation
General emulation resources:
Zophar -- This is where I got my start with emulation, first downloading emulators and eventually plundering their immense archives of documentation. This is the absolute best resource you can possibly have.
NGEmu -- Not many direct resources, but their forums are unbeatable.
RomHacking.net -- The documents section contains resources regarding machine architecture for popular consoles
Emulator projects to reference:
IronBabel -- This is an emulation platform for .NET, written in Nemerle and recompiles code to C# on the fly. Disclaimer: This is my project, so pardon the shameless plug.
BSnes -- An awesome SNES emulator with the goal of cycle-perfect accuracy.
MAME -- The arcade emulator. Great reference.
6502asm.com -- This is a JavaScript 6502 emulator with a cool little forum.
dynarec'd 6502asm -- This is a little hack I did over a day or two. I took the existing emulator from 6502asm.com and changed it to dynamically recompile the code to JavaScript for massive speed increases.
Processor recompilation references:
The research into static recompilation done by Michael Steil (referenced above) culminated in this paper and you can find source and such here.
Addendum:
It's been well over a year since this answer was submitted and with all the attention it's been getting, I figured it's time to update some things.
Perhaps the most exciting thing in emulation right now is libcpu, started by the aforementioned Michael Steil. It's a library intended to support a large number of CPU cores, which use LLVM for recompilation (static and dynamic!). It's got huge potential, and I think it'll do great things for emulation.
emu-docs has also been brought to my attention, which houses a great repository of system documentation, which is very useful for emulation purposes. I haven't spent much time there, but it looks like they have a lot of great resources.
I'm glad this post has been helpful, and I'm hoping I can get off my arse and finish up my book on the subject by the end of the year/early next year.
A guy named Victor Moya del Barrio wrote his thesis on this topic. A lot of good information on 152 pages. You can download the PDF here.
If you don't want to register with scribd, you can google for the PDF title, "Study of the techniques for emulation programming". There are a couple of different sources for the PDF.
Emulation may seem daunting but is actually quite easier than simulating.
Any processor typically has a well-written specification that describes states, interactions, etc.
If you did not care about performance at all, then you could easily emulate most older processors using very elegant object oriented programs. For example, an X86 processor would need something to maintain the state of registers (easy), something to maintain the state of memory (easy), and something that would take each incoming command and apply it to the current state of the machine. If you really wanted accuracy, you would also emulate memory translations, caching, etc., but that is doable.
In fact, many microchip and CPU manufacturers test programs against an emulator of the chip and then against the chip itself, which helps them find out if there are issues in the specifications of the chip, or in the actual implementation of the chip in hardware. For example, it is possible to write a chip specification that would result in deadlocks, and when a deadline occurs in the hardware it's important to see if it could be reproduced in the specification since that indicates a greater problem than something in the chip implementation.
Of course, emulators for video games usually care about performance so they don't use naive implementations, and they also include code that interfaces with the host system's OS, for example to use drawing and sound.
Considering the very slow performance of old video games (NES/SNES, etc.), emulation is quite easy on modern systems. In fact, it's even more amazing that you could just download a set of every SNES game ever or any Atari 2600 game ever, considering that when these systems were popular having free access to every cartridge would have been a dream come true.
I know that this question is a bit old, but I would like to add something to the discussion. Most of the answers here center around emulators interpreting the machine instructions of the systems they emulate.
However, there is a very well-known exception to this called "UltraHLE" (WIKIpedia article). UltraHLE, one of the most famous emulators ever created, emulated commercial Nintendo 64 games (with decent performance on home computers) at a time when it was widely considered impossible to do so. As a matter of fact, Nintendo was still producing new titles for the Nintendo 64 when UltraHLE was created!
For the first time, I saw articles about emulators in print magazines where before, I had only seen them discussed on the web.
The concept of UltraHLE was to make possible the impossible by emulating C library calls instead of machine level calls.
Something worth taking a look at is Imran Nazar's attempt at writing a Gameboy emulator in JavaScript.
Having created my own emulator of the BBC Microcomputer of the 80s (type VBeeb into Google), there are a number of things to know.
You're not emulating the real thing as such, that would be a replica. Instead, you're emulating State. A good example is a calculator, the real thing has buttons, screen, case etc. But to emulate a calculator you only need to emulate whether buttons are up or down, which segments of LCD are on, etc. Basically, a set of numbers representing all the possible combinations of things that can change in a calculator.
You only need the interface of the emulator to appear and behave like the real thing. The more convincing this is the closer the emulation is. What goes on behind the scenes can be anything you like. But, for ease of writing an emulator, there is a mental mapping that happens between the real system, i.e. chips, displays, keyboards, circuit boards, and the abstract computer code.
To emulate a computer system, it's easiest to break it up into smaller chunks and emulate those chunks individually. Then string the whole lot together for the finished product. Much like a set of black boxes with inputs and outputs, which lends itself beautifully to object oriented programming. You can further subdivide these chunks to make life easier.
Practically speaking, you're generally looking to write for speed and fidelity of emulation. This is because software on the target system will (may) run more slowly than the original hardware on the source system. That may constrain the choice of programming language, compilers, target system etc.
Further to that you have to circumscribe what you're prepared to emulate, for example its not necessary to emulate the voltage state of transistors in a microprocessor, but its probably necessary to emulate the state of the register set of the microprocessor.
Generally speaking the smaller the level of detail of emulation, the more fidelity you'll get to the original system.
Finally, information for older systems may be incomplete or non-existent. So getting hold of original equipment is essential, or at least prising apart another good emulator that someone else has written!
Yes, you have to interpret the whole binary machine code mess "by hand". Not only that, most of the time you also have to simulate some exotic hardware that doesn't have an equivalent on the target machine.
The simple approach is to interpret the instructions one-by-one. That works well, but it's slow. A faster approach is recompilation - translating the source machine code to target machine code. This is more complicated, as most instructions will not map one-to-one. Instead you will have to make elaborate work-arounds that involve additional code. But in the end it's much faster. Most modern emulators do this.
When you develop an emulator you are interpreting the processor assembly that the system is working on (Z80, 8080, PS CPU, etc.).
You also need to emulate all peripherals that the system has (video output, controller).
You should start writing emulators for the simpe systems like the good old Game Boy (that use a Z80 processor, am I not not mistaking) OR for C64.
Emulator are very hard to create since there are many hacks (as in unusual
effects), timing issues, etc that you need to simulate.
For an example of this, see http://queue.acm.org/detail.cfm?id=1755886.
That will also show you why you ‘need’ a multi-GHz CPU for emulating a 1MHz one.
Also check out Darek Mihocka's Emulators.com for great advice on instruction-level optimization for JITs, and many other goodies on building efficient emulators.
I've never done anything so fancy as to emulate a game console but I did take a course once where the assignment was to write an emulator for the machine described in Andrew Tanenbaums Structured Computer Organization. That was fun an gave me a lot of aha moments. You might want to pick that book up before diving in to writing a real emulator.
Advice on emulating a real system or your own thing?
I can say that emulators work by emulating the ENTIRE hardware. Maybe not down to the circuit (as moving bits around like the HW would do. Moving the byte is the end result so copying the byte is fine). Emulator are very hard to create since there are many hacks (as in unusual effects), timing issues, etc that you need to simulate. If one (input) piece is wrong the entire system can do down or at best have a bug/glitch.
The Shared Source Device Emulator contains buildable source code to a PocketPC/Smartphone emulator (Requires Visual Studio, runs on Windows). I worked on V1 and V2 of the binary release.
It tackles many emulation issues:
- efficient address translation from guest virtual to guest physical to host virtual
- JIT compilation of guest code
- simulation of peripheral devices such as network adapters, touchscreen and audio
- UI integration, for host keyboard and mouse
- save/restore of state, for simulation of resume from low-power mode
To add the answer provided by #Cody Brocious
In the context of virtualization where you are emulating a new system(CPU , I/O etc ) to a virtual machine we can see the following categories of emulators.
Interpretation: bochs is an example of interpreter , it is a x86 PC emulator,it takes each instruction from guest system translates it in another set of instruction( of the host ISA) to produce the intended effect.Yes it is very slow , it doesn't cache anything so every instruction goes through the same cycle.
Dynamic emalator: Qemu is a dynamic emulator. It does on the fly translation of guest instruction also caches results.The best part is that executes as many instructions as possible directly on the host system so that emulation is faster. Also as mentioned by Cody, it divides the code into blocks ( 1 single flow of execution).
Static emulator: As far I know there are no static emulator that can be helpful in virtualization.
How I would start emulation.
1.Get books based around low level programming, you'll need it for the "pretend" operating system of the Nintendo...game boy...
2.Get books on emulation specifically, and maybe os development. (you won't be making an os, but the closest to it.
3.look at some open source emulators, especially ones of the system you want to make an emulator for.
4.copy snippets of the more complex code into your IDE/compliler. This will save you writing out long code. This is what I do for os development, use a district of linux
I wrote an article about emulating the Chip-8 system in JavaScript.
It's a great place to start as the system isn't very complicated, but you still learn how opcodes, the stack, registers, etc work.
I will be writing a longer guide soon for the NES.
I'm using the RxTx library over usbserial on a Linux distro. The RxTx lib seems to behave quite differently (in a bad way) than how it works over serial.
One of my biggest problems is that the RxTx SerialPortEvent.OUTPUT_BUFFER_EMPTY does not work on linux over usb serial.
How do I know when I should write to the stream? Any indicators I might have missed?
So far my experience with writing and reading concurrently have not been great. Does anyone know if I should lock the DATA_AVAILABLE handler from being invoked while I'm writing on the stream? Or RxTx accepts concurrent read/writes?
(perhaps slightly off-topic, but here goes)
I'm not familiar with that particular library, but I can assure you from dire experience (I work in the security systems (as in: hardware security devices) business, where RS-232 is heavily used) that not all USB-serial converters are born equal. Many such devices so not properly emulate all RS-232 lines, and many don't even handle any comms without flow control. Before blaming the library, try to confirm that the hardware actually does what it's supposed to do.
Without wanting to endorse a particular product or brand, the best (as in: least buggy) USB-serial converter I have come across in years is the USA-19HS.
Using RxTx over usb-to-serial you can't set notifyOnOutput to true otherwise it locks up completely.
I've learned this the hard way. This problem is documented on a few web sites over the internet.
I'm running it on Linux and I believe that this is a Linux only issue, although I can't confirm that.
As for the link you've given me... I've seen the SimpleReader and SimpleWriter examples, but these don't represent a real world application. It is not multi-threaded, assumes a read has the full data it needs instead of buffering reads, etc.
Thanks,
Jeach!
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I'd like to gain better knowledge of operating system internals. Process management, memory management, and stuff like that.
I was thinking of learning by getting to know either linux or BSD kernel.
Which one kernel is better for learning purposes?
What's the best place to start?
Can you recommend any good books?
In college, I had an operating systems class where we used a book by Tanenbaum. In the class, we implemented a device driver in the Minix operating system. It was a lot of fun, and we learned a lot.
One thing to note though, if you pick Minix, it is designed for learning. It is a microkernel, while Linux and BSD are a monolithic kernel, so what you learn may not be 100% translatable to be able to work with Linux or BSD, but you can still gain a lot out of it, without having to process quite as much information.
As a side note, if you've read Just for Fun, Linus actually was playing with Minix before he wrote Linux, but it just wasn't enough for his purposes.
As a Linux user I'd say Linux has a great community for people to learn about the kernel. http://kernelnewbies.org is a great place to start asking questions and learning about how the kernel works. I can't make a book reccomendation, but once you've read the starting material on kernelnewbies the source is very well documented.
Aside from the good books already mentioned (Opeating System Design & Implementation is particularly good), get a hold of a 1.x release Linux Kernel, load it into VMWare or VirtualBox and start playing around from there.
You will need to spend a lot of time browsing source code. For this, check out http://lxr.linux.no/ which is a browsable linked version of the source and makes life a lot easier. For the very first version of Linux (0.01) check out http://lxr.linux.no/linux-old+v0.01/. The fun begins at http://lxr.linux.no/linux-old+v0.01/boot/boot.s. As you progress from version to version, check out the ChangeLog and dig into those parts that have changed to save you re-reading the whole thing again.
Once you've gotten a hold of the concepts, look at 2.0, then 2.2, etc. Be prepared to sink A LOT of time into the process.
Linux
Device Drivers
Linux Core Kernel Commentary
Operating Systems Design and Implementation
I had previously bought these books on recommendation for the same purpose but I never got to studying them myself so only take them as second-hand advice.
I recommend you the BSD kernels! BSD kernels have far fewer hackers so following their evolution is easier. Either BSD and Linux kernels have great hackers, but some people argue that BSD lower fame filters out novice ones. Also taking design decisions is easier when the sources are not being updated 100 times a day.
Among the BSD choices, my favorite one is NetBSD. It might not be the pain-free choice you want for your desktop, but because it has a strong focus on portability, the quality is quite good. I think this part say it all:
Some systems seem to have the philosophy of “If it works, it's right”. In that light NetBSD's philosophy could be described as “It doesn't work unless it's right”
If you have been working long enough, you will know that NetBSD is a quite joy for learning good coding. Although professionally you will find more chances with Linux
Whichever choice you take, start joining their mail lists, follow the discussions. Study some patches and finally try to do your own bug-fixing. Regarding books, search for Diomidis Spinellis articles and his book. It is not exactly a kernel book, but has NetBSD examples and helps a lot to tackle large software.
Noting the lack of BSDs here, I figured I'd chip in:
The Design and Implementation of the FreeBSD Operating System (dead-tree book)
Unix and BSD Courses (courses and videos)
FreeBSD Architecture Handbook (online book)
I haven't taken any of the courses myself, but I've heard Marshall Kirk McKusick speak on other occasions, and he is really good at what he does.
And of course the BSD man pages, which are an excellent resource as they are maintained to a far greater extent than your average Linux man-page. Take for instance the uvm(9) man-page, describing the virtual memory interface in OpenBSD.
Not quite related, but I'll also recommend the video History of the Berkeley Software Distributions as it gives a nice introduction to the BSD parts of the UNIX history and culture as well as plenty of hilarious anectodes from back when.
There's no substitute for diving into the code. Try to find a driver or subsystem that you're interested in and poke around with it. With tools like VMware Workstation it's super easy to make whatever changes you want, snapshot the VM, and run your modified kernel. If the kernel panics on boot, who cares? Just jump back to the snapshot and fix the problem.
For books, I strongly recommend Linux Kernel Development by Robert Love. It's a wonderfully written book -- lots of information, organized sanely, and humorous... not dry reading at all.
Take Mike Stone's advice and start with Minix. That's what Linus did! The textbook is really well written, and Tannenbaum does a great job of showing how the various features are implemented in a real system.
Nobody seems to have mentioned that code-wise BSD is much cleaner and more consistent. The documentation's way better too (as already mentioned). But since there's a whole lot of fiddling with whatever system you choose - I'd pick the one you use more often.
Linux and Minix are fun to learn. If you also want to learn how a modern micro-kernel operating system looks like, you can look at QNX. The complete documentation is available online and it is very accessible. For example, this online book.
When I was at uni I spent a semester studying operating systems, and as part of this had an assignment where we had to implement a RAM-based filesystem in Linux.
It was a fantastic way to get to understand the internals of the Linux keurnel and to get a grasp on how everything fits together - And a heck of a lot of fun playing around with how it interacts with standard tools too.
I haven't tried it myself, but you can go to Linux From Scratch and start building your own Linux distribution. Sounds like something that'll take a junkload of time, but will result in an intimate knowledge of the guts of the Linux kernel and how each part works. Of course, you can supplement this learning by following any of the other tips here.