Kernel Programming From Beginner to Expert using C - programming-languages

Hello i am new to Kernel Programming and i would you to answer me how and how much it could take me to get started with confidence to play with the code and the meanings.
I am listing some Questions that they are also my Goals.Please answer any of them if not all.
1.I would like to evaluate myself on "how much" C i know in order to be able to understand the code and general concepts of the source code.
How i could evaluate myself to find my skill level of programming in C?
I have read K&R but i don't yet have a serious understanding on pointers and structs. When i mean serious i mean i am not able to easily comprehend the source code,yet.
Where i could find puzzles,projects in order to gain real world experience?
What books do you suggest in order to solidfy my knowledge in C programming?
How long it could take me to be a Competent Programmer(if i am not already?)
Expert
Professional
Competent
Advanced Beginner
Novice
A path of books or links well explained in time&skill level order it would be great.
2.I would like to be able to play with the code and create some new in order to experiment with the Kernel.
A path of books or links, Kernel Oriented, well explained in time&skill level order it would be great help.
Some suggested mailing lists and communites in general ?
How long it could take me to be a Competent Kernel Programmer?
(see list above about skill levels)
3. Some general suggestion about
those subsjects?

These might help getting you there:
The C Programming Language
Understanding the Linux Kernel
Linux Device Drivers
Maybe you should start with something other than the kernel first. Take any open source project written in C, get the source, try to fix open bugs or implement missing features. That will teach you a lot about C, and you'll get feedback about the code you write.
Once you're fluent in C you can start diving into the kernel source.

Related

How to proceed with Linux source code customization?

I am a non CS/IT student, but having knowledge of C, Java, DS and Algorithms. Now-a-days I am focusing on operating system and had gained some of its concepts. But I want some practical knowledge of it. Merely writing algo code in java/c has no fun in doing. I have gone through many articles where they mentioned we can customize source code of Linux-kernel.
I want to start customizing the kernel as I move ahead in the learning of OS concepts and apply the same. It will make two goals achievable 1. I will gain practical idea of the operating system 2. I will have a project.
Problem which I face-
1. From where to get the source code? Which source code should I download? Also the documentation if possible.
https://www.kernel.org/
I went in there but there are so many of them which one will be better?
2. How will I customize the code once I have it?
Please give me suggestions with detail about how I should start this journey (of changing source code to customize Linux).
Moreover I am using Windows 8.
I recommend first reading several books on OSes and on programming. You need a broad CS culture (if possible get a CS degree)
I am a non CS/IT student,
You'll better become one, or else spend years of work to learn all the stuff a CS graduate student has learnt.
First, you need to be very familiar with Linux programming on user side (application programs). So read at least Advanced Linux Programming and study the source code of several programs, including shells (and some kind of servers). Read also carefully syscalls(2). Explore the state of your kernel (e.g. thru proc(5)...). Look into https://kernelnewbies.org/
I also recommend learning several programming languages. You should in particular read SICP, an excellent introduction to programming. Read also some book like programming language pragmatics. Read something about continuation and continuation passing style. Read the Dragon book. Read some Introduction to Algorithms. Read something about computer architecture and instruction set architecture
Merely writing algo code in java/c has no fun in doing.
But the kernel is also written in C (mostly) and full of algorithmic code. What makes you think you'll get more fun in it?
I want to start customizing the kernel as I move ahead in the learning of OS concepts and apply the same.
But why? Why don't you also consider studying and contributing to some user-level code
I would recommend first reading a good book on OSes in general, notably Operating Systems: Three Easy Pieces. Look also on OSdev.
At last, the general advice about kernel programming is don't. A common mistake is to try adding code inside the kernel to solve some issue that can and should be solved in user-land.
How will I customize the code once I have it?
You probably should not customize the kernel, but if you did you'll use familiar tools (a good source code editor like emacs or vim, a compiler and linker on the command line, a build automation tool like make). Patching the kernel is similar to patching some other free software. But testing your kernel is harder (because you'll often reboot).
You'll also find several books explaining the Linux kernel.
If you still want to customize the kernel you should first try to code some kernel module.
Moreover I am using Windows 8.
This is a huge mistake. You first need to be an advanced Linux user. So wipe out Windows from your computer, and install some Linux distribution -I recommend Debian- (and use only Linux, no more Windows). Become familiar with command line.
I seriously recommend to avoid working on the kernel as your first project.
I strongly recommend looking at some existing user-land free software project first (there are thousands of them, notably on github, e.g. choose some package in your distribution, study its source code, work on it, propose the patch to the community). Be able to build from source code a lot of things.
A wise man once said you "must act your way into right thinking, as you cannot think your way into right acting". In your case, you'll need to act as an experienced programmer would act, which means before we write any code, we need to answer some questions.
What do we want to change?
Why do we want to change it?
What are the repercussions of this change (ie what other functions - out of all the 10's of millions of lines of source code - call this function)?
After we've made the change, how are we going to compile it? In other words, there is a defined process for this. What is it?
After we compile our new kernel/module, how are we going to test it?
A good start, in addition to the answer that was just posted, would be to run LFS (Linux from Scratch). Get a successful install of that and use it as a starting point.
Now, since we're experienced programmers, we know that tinkering with a 10M+ line codebase is a recipe for trouble; we need a bit more direction than that. Here's a list of bugs that need to be fixed: https://bugzilla.kernel.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=7d
I, for one, would be glad to see the one called "AUFS hangs on fanotify" go away, as I use AUFS with Docker on a daily basis.
If, down the line, you decide you'd rather hack on something besides the kernel, there are plenty of other options.
From your question it follows that you've already gained some concepts of an operating system. However, if you feel that it's still insufficient, it is OK to spend more time on learning. An operating system (mainly, a kernel) has certain tasks to perform like memory management (or memory protection), multiprogramming, hardware abstraction and so on. Neither of the topics may be neglected - they are all as important. So, if you have some time, you may refer to such useful books as "Modern Operating Systems" by Andrew Tanenbaum. Special books like that will shed much light on all important aspects of a modern OS. Suffice it to say, Linux kernel itself was started by Linus Torvalds because of a strong inspiration by MINIX - an educational project by A. Tanenbaum.
Such a cumbersome project like an OS kernel (BSD, Linux, etc.) contains lots of code. Many people are collaborating to write or enhance whatever parts of the kernel. So, there is a common and inevitable need to use a version control system. So, if you have an intention to submit your code to the kernel in future, you also have to have hands on with version control. Particularly, Linux relies on Git SCM (software configuration management - a synonym for version control).
So, once you have some knowledge of Git, you can install it on your computer and download Linux source code: git clone https://github.com/torvalds/linux.git
Determine your goals at Linux kernel modification. What do you want to achieve? Perhaps, you have a network card which you suspect to miss some features in Linux? Take a look at the other vendors' drivers and make an attempt to fix the driver of interest to include the features. Of course, this will require some knowledge of the HW, and, if the features are HW dependent, you will unlikely succeed to elaborate your code without special knowledge. But, in general, - if you are trying to make an enhancement, it assumes that you are an experienced Linux user yourself. Otherwise, how will you understand that some fixes/enhancements/etc. are required? So, I can't help but agree with the proposal to postpone Windows 8 for a while and start using some Linux distribution (eg. Debian).
If you succeed to determine your goals (eg. if you find a paper describing some desired changes in Linux kernel or if you decide to enhance some device drivers / write your own), you will be able to try it hands on. However, you still might need some helpful books, but, in this case, some Linux-specific ones. Also, writing C code for the kernel itself will require one important detail - you will need to comply with a so called coding standard, otherwise Linux kernel maintainers will not be able to accept your patches.
So, I made an attempt to outline some tips based on your current question. Of course, the job of kernel development has far more broad prerequisites, but these are which are just obvious.

Advice on learning new technology, first year COOP

I'm a second year Computer Science student, and am currently applying for jobs for the Summer '11 coop term, this will be my first.
I have found a lot of the jobs 'perfect' candidate have many technical skills I do not possess.
For example, one which I have an interview Monday for, uses Java Enterprise Edition, but I've only used regular Java. Also many things I have no experience with whatsoever, like XML, Adobe FLEX, and Ruby.
Obviously the employer is not expecting a potential employee to have all of these skills, but I was wondering how difficult it would be to pick them up?
I have a strong knowledge of C and Java, and of many concepts and data structures.
With this in mind, is it difficult to pick up languages like Ruby or less related tech like XML or AJAX for example?
Or if I have a good background in compsci, do the concepts apply broadly, and mostly it's just the syntax and rudimentary concepts I will need to pick up to get started?
Please if you have any advice, feel free to share.
Thanks for all your help!
PS: I noticed that most people seem to call Java Enterprise 'J2EE' isnt it now just 'Java EE'? Whats up with that?
Once you've mastered one programming language, you'll easily catch up on a whole family of related languages. The differences are more than just syntax, but there is more or less a fixed set of features that most of the popular languages provide.
A good way to gather programming experience would be to become very proficient in at least one language. I don't know what your experience with C is, but unless it's over 3 years it won't take you much time to get to the same level in any other language, and it won't be too hard anyway. If you ask me, knowing C is a "must" because you learn a little about how the computer executes your programs. Some of the higher-level languages have been (initially) implemented in C themselves.
I'm not a Ruby programmer but i'm guessing it's not a big deal if you already know some C and Java. It's probably even easier to learn than those two. XML is just a data format, and AJAX is a buzzword for "doing stuff dynamically in the browser-based client side of a web application". So there isn't much "tech" to learn here, it's just a matter of knowing and mixing skills to get that kind of thing done. (Basically you'll need to learn at least a little bit about javascript, HTML and HTTP, and know how browsers and web servers work).
Computer science is a branch of mathematics, and as such it will always be relevant. Learning about specific algorithms, data structures, etc. is important but more important (from a practical point of view) is the knowledge of how algorithms or data structures perform, how they can be analyzed, what affects their performance, etc. If you got the basics, you can always open a book by yourself and learn more algorithms and data structures, and it's probably a good idea.
Finally, a useful set of things to know in today's software world is: networking (esp. TCP/IP and HTTP), C, Java / C# or both, minimal knowledge of Javascript, and a little experience with programmatic access to XML

Planned self-learning curriculum?

I'm currently taking a year off between high school and college. I'm working as a junior IT technician, so I'm getting plenty of experience on the hardware side of things. I want to use this year off to also get started on some programming; I have experience in Visual basic from high school courses, but want to further my learning before going to school. Now, obviously I will not be able to become overly proficient in all of these, but these are the languages that I plan on learning over the course of the next few years:
PHP
Ruby (on Rails)
Python
(Objective) C__ (I'll research my college program and see what C they use, and learn that)
Java
Lisp
Will being proficient in these languages give me a good base to work from? I tried to pick a selection of languages that seem to offer good employability, ability to develop on a number of platforms (desktop, web, mobile), and ones that are currently popular and sought-after.
Am I missing anything? Does anyone see anything important that I've missed, things I've picked that are a waste of time, or otherwise?
Thanks a lot guys.
I don't know if you've read the Pragmatic Programmer (a great read) But there's a section in there on expanding your knowledge portfolio, and they suggest learning one language a year, and I must confess I agree with them.
So I would work out what you want to write, and then pick the language that fulfills the requirements you want. And as you're going to college, I would also consider what language that the course you're attending is going to be teaching. I'd most certainly place my effort in to learning that language.
I'd also recommend (assuming they're going to let you use linux) learning a scripting language, such as BASH, and learn to create make or build files in your chosen environment, it's a heck of a lot easier than remembering compiler options.
Python is good to start with and then do Java. It would be enough for a starter on my opinion.
If you thought of learing php instead then learn mysql too. And more over you have overall missed the database. :(
Atleast learn my sql or ms sql of your choice.

Learning a language while on a project? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
Have you ever tried learning a language while on a project? I have, and from my personal experience I can say that it takes courage, effort, time, thinking, lots of caffeine and no sleep. Sometimes this has to be done without choice, other times you choose to do it; if you are working on a personal project for example.
What I normally do in this kind of situation, and I believe everyone does, is "build" on top of my current knowledge of languages, structures, syntax and logic. What I find difficult to cope with, is the difference of integrity in some cases. Some languages offer a good background for future learning and "language study", they pose as a good source of information or a frame of reference and can give a "firm" grasp of what's to come. Other languages form or introduce a new way of thinking and are harder to get used to.
Sometimes you unintentionally think in a specific language and when introduced to a new way of thinking, a new language, can cause confusion or make you get lost between the "borders" of your new and your current knowledge of languages.
What can be a good solution in this case? What should be used to broaden the knowledge of the new language, a new way of thinking, and maintain or incorporate the current knowledge of other languages inside the "borders" of the new language?
I find I need to do a project to properly learn a language, but those can be personal projects. When I learned Python on the job, I first expected (and found) a significant slowdown in my productivity for a while. I read the standard tutorials, coding standards and I lurked on the Python list for a while, which gave me a much better idea of the best practices of the language.
Doing things like coding dojos and stuff when learning a language can help you get a feel for things. I just recently changed jobs and went back to Java, and I spent some time working on toy programs just to get back in the feel for things (I'm also reading Effective Java, 2nd edition as my previous major experience had been with Java 1.4).
I think, in some respects no matter what the impetus for learning the language, you have to start by imitating good patterns in the new language. Whether that means finding a good book, with excellent code examples, good on-line tutorials, or following the lead of a more experienced developer, you have to absorb what it means to write good code in a particular language first. Once you have developed a level of comfort, you can start branching out and and experimenting with alternatives to the patterns that you've learned, looking for ways to apply things you've learned from other languages, but keeping within the "rules" of the language. Eventually, you'll get to the point where you know you can 'break the rules" that you learned earlier because you have enough experience to know when they do/don't apply.
My personal preference, even when forced to learn a new language, is to start with some throw away code. Even starting from good tutorials, you'll undoubtedly write code that later you will look back on and not understand how stupid you could have been. I prefer, if possible, to write as my first foray into a language code that will be thrown away and not come back to haunt me later. The alternative is to spend a lot of time refactoring as you learn more and more. Eventually, you'll end up doing this, too.
I would like to mention ALT.NET here
Self-organizing, ad-hoc community of developers bound by a desire to improve ourselves, challenge assumptions, and help each other pursue excellence in the practice of software development.
So in the spirit of ALT.NET, it is challenging but useful to reach out of your comfort zone to learn new languages. Some things that really helped me are as follows:
Understand the history behind a language or script. Knowing evolution helps a lot.
Pick the right book. Research StackOverflow and Amazon.com to find the right book to help you ease the growing pains.
OOP is fairly common in most of the mature languages, so you can skip many of the chapters related to OOP in many books. Syntax learning will be a gradual process. I commonly bookmark some quick handy guides for that.
Read as many community forums as possible to understand the common pitfalls of the new language.
Attend some local meetups to interact with the community and share your pains.
Take one pitch at a time by building small not so complicated applications and thereby gaining momentum.
Make sure you create a reference frame for what you need to learn. Things like how security, logging, multithreading are handled.
Be Open minded, you can be critical, but if you hate something then do not learn that language.
Finally, I think it is worthwhile to learn one strong languages like C# or Java, one functional language and one scripting language like ruby or python.
These things helped me tremendously and I think will help all software engineers and architects to really gear for any development environment.
I learned PHP after I was hired to be the project lead on the Zend Framework project.
It helped that I had 20 years of professional programming background, and good knowledge of C, Java, Perl, JavaScript, SQL, etc. I've also gravitated towards dynamic scripting languages for most of my career. I've written applications in awk, frameworks in shell, macro packages in troff, I even wrote a forum using only sed.
Things to help learn a language on the job:
Reading code and documentation.
Listening to mailing lists and blogs of the community.
Talking to experts in the language, fortunately several of whom were my immediate teammates.
Writing practice code, and asked for code reviews and coaching.(Zend_Console_Getopt was my first significant PHP contribution).
Learning the tools that go along with the language. PHPUnit, Xdebug, phpDoc, phing, etc.
Of course I did apply what I knew from other programming languages. Many computer science concepts are language-universal. The differences of a given language are often simply idiomatic, a way of stating something that can be done another way in another language. This is especially true for languages like Perl or PHP, which both borrow a lot of idioms from earlier languages.
It also helped that I took courses in Compiler Design in college. Having a good foundation in how languages are constructed makes it easier to pick up new languages. At some level, they're all just ways of abstracting runtime stacks and object references.
If you're a junior member of the team and don't know the language, this is not necessarily an issue at all. As long as there is some code review and supervision, you can be a productive.
Language syntax is one issue, but architectural differences are a more important concern. Many languages are also development platforms, and if you don't have experience with the platform, you don't know how to create a viable solution architecture. So if you're the project lead or working solo, you'd better have some experience on the platform before you do your design work.
For example, I would say an experienced C# coder with no VB experience would probably survive a VB.NET project just fine. In fact, it would be more difficult for a developer who only had experience in C#/ASP.NET to complete a C# WPF project than a VB ASP.NET project. An experienced PHP developer might hesitate a bit on a ColdFusion project, but they probably won't make any serious blunders because they are familiar with a script based web development architecture.
Many concepts, such as object modelling and database query strategies, translate just fine between languages. But there is always a learning curve for a new platform, and sometimes it can be quite nasty. The worst case is that the project must be thrown out because the architure is too wrong to refactor.
I like to learn a new language while working on a project, because a real project will usually force me to learn aspects of the language that I might otherwise skip. One of the first things I like to to is read code in that language, and jump in. I find resources (such as books and various internet sites) to help as I go along.
Then, after I've been working on it for a while, I like to read (or re-read) books or other resources on the language. By this time I have some knowledge, so this will help solidify some things and also point out areas where I am flat-out wrong in my understanding. For instance, I can see that I was making incorrect assumptions about similarities between languages.
This also applies to tools -- after using a tool for a while and learning the basics, reading (or skimming) the documentation can teach me a lot.
In my opinion, you should try to avoid that. I know, most of the times you can't but in any case try not to mix the new language with the old one, and never add to the mixture old habits, practices and patterns.
Always try to find resources that will help you get through the new language in the way the language works, not in the way other languages do; that will never have a happy ending, and if it does it will be very hard to modify it to the right way.
Cheers.
Yes I have.
I mean, is there another way? The only language I ever learned that was not on a project was ABC basic, which was what you used on my first computer.
I would recommend if you start with a certain language, stick with it. I only say that because many times in the past I tried more and more different ones, and the one I started out with was the best :D
Everytime I have/want to learn a new language, I force myself to find something to code.
But to be sure I did it well, I always want to be able to check my code and what it ouputs.
To do so, I just try to do the same kind of stuff with languages I know and to compare the outputs. For that, I created a little project (hosted on Github) with an exercise sheet and the correction for every language I learnt. It's a good way to learn in my opinion because it gives you a real little project.

Expert System engine

Next year will be my graduate year to be an informatics engineering person and I am trying to find ideas about the jounior project. Actually, I have an idea of making an expert system engine. I worked with clips and prolog and I liked clips very much but it seems to be an old engine. Can any one advice me about this idea or give me sources for papers or any topics that may help me? I am thinking to use C language to obtain the high performance, and to build a robust data structure. Also, I am thinking about an idea (I dont know if it could be done) of writing facts and rules (like clips) and then generate a C++ optimal code from these rules such that I can obtain the speed of the machine and use exe file.
I need help to make this idea more clear and how it can be done. Specially because I read about fuzzy logic, nueral network and heard about the new generation of expert system, so I dont know how that can be related to such topic.
For your junior project, I would recommend against writing it in C. Your problem sounds like it needs correctness more than it needs speed. Writing it in C will take longer because you will need to implement a lot of primitives that are not included in the language or any standard library. Also, since C is relatively low-level, there are a lot of opportunities to make low-level mistakes. Write it in a higher level language that is closer to the problem domain. You will have more time to focus on your actual problem because you will spend less time getting the framework set up. If you already know Prolog, it would be good to stick with that. Perhaps you might consider Mercury. It is similar to Prolog, but also designed for speed.
JBoss Rules (also known as Drools) offers the best approach to rule-processing. It's written in Java. It allows you to integrate program components in the rules, and rule-bases into your program components. You can even build or modify rule-bases on the fly.
I've heard that Java is catching up in its ability to do math, but outside of that, you have nothing to fear from performance.

Resources