I'm interesting in learning about the different layers of abstraction available for making graphical applications.
I see a lot of terms thrown around: At the highest level of abstraction, I hear about things like C#, .NET, pyglet and pygame. Further down, I hear about DirectX and OpenGL. Then there's DirectDraw, SDL, the Win32 API, and still other multi-platform libraries like WxWidgets.
How can I get a good sense of where one of these layers ends and where the next one begins? What is the "lowest possible level" way of creating a window in Windows, in C? What about C++? (A code sample would be divine.) What about in X11? Are the Windows implementations of OpenGL and DirectX built on top of the Win32 API? Where can I begin to learn about these things?
There's another question on SO where Programming Windows is suggested. What about for Linux? Is there an equivalent such book?
I'm aware that this is very low-level, and that there are many friendlier tools available, but I would like to at least learn the basics of what's going on beneath the surface. As much as I'd like to begin slinging windows and vectors right off the bat, starting with something like pygame is too high-level for me; I really need to make the full conceptual circuit of how you draw stuff on a computer.
I will certainly appreciate suggestions for books and resources, but I think it would be stupendously cool if the answers to this question filled up with lots of different ways to get to "Hello world" with different approaches to graphics programming. C? C++? Using OpenGL? Using DirectX? On Windows XP? On Ubuntu? Maybe I ask for too much.
The lowest level would be the graphics card's video RAM. When the computer first starts, the graphics card is typically set to the 80x25 character legacy mode.
You can write text with a BIOS provided interrupt at this point. You can also change the foreground and background color from a palette of 16 distinctive colors. You can use access ports/registers to change the display mode. At this point you could say, load a different font into the display memory and still use the 80x25 mode (OS installations usually do this) or you can go ahead and enable VGA/SVGA. It's quite complicated, that's what drivers are for.
Once the card's in the 'higher' mode you'd change what's on screen by accessing the memory mapped to the video card. It's stored horizontally pixel by pixel with some 'dirty regions' of pixels that aren't mapped to screen at the end of each line which you have to compensate for. But yeah, you could copy the pixels of an image in memory directly to the screen.
For things like DirectX, OpenGL. rather than write directly to the screen, commands are sent to the graphics card and it updates its screen automatically. Commands like "Hey you, draw this image I've loaded into the VRAM here, here and here" or "Draw these triangles with this transformation matrix..." take a fraction of the time compared to pixel by pixel . The CPU will thank you.
DirectX/OpenGL is a programmer friendly library for sending those commands to the card with all the supporting functions to help you get it done smoothly. A more direct approach would only be unproductive.
SDL is an abstraction layer so without bothering to read up on it I'd guess it would have different ways of working on each system. On one it might use semi-direct screen writing, another Direct3D, etc. Whatever's fastest as long as the code stays cross-platform..able.
The GDI/GDI+ and XWindow system. They're designed specifically to draw windows. Originally they drew using the pixel-by-pixel method (which was good enough because they'd only have to redraw when a button was pressed or a window moved, etc.) but now they use Direct3D/OpenGL for accelerated drawing (and special effects). Optimizations depend on the versions and implementations of these libraries.
So if you want the most power and speed, DirectX/openGL is the way to go. SDL is certainly useful for getting the most from a cross-platform environment and integrates with OpenGL anyway. The windowing system comes last but don't underestimate it. Especially with the stuff Microsoft's coming up with lately.
Michael Abrash's Graphics Programming 'Black Book' is a great place to start. Plus you can download it for free!
If you really want to start at the bottom then drawing a line is the most basic operation. Computer graphics is simply about filling in pixels on a grid (screen), so you need to work out which pixels to fill in to get a line that goes from (x0,y0) to (x1,y1).
Check out Bresenham's algorithm to get a feel for what is involved.
To be a good graphics and image processing programmer doesn't require this low level knowledge, but i do hate to be clueless about the insides of what i'm using. I see two ways to chase this - high-level down, or bottom-level up.
Top-down is a matter of following how the action traces from a high-level graphics operation such as to draw a circle, to the hardware. Get to know OpenGL well. Then the source to Mesa (free!) provides a peek at how OpenGL can be implemented in software. The source to Xorg would be next, first to see how the action goes from API calls through the client side to the X server. Finally you dive into a device driver that interfaces with hardware.
Bottom up: build your own graphics hardware. Think of ways it could connect to a computer - how to handle massive numbers of pixels through a few byte-size registers, how DMA would work. Write a device driver, and try designing a graphics library that might be useful for app programmers.
The bottom-up way is how i learned, years ago when it was a possibility with the slow 8-bit microprocessors. The direct experience with circuitry and hardware-software interfacing gave me a good appreciation of the difficult design decisions - e.g. to paint rectangles using clever hardware, in the device driver, or higher level. None of this is of practical everyday value, but provided a foundation of knowledge to understand newer technology.
see Open GPU Documentation section:
http://developer.amd.com/documentation/guides/Pages/default.aspx
HTH
On MSWindows it is easy: you use what the API provides, whether it is the standard windows programming API or the DirectX-family API's: that's what you use, and they are well documented.
In an X windows environment you use whatever X11-libraries that are provided. If you want to understand the principles behind windowing on X, I suggest that you do this, nevermind that many others tell you not to, it will really help you to understand graphics and windowing under X. You can read the documentation on X-programming (google for it). (After this exercise you would appreciate the higher level libraries!)
Apart from the above, at the absolutely lowest level (excluding chip-level) that you can go is to call the interrupts that switch to the various graphics modes available - there are several - and then write to the screen buffers, but for this you would have to use assembler, anything else would be too slow. Going this way will not be portable at all.
Another post mentions Abrash's Black Book - an excellent resource.
Edit: As for books on programming Linux: it is a community thing, there are many howto's around; also find a forum, join it, and as long as you act civilized you will get all the help you can ever need.
Right off the bat, I'd say "you're asking too much." From what little experience I've had, I would recommend reading some tutorials or getting a book on either directX or OpenGL to start out. To go any lower than that would be pretty complex. Most of the books I've seen in OGL or DX have pretty good introductions that explain what the functions/classes do.
Once you get the hang of one of these, you could always dig in to the libraries to see what exactly they're doing to go lower.
Or, if you really, absolutely MUST learn the LOWEST level... read the book in the above post.
libX11 is the lowest level library for X11. I believe the opengl/directx talk to the driver/hardware directly (or emulate unsupported ops), so they would be the lowest level library.
If you want to start with very low level programming, look for x86 assembly code for VGA and fire up a copy of dosbox or similar.
Vulkan api is an api which gives you very low level access to most if not all features of the gpu, computational and graphical, it works on amd and Nvidia gpus (not all)
you can also use CUDA, but it only works on Nvidia gpus and has access to computational features only, no video output.
Related
One way of doing GPU programming is OpenCL, which will work with parallelized, number-crunching operations.
Now think of your favorite 3D PC game. When the screen renders, what's going on? Did the developers hand-craft an OpenCL kernel (or something like it), or are they using pre-programmed functions in the graphics card?
Sorry to make this sound like a homework problem, I couldn't think of a better way to ask it.
H'okay, so, I'ma answer this in terms of history. Hopefully that gives a nice overview of the situation and lets you decide how to proceed.
Graphics Pipeline
3D graphics have an almost set-in-stone flow of calculations. You start with your transformation matrices, you multiply out your vertex positions (maybe generate some more on the fly), figure out what your pixels ought to be colored, then spit out the result. This is the (oversimplified) gist of 3D graphics. To change anything in it, you just twiddle one aspect of the pipeline a bit with a 'shader', i.e. little programmable elements with defined inputs and outputs so that they could be slotted into the pipeline.
Early GPGPU
Back when GPGPU was still in its infancy, the only way people had access to the massively parallel prowess of the GPU was through graphics shaders. For example, there were fragment shaders, which basically calculated what colors should be on each pixel of the screen (I'm kind of oversimplifying here, but that's what they did).
So, for example, you might use a vertex shader to chuck data about the screen before reducing a bunch of values in the fragment shader by taking advantage of color blending (effectively making the tricky transformation of mathematical problem space to... well, color space).
The gist of this is that old GPGPU stuff worked within the confines of 3D graphics, using the same 'pre-programmed functions in the graphics card' that the rest of the 3D graphics pipeline used.
It was painful to read, write, and think about (or at least, I found it so painful that I was dissuaded).
CUDA and OpenCL and [all of the other less popular GPGPU solutions]
Then some folks came along and said, "Wow, this is kind of dumb - we're stuck in the graphics pipeline when we want to be doing more general calculations!"
Thus GPGPU escaped from the confines of the graphics pipeline, and now we have OpenCL and CUDA and Brook and HSA and... Well, you get the picture.
tl;dr
The difference between GPGPU kernels and 3D graphics kernels are that the latter are stuck in a pipeline with (convenient) constraints attached to them, while the former have far more relaxed requirements, the pipeline is defined by the user, and the results don't have to be attached to a display (although they can be if you're masochistic like that).
When you run a game there may be two distinct systems operating on your GPU:
OpenGL renders images to your screen (graphics)
OpenCL does general-purpose computing tasks (compute)
OpenGL is programed with shaders. OpenCL is programmed with kernels.
If you would like to learn in more detail how games work on the GPU, I recommend reading about OpenCL, OpenGL, and game engine architecture.
I want to read the framebuffer of the videocard at the lowest level possible for a security application I'm writing.
I want to be as sure as possible that what I'm reading is exactly what will be finally
put on the bits of the hardware lighting the pixels of the screen,
and that no software layer is in the middle (or at least I want to have the lowest number possible of layers in the middle).
I've seen it's pretty easy to use X to grab the screen in a precise moment, but that call
is still passing through the X server.
I would like to have something really more low level,
even if this means messing up with some ioctl with the video card.
I've seen the existence of DRI and DRI2, but they are very very badly documented, especially
the latter.
I can't really understand how they work.
Do you have any idea, reference or starting point for a good research?
Anything would be appreciated!
I'm not sure how much reading the framebuffer will help you (even disregarding the issue pointed out by timday in his comment, deciding whether what you read there is what you want it to be may not be very easy), but if you are doing this on Linux you could map the kernel framebuffer devices, possibly using DirectFB to help you. Alternatively, if you are on a non-Linux PC platform you could use VESA (take a look at the VESA code in X.Org and the X.Org VESA driver (the actual code is split between the two). Be aware that you will probably also have some fun with things like multi-monitor setups.
Hey guys, I would like to develop a light/laser show editor and simulator, and for this of course I am going to learn some graphics programming. I am thinking about using C# and XNA.
I was just wondering what aspects of graphics programming I should research or focus on given the project I am working on. I am new to graphics programming so I don't know much about it, but for example I imagine something that I might look into would (possibly?) be volumetric lighting.
For example, what would be a practical way to go about rendering a 'laser' of varied width/color? I read somewhere to just draw a cylinder and apply a shader to it, I would like to confirm that this is the way.
Given that this seems like a big project, I was thinking about starting off by creating light sources and giving them properties so that I can easily manipulate them. I have (mis)read that only a certain amount of lights can be rendered at any given time, I believe eight. Does this only apply to ambient lights? Given this possible limitation, and the fact that most of the lights I will use will be directional, such as head-lights or lasers, what would be a different way to render these? Is that what volumetric lighting would be?
I'd just like to get some things clear before I dive into it. Since I'm new to this I probably didn't make the best use of words, so if something doesn't make sense please let me know. Thanks and sorry for my ignorance.
The answer to this depends on the level of sophistication that you need in your display simulation. Computer graphics is ultimately a simulation of the transport of light; that simulation can be as sophisticated as calculating the fraction of laser light deflected by particles in the atmosphere to the viewer's eyepoint, or as simple as drawing a line. Try out the cylinder effect and see if it works for your project. If you need something more sophisticated, look into shader programming (using Nvidia Cg, for example), and volumetric shading as you mentioned; also post-processing glow effects may be useful. For OpenGL, I believe there is a limit of 8? light sources in a scene, but you could conceivably work around this limit by doing your own shading logic.
Well if it's just for light show simulations I'd imagine your going to need a lot of custom lighting effects - so regardless if you decide to use XNA or straight DirectX your best bet would be to start by learning shader languages and how to program various lighting effects using them. Once you can reproduce the type of laser lighting you want, then you can experiment with the polygons you want to use to represent the lasers. (I've used the cylinder method in some of my work for personal purposes, but I'm not sure how well straight cylinders will fit your purpose).
Although its faster, I think its best not to use vanilla hardware lighting because of its limitations. Pixel shaders can help with you task. Also you may want to chose OpenGL because of portability and its clarity in rendering methods. I worked on Direct3D for several years before switching to OpenGL. OpenGL functions and states are easier to learn and rendering methods (like multi-pass rendering) is a lot clear. If you like to code on C# (which I dont recommend for these tasks), you can use CsGL library to access OpenGL functions.
I've developed a running Super Mario Sprite using Visual C++ 6.0 and DirectX. But this isn't very satisfying to me (raping a 3D-Multimedia-framework for displaying a 2D sprite only), so I would like to be able to program an animated sprite using C and assembler only.
So, when looking to old games (Wolfenstein, for example) it looks that most of the game is written in C and everytime it comes to graphics output there is the use of assembler.
Unfortunatly when trying to use this old assembler code there is always the error message "NTVDM.exe has found an invalid instruction" so this things don't seem to work nowadays.
Is there any tutorial on graphics programming in assembler that is still usefull?
(I don't want to use any bloated frameworks or libraries, I just want to develop everything on my own. WinAPI would be OK for creating a full screen window and for catching user input, but not for graphics because I read GDI is too slow for fast graphics.)
I'm using WindowsXP and MASM or A86.
I totally agree with samcl
The main reason for not using assembler anymore is that you cannot access the Videomemory anymore. Back in the early days (you mentioned Castle Wolfenstein) there was a special video mode called 0x13h where your graphic was just a block of memory(each pixel was a palette color ranging from 0-255<--1 Byte) You were able to access this memory through this specific video mode, however, today things are much more complicated
Today you have very fast Videomemory and using your CPU for accessing it will just tear down all performance, as you CPU is connected through PCI-Express/AGP/PCI/VESA-LOCALBUS/ISA (<- remembering anyone!?)
Graphicsprogramming is often a lot of read and write accesses(read pixel, check if it is transparent, multiply with alpha, write pixel, etc.)
The modern memory Interfaces are much slower than direct access inside the graphic card. That's why you really should use shaders, as Robert Gould suggests. In this way you can write faster and easier to understand code and it will not stall your GFX-Memory.
IF you are more interested in GFX Programming, you can wet your appetite with shadertoy, a community dedicated to shaderbased effects complete with WebGLbased Shadercode execution.
Also your beginner assembler code will be pretty lame. in quality as in performance. Trust me. It needs a lot of time for optimizing such primitive code. So your compiled C/C++ Code will outperform your handwritten asm easily.
If you are interested in Assembler, try to code something like diskaccess. This is where you can gain a lot of performance.
It sounds like you only use Assembler because you seem to think that this is necessary. This isn't the case. If you don't have any other reason for it (i.e. wanting to learn it), don't use Assembler here, unless you know exactly what you're doing.
For your average graphics engine, Assembler programming is completely unnecessary. Especially when it comes to a Super Mario style 2D sprite engine. Even “slow” scripting languages like Python are fast enough for such things nowadays.
Adding to that, if you don't know very precisely what you're doing, Assembler will not be faster than C (in fact, chances are it will be slower because you'll re-implement existing C functions less efficiently).
I'm guessing if you are already using C with DirectX, speed is not the issue, and that this is more of a learning exercise. For 2D under Windows, C and DirectX is going to be very fast indeed, and as Konrad Rudolph points out, hand cranked assembler is unlikely to be faster than a highly optimized SDK.
From a purely educational standpoint, it is an intersting activity, but quite complex. Back in the early days of home computers and the first PCS, the graphics screen appeared pretty much as a block of memory, where bytes corresponded to one or more coloured pixels. By changing the value of the screen memory you could plot points, and hence lines, and on to sprites etc... On modern PCs this tends not to be an option, in that you program a graphics card, usually via an SDK, to do the same job. The card then does the hard work, and you are provided with a much higher level of abstraction. If you really wanted to get a feel for what it was like back in the day, I would recommend an emulator. For a modern game, stick with your SDKs.
It is possible to program your own 2D engine in a recent version of Directx, if you wish to investigate this avenue. You can create a "screen space" aligned polygon, with no perspective correction, of which is texture mapped. You can then plot your sprites on a pixel-by-pixel basis onto this texture map.
As for mode 13h (Peter Parker), it brings back some memories!
__asm
{
mov ax,0x13
int 10h // 16-bit code only, not Windows
}
But of course this will fault in a 32-bit or 64-bit Windows program; 16-bit BIOS calls are not supported by the Windows kernel (which installs its own interrupt table as part of booting and switching the CPU to 64-bit mode.)
I would tend to avoid assembler with a barge pole, it can be particulary difficult to debug, and maintain; however if you wish to explore this subject in more detail, I can recommend Michal Abrash's Graphics Programming Black Book. It's a bit old, but a good read and will give you some insight into graphics programming techniques before 3D hardware.
Assembler for graphics was because, back then, most people lacked graphics card with 3d support, so it had to be done on the CPU, not anymore. Nowadays it's about shader programming. Shader languages allow you to cuddle up with the bare metal. So if anything you should try to code your 2d graphics to be shadered base, that way the experience will have value as a career skill.
Try CUDA for a starter.
My recommendation is to experiment. Take your sprite code and write in a number of forms, starting with C/GDI and C++/DirectDraw. Don't worry about assembler yet.
DirectX is your best bet for fast action graphics. Learn it, then figure out how to micro-optimize with assembler. In general, assembler isn't going to make your API calls faster. It is going to open up flexibility for faster computation for things like 3D rotation, texture mapping, shading, etc.
Start with DirectDraw. Here's a FAQ. Technically, DirectDraw is deprecated after DirectX 7, but you can still use it and learn from it. It'll allow you direct framebuffer modification, which is what you're probably looking for.
There's some helpful tutorials and forums at TripleBuffer Software.
Also consider upgrading your compiler to Visual C++ 2008 Express. VC++ 6 has a buggy compiler that can be problematic with trying to compile certain C++ libraries.
I woud like to create a cross-platform drawing program. The one requirement for writing my app is that I have pixel level precision over the canvas. For instance, I want to write my own line drawing algorithm rather than rely on someone elses. I do not want any form of anti-aliasing (again, pixel level control is required.) I would like the users interactions on the screen to be quick and responsive (pending my ability to write fast algorithms.)
Ideally, I would like to write this in Python, or perhaps Java as a second choice. The ability to easily make the final app cross-platform is a must. I will submit to different API's on different OS'es if necessary as long as I can write an abstraction layer around them. Any ideas?
addendum: I need the ability to draw on-screen. Drawing out to a file I've got figured out.
I just this week put together some slides and demo code for doing 2d graphics using OpenGL from python using the library pyglet. Here's a representative post: Pyglet week 2, better vertex throughput (or 3D stuff using the same basic ideas)
It is very fast (relatively speaking, for python) I have managed to get around 1,000 independently positioned and oriented objects moving around the screen, each with about 50 vertices.
It is very portable, all the code I have written in this environment works on windows and Linux and mac (and even obscure environments like Pypy) without me ever having to think about it.
Some of these posts are very old, with broken links between them. You should be able to find all the relevant posts using the 'graphics' tag.
The Pyglet library for Python might suit your needs. It lets you use OpenGL, a cross-platform graphics API. You can disable anti-aliasing and capture regions of the screen to a buffer or a file. In addition, you can use its event handling, resource loading, and image manipulation systems. You can probably also tie it into PIL (Python Image Library), and definitely Cairo, a popular cross-platform vector graphics library.
I mention Pyglet instead of pure PyOpenGL because Pyglet handles a lot of ugly OpenGL stuff transparently with no effort on your part.
A friend and I are currently working on a drawing program using Pyglet. There are a few quirks - for example, OpenGL is always double buffered on OS X, so we have to draw everything twice, once for the current frame and again for the other frame, since they are flipped whenever the display refreshes. You can look at our current progress in this subversion repository. (Splatterboard.py in trunk is the file you'll want to run.) If you're not up on using svn, I would be happy to email you a .zip of the latest source. Feel free to steal code if you look into it.
If language choice is open, a Flash file created with Haxe might have a place. Haxe is free, and a full, dynamic programming language. Then there's the related Neko, a virtual machine (like Java's, Ruby's, Parrot...) to run on Mac, Windows and Linux. Being in some ways a new improved form of Flash, naturally it can draw stuff. http://haxe.org/
QT's Canvas an QPainter are very good for this job if you'd like to use C++. and it is cross platform.
There is a python binding for QT but I've never used it.
As for Java, using SWT, pixel level manipulation of a canvas is somewhat difficult and slow so I would not recommend it. On the other hand Swing's Canvas is pretty good and responsive. I've never used the AWT option but you probably don't want to go there.
I would recommend wxPython
It's beautifully cross platform and you can get per pixel control and if you change your mind about that you can use it with libraries such as pyglet or agg.
You can find some useful examples for just what you are trying to do in the docs and demos download.