I have this obsession with doing realtime character animations based on inverse kinematics and morph targets.
I got a fair way with Animata, an open source (FLTK-based, sadly) IK-chain-style animation program. I even ported their rendering code to a variety of platforms (Java / Processing and iPhone) alt video http://ats.vimeo.com/612/732/61273232_100.jpg video of Animata renderers
However, I've never been convinced that their code is particularly optimised and it seems to take a lot of simulation on the CPU to render each frame, which seems a little unnecessary to me.
I am now starting a project to make an app on the iPad that relies heavily on realtime character animation, and leafing through the iOS documentation I discovered a code snippet for a 'two bone skinning shader'
// A vertex shader that efficiently implements two bone skinning.
attribute vec4 a_position;
attribute float a_joint1, a_joint2;
attribute float a_weight1, a_weight2;
uniform mat4 u_skinningMatrix[JOINT_COUNT];
uniform mat4 u_modelViewProjectionMatrix;
void main(void)
{
vec4 p0 = u_skinningMatrix[int(a_joint1)] * a_position;
vec4 p1 = u_skinningMatrix[int(a_joint2)] * a_position;
vec4 p = p0 * a_weight1 + p1 * a_weight2;
gl_Position = u_modelViewProjectionMatrix * p;
}
Does anybody know how I would use such a snippet? It is presented with very little context. I think it's what I need to be doing to do the IK chain bone-based animation I want to do, but on the GPU.
I have done a lot of research and now feel like I almost understand what this is all about.
The first important lesson I learned is that OpenGL 1.1 is very different to OpenGL 2.0. In v2.0, the principle seems to be that arrays of data are fed to the GPU and shaders used for rendering details. This is distinct from v1.1 where more is done in normal application code with pushmatrix/popmatrix and various inline drawing commands.
An excellent series of blog posts introducing the latest approaches to OpenGL available here: Joe's Blog: An intro to modern OpenGL
The vertex shader I describe above is a runs a transformation on a set of vertex positions. 'attribute' members are per-vertex and 'uniform' members are common across all vertices.
To make this code work you would feed in an array of vector positions (the original positions, I guess), corresponding arrays of joints and weights (the other attribute variables) and this shader would reposition the input vertices according to their attached joints.
The uniform variables relate first to the supplied texture image, and the projection matrix which I think is something to do with transforming the world coordinate system to something more appropriate to the particular requirements.
Relating this back to iPhone development, the best thing to do is to create an OpenGL ES template project and pay attention to the two different rendering classes. One is for the more linear and outdated OpenGL 1.1 and the other is for OpenGL 2.0. Personally I'm throwing out the GL1.1 code given that it applies mainly to older iPhone devices and since I'm targeting the iPad it's not relevant any more. I can get better performance with shaders on the GPU using GL2.0.
Related
I am trying to use the in-built post processing effects attached to a Camera3D while also applying a custom post processing effect to run in combination with the other effects.
I have read tutorials on how to create custom post processing effects, like the one found on the official docs. It tells me to create a MeshInstance with a QuadMesh (well, in Godot 4.0, it is actually now a PlaneMesh) and transform it into clip space.
For one, the transformation explained in the docs did not work, the quad just disappeared when I applied the following vertex shader and applied a large value to extra_cull_margin:
shader_type spatial;
render_mode cull_disabled, unshaded;
void vertex() {
POSITION = vec4(VERTEX, 1.0);
}
Then, I managed to work around this by actually manually rotating the plane such that it faces the camera and with a Z offset of something small but larger than the camera near field.
The issue is that with this plane in front, none of the world environment post processing effects work. Now, I think it might work better if I get the transform working of the quad to clip space, but it doesn't work for me.
Has anyone tried this yet for Godot 4.0 beta 1?
Okay, so reading up on how to do this in general, I stumbled upon this question.
Based on the answer from derhass, I wrote the following vertex shader code:
shader_type spatial;
render_mode cull_disabled, unshaded;
const vec2 vertices[3] = {vec2(-1,-1), vec2(3,-1), vec2(-1, 3)};
void vertex() {
POSITION = vec4(vertices[VERTEX_ID],0.0,1.0);
}
This draws a triangle and it also transforms it successfully into clip space.
Now the world environment effects are working together with the custom post processing shader:
With shader
Without shader
If this question is off, please let me know as I don't want to clutter the platform with off-topic questions!
Anyways, I'm having a hard time finding information about what's actually going on when an image is rendered because of some code I've written.
Say I wanted to add the numbers 5 and 3. The CPU would write 5 to one register and 3 to another one. The ALU would take care of the calculation and output 8. That's fine, the CPU uses MOVE and ADD to produce a result.
What I don't find any information on however, is what's going on when I want to draw a rectangle. There are importable frameworks for most programming languages which lets you do this. In SpriteKit (Swift & Objc) for example, you would write something like
let node = SKSpriteNode(color: .white, size: CGSize(width: 200, height: 300))
and add node to an SKScene (just a scene containing childNodes) and a white rectangle would "magically" get rendered. What I would like to know is what goes on under the hood. Why does this exact framework let you draw a rectangle. What is the assembly code (say, for Intel Core M) which makes the GPU calculate what this rectangle will look like? And how does SpriteKit build on the basics of Swift/Objective C to actually do this (and could I do this myself)?
Maybe a weird question, but I feel like I have to know (yes, sometimes I'm too curious). Thank you.
P.S. I would love a really detailed answer, not "the CPU 'tells' the GPU to draw a rectangle" - CPUs can't talk!
There are many ways to render convex polygon. The most used in past was ScanLine algorithm where you simply rasterize all the lines of circumference into left/right buffers and then just render using horizontal lines and interpolating the other coordinates along the way (like z,r,g,b,tx,ty,nx,ny,nz...). This was suited for single-thread CPU based SW rendering.
With parallelization (like on GPU) different approach get more popular. It simply renders only triangles (so you need to triangulate your polygons) and renders like this:
compute AABB
so simply min,max of x,y coordinates of the triangle vertexes.
loop through AABB
this is done in parallel and its done by GPU interpolators. Each interpolated (looped) "pixel" is called fragment (as it usually contains more than just color)
for each fragment
compute barycentric coordinates and from the result decide if fragment is inside (s+t<=1) or outside (s+t>1) triangle. If inside invoke Fragment shader.
All this gets done just before Fragment shader stage and usually all this (or majority of it) is implemented in HW so no code.
Nowadays GPU rendering is done by passing geometry to the gfx driver itself. What drivers does under the hood is just guess work for us but most likely they also just pass the geometry and configuration setting to the right places on the GPU (memory, registers, ...).
What the minimum configuration for the program I need to build 3D Graphics from scratch, for example I have only SFML for working with 2d graphics and I need to implement the Camera object that can move & rotate in a space
Where to start and how to implement vector3d -> vector2d conversion functions and other neccessary things
All I have for now is:
angles Phi, Xi, epsilon 1-3 and some object that I can draw on the screen with the following formula
x/y = center.x/y + scale.x/y * dot(point[i], epsilon1/epsilon2)
But this way Im just transforming "world" axis, not the Object points
First you need to implement transform matrix and vector math:
Mathematically compute a simple graphics pipeline
Understanding 4x4 homogenous transform matrices
The rest depends on kind of rendering you want to achieve:
boundary polygonal mesh rendering
This kind of rendering is the native for nowadays gfx cards. You need to implement buffers for:
depth (for filled polygons without z-sorting)
screen (to avoid flickering and also serves as Canvas)
shadow,stencil,aux (for advanced rendering techniques)
they have usually the same resolution as target rendering area. On top of this you need to implement supported primitives rendering at least point,line,triangle. see:
Algorithm to fill triangle
on top of all this you can add textures,shaders and whatever else you want to ...
(back)ray tracing
this kind of rendering is very different and current gfx HW is not build for it. This involves implementing ray/primitives intersections computation, Snell's law and analytical representation of meshes. This way you can also do multi-spectral rendering and more physically accurate effects/processes see:
How can I render an 'atmosphere' over a rendering of the Earth in Three.js? hybrid approach #1+#2
Algorithm for 2D Raytracer
How to implement 2D raycasting light effect in GLSL
Multi-Band Image raster to RGB
The difference between 2D and 3D ray tracer is almost none the only difference is how to compute perpendicular vector ...
There are also different rendering methods like Volume rendering, hybrid methods and others but their implementation is usually task oriented and generic description would most likely just mislead ... Here some 3D ray tracers of mine:
back raytrace through 3D mesh
back raytrace through 3D volume
I've created something of a simplistic renderer on my own using OpenGL ES 2.0. Essentially, it's just a class for rendering quads according to a given sprite texture. To elaborate, it's really just a single object that accepts objects that represent quads. Each quad object maintains a a world transform and object transform matrix and furnishes methods for transforming them over a given number of frames and also specifies texture offsets into the sprite. This quad class also maintains a list of transform operations to execute on its matrices. The renderer class then reads all of these properties from the quad and sets up a VBO to draw all quads in the render list.
For example:
Quad q1 = new Quad();
Quad q2 = new Quad();
q1->translate(vector3( .1, .3, 0), 30); // Move the quad to the right and up for 30 frames.
q2->translate(vector3(-.1, -.3, 0), 30); // Move the quad down and to the left for 30 frames.
Renderer renderer;
renderer.addQuads({q1, q2});
It's more complex than this, but you get the simple idea.
From the implementation perspective, on each frame, it transforms the base vertices of each object according to instruction, loads them all into a VBO including info on alpha value, and passes to a shader program to draw all quad at once.
This obviously isn't what I would call a rendering engine, but performs a similar task, just for rendering 2D quads instead of 3D geometry. I'm just curious as to whether I'm on the right track for developing a makeshift rendering engine. I agree that in most cases it's great to use an established rendering engine to get started in understanding them, but from my point of view, I like to have something of an understanding of how things are implemented, as opposed to learning something prebuilt and then learning how it works.
The problem with this approach is that adding new geometry, textures or animations requires writing code. It should be possible to create content for a game engine using established tools, like 3DS, Maya or Blender, which are completely interactive. This requires reading and parsing some standard file format like Collada. I don't want to squash your desire to learn by implementing code yourself, but you really should take a look at the PowerVR SDK, which provides a lot of the important parts for building game engines. The source code is provided and it's free.
Is it possible to export or convert my 3D models into GLSL ES 2.0? Is there any converter or any exporter tool/addon existing for any editor programs like Blender/3DS MAX/Maya that creates GLSL ES 2.0 code?
I'd like to create my models conveniently in any of the above mentioned editors and then I'd like to export/convert them into GLSL ES 2.0.
I already have a template WebGL code that displays my shaders. I want to replace my fragment shader and vertex shader parts with the GLSL ES code created automatically by a converter or an exporter tool.
I'd like to do something like this (but for GLSL ES 2.0):
Blender to GLSL
You're comparing apples with cars here. OpenGL is a drawing API, GLSL is a programming language for implementing shader code.
3D models are neither of that. The sole question "how can I convert my 3D model to OpanGL?" makes no sense.
Is it possible?
No. Because that's not the purpose of GLSL
Choose a model file format (preferrably implementing a reading parser is straightforward for) implement the parser, fill in apropriate data structures and feed those into the right parts of OpenGL, making the right calls to draw them.
OpenGL itself doesn't deal with models, scenes or even files. GLSL is not even a file format, it's a language.
I'd start with OBJ or STL files. They're reasonably easy to read and interpret and match very closely the primitive types OpenGL uses.
Probably the hardest format to read is .blend files; effectively a .blend file is a dump of the Blender process memory image. It takes a fully featured Blender (or something very similar to it) to make sense of a .blend file.
Update due to comment:
Please, please carefully read what this exporter script you linked to does: It takes an objects material settings (not the model itself) and generates GLSL code, that when used in the right framework (i.e. apropriate uniform and attribute names, matrix setup, etc.) will result in shading operations that resemble those material settings as close as possible. The script does not export a model!
You asked about exporting a 3D model. That would be the mesh of the model and it's attributes to place it in the world. Materials are not what's stored in a OBJ or STL file. They're textures, and yes, shaders. But they're completely independent of the model data itself. It's perfectly possible to use the same material settings on multiple models, or to freely exchange a model's material (textures and shaders), as long as the model provides all the required vertex attributes to make this material work.
Update 2 due to comment:
Do you even understand what a shader does? If not, here's a short synopsis: You have vertex attribute data (in buffers). These indexed attributes are submitted to OpenGL. Using a call to glDrawElements or glDrawArrays the attributes are interpreted as primitives (points, lines or triangles (or quads on older OpenGL versions)). Each primitive is then subjected to a number of transformations.
Mandatory: First step is the vertex shader which responsibility is to determine its final position in the viewport.
Optional: After vertex shading vertices the primitives formed by the vertices undergo tesselation shading. Tesselation is used to refine geometry, for example adding detail to terrain or making curved surfaces smoother.
Optional: Next comes geometry shading which can replace a single vertex with a (small) number of vertices. A geometry shader may even change the primitive type. So a single point could be replaced with a triangle for example (usefull for rendering particle systems).
Mandatory: The last step is fragment shading the primitive. After a primitive's position in the viewport has been determined, each of the pixels it covers is processed in one or more fragments. The fragment shader is a program that determines the final color and translucency in the target framebuffer.
Each shading step is controlled by a user defined program. It is these programs, shaders they are called, that are written in GLSL. Not geometry, no models. Programs! And very simple programs at that. They don't produce geometry from nothing, they always process already existing geometry passed to OpenGL.
Shaders are not used for defining or storing models. They just modify them at rendering time.
Have a look at http://www.inka3d.com which converts your Maya shaders to GLSL. For the models do you need WebGL or OpenGL ES 2.0?