I'm trying to figure out some simple concepts about image based lighting for PBR. In many documents and code, I've seen the light direction (FragToLightDir) being set to the reflection vector (reflect(EyeToFragDir,Normal)). Then they set the half vector to the mid-way point between the light and view direction: HalfVecDir = normalize(FragToLightDir+FragToEyeDir); But doesn't this just result in the half vector being identical to the surface normal? If so, this would mean that terms like NDotH are always 1.0. Is this correct?
Here is another source of confusion for me. I'm trying to implement specular cube maps from the app Lys, using their algorithm for generating the correct roughness value to use for mip-level sampling based on roughness (here: https://docs.knaldtech.com/doku.php?id=specular_lys#pre-convolved_cube_maps_vs_path_tracers in the section Pre-convolved Cube Maps vs Path Tracers). In this document, they ask us to use NDotR as a scalar. But what is this NDotR in respect to IBL? If it means dot(Normal,ReflectDir), then isn't that exactly equivalent to dot(Normal,FragToEyeDir)? If I use either of these dot product results, the final result is too glossy at grazing angles (when compared to their more simplistic conversion using BurleyToMipSimple()), which makes me think I'm misunderstanding something about this process. I've tested the algorithm using NDotH, and it looks correct, but isn't this simply canceling out the rest of the math, since NDotH==1.0? Here is my very simple function to extract the mip level using their suggested logic:
float computeSpecularCubeMipTest(float perc_ruf)
{
//float n_dot_r = dot( Normal, Reflect );
float specular_power = ( 2.0 / max( EPSILON, perc_ruf*perc_ruf*perc_ruf*perc_ruf ) ) - 2.0;
specular_power /= ( 4.0 * max( NDotR, EPSILON ) );
return sqrt( max( EPSILON, sqrt(2.0/( specular_power + 2.0 )))) * MipScaler;
}
I realize this is an esoteric subject. Since everyone is using popular game engines these days, no one is forced to understand this madness! But I appreciate any advice on how to go about this.
Edit: Just to make sure I'm clear, I'm referring to pure image based lighting, with no directional lights, no spot lights, etc. Just a cube map that lights the whole scene, similar to the lighting in apps like Substance Painter and Blender's Viewport shading mode.
I'm not familiar with this particular app, but it looks like you're on the right track here. Part of the advantage of pre-convoluting the cube maps is to customize each pixel to be the light source for a particular reflection vector, so indeed NdotV is identical to NdotR as you've noticed. The R still needs to be calculated, for the texture lookup, so it doesn't matter much which one you use for the dot. There is no such thing as H or NdotH used for IBL lookups; those are only for point lights.
If the grazing angles look wrong, perhaps there's a Fresnel term missing somewhere? Reflections start to work differently at those angles.
For what it's worth, the glTF Sample Viewer is using NdotV for its specular IBL lookup.
If this question is off, please let me know as I don't want to clutter the platform with off-topic questions!
Anyways, I'm having a hard time finding information about what's actually going on when an image is rendered because of some code I've written.
Say I wanted to add the numbers 5 and 3. The CPU would write 5 to one register and 3 to another one. The ALU would take care of the calculation and output 8. That's fine, the CPU uses MOVE and ADD to produce a result.
What I don't find any information on however, is what's going on when I want to draw a rectangle. There are importable frameworks for most programming languages which lets you do this. In SpriteKit (Swift & Objc) for example, you would write something like
let node = SKSpriteNode(color: .white, size: CGSize(width: 200, height: 300))
and add node to an SKScene (just a scene containing childNodes) and a white rectangle would "magically" get rendered. What I would like to know is what goes on under the hood. Why does this exact framework let you draw a rectangle. What is the assembly code (say, for Intel Core M) which makes the GPU calculate what this rectangle will look like? And how does SpriteKit build on the basics of Swift/Objective C to actually do this (and could I do this myself)?
Maybe a weird question, but I feel like I have to know (yes, sometimes I'm too curious). Thank you.
P.S. I would love a really detailed answer, not "the CPU 'tells' the GPU to draw a rectangle" - CPUs can't talk!
There are many ways to render convex polygon. The most used in past was ScanLine algorithm where you simply rasterize all the lines of circumference into left/right buffers and then just render using horizontal lines and interpolating the other coordinates along the way (like z,r,g,b,tx,ty,nx,ny,nz...). This was suited for single-thread CPU based SW rendering.
With parallelization (like on GPU) different approach get more popular. It simply renders only triangles (so you need to triangulate your polygons) and renders like this:
compute AABB
so simply min,max of x,y coordinates of the triangle vertexes.
loop through AABB
this is done in parallel and its done by GPU interpolators. Each interpolated (looped) "pixel" is called fragment (as it usually contains more than just color)
for each fragment
compute barycentric coordinates and from the result decide if fragment is inside (s+t<=1) or outside (s+t>1) triangle. If inside invoke Fragment shader.
All this gets done just before Fragment shader stage and usually all this (or majority of it) is implemented in HW so no code.
Nowadays GPU rendering is done by passing geometry to the gfx driver itself. What drivers does under the hood is just guess work for us but most likely they also just pass the geometry and configuration setting to the right places on the GPU (memory, registers, ...).
The function insertObservation in COccupancyGridMap2D takes in two parameters which are the CPose3D and CObservation2DRangeScan values, even though both of these values are accurate with no noise, the grid is producing warped boundaries. The only thing I can think of is the scan.aperture settings might be producing this effect but these are correct with a range of 2*PI and other visual aides for point clouds show no warpage at all. Below is an illustration of this.
On the right the occupancy grid is warped compared to the ground truth square boundary. The left points look fine and are using the same aperture and load FromVectors settings.
Here is example code to try to verify the warp effect your self.
COccupancyGridMap2D gridmap;
gridmap.setSize(-4.0,4.0,-4.0,4.0,0.025f);
#define SCANS_SIZE 100
char SCAN_VALID[] = {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1};
CPose3D transform = CPose3D(0,0,0,0,0,0);
CObservation2DRangeScan read_scan;
read_scan.aperture = 2*M_PIf;
read_scan.rightToLeft = true;
vector<float> landmark = {2.9f,2.906f,2.924f,2.953f,2.996f,3.052f,3.124f,3.212f,3.319f,3.447f,3.601f,3.786f,4.007f,3.948f,3.736f,3.560f,3.413f,3.290f,3.188f,3.104f,3.037f,2.984f,2.945f,2.918f,2.903f,2.900f,2.909f,2.930f,2.963f,3.009f,3.069f,3.144f,3.237f,3.349f,3.483f,3.644f,3.837f,4.069f,3.891f,3.689f,3.521f,3.380f,3.263f,3.166f,3.086f,3.022f,2.973f,2.937f,2.913f,2.901f,2.901f,2.913f,2.937f,2.973f,3.022f,3.086f,3.166f,3.263f,3.380f,3.521f,3.689f,3.891f,4.069f,3.837f,3.644f,3.483f,3.349f,3.237f,3.144f,3.069f,3.009f,2.963f,2.930f,2.909f,2.900f,2.903f,2.918f,2.945f,2.984f,3.037f,3.104f,3.188f,3.290f,3.413f,3.560f,3.736f,3.948f,4.007f,3.786f,3.601f,3.447f,3.319f,3.212f,3.124f,3.052f,2.996f,2.953f,2.924f,2.906f,2.900f};
float *SCAN_RANGES = &landmark[0];
read_scan.loadFromVectors(SCANS_SIZE, SCAN_RANGES,SCAN_VALID);
gridmap.insertObservation(&read_scan,&transform);
CSimplePointsMap m3;
m3.insertObservation(&read_scan);
m3.getAllPoints(map_xs,map_ys,map_zs);
Here is a image of the simplePointsMap plot (red points) vs the OccupanyGrid
The angles being casted from the occupany grid look correct, with a consistent interval, but the angle is still off from simplepoints map, length looks ok and it seems each ray could be rotated to match with one of the red points. Possibly what could be happening is a mapping issue, and since we try to make the angles into discrete horizontal and vertical steps this causes the misalignment. I've tried increasing the resolution but this does not help, I guess that makes sense since scaling a horizontal/vertical ratio would still result in the same ratio and mismatch. I might be missing something though, what else could be causing this distortion, is this expected and the best we can do? Thank you for any help.
It seems to me that the problem is in the assumption of which are the angles of each scan "ray".
Take a look at the class mrpt::obs::CSinCosLookUpTableFor2DScans, generate one such sin/cos LUT for your specific scan object, and double check if the sin/cos values coincide with yours, as used to generate the scan.
By the way, COccupancyGridMap2D has one method to simulate a 2D scan from a gridmap image, give it a try, and if that one generates warped results, please fill up a bug report (!) ;-)
Cheers.
I just realized what was going on, CSimplePointsMap and COccupancyGridMap2D use two slightly different references for point angles. CSimplePointsMap is expecting an overlap between the first and last point while COccupancyGridMap2D is not. The simple fix to all of this then is to read in one less scan for the COccupancyGridMap2D and then everything lines up. This is if your angles are being defined as so, which is fine for CSimplePointsMap.
for (int i = 0; i < Raysize; i++)
{
float angle = -angle_range / 2 + i * (angle_range) / (Raysize-1);
Here is the fix for OccupancyGridMap2D insertObservation using SCANS_SIZE-1 instead and CSimplePointsMap can still use SCANS_SIZE.
read_scan.loadFromVectors(SCANS_SIZE-1, SCAN_RANGES,SCAN_VALID);
gridmap.insertObservation(&read_scan,&transform);
Back story: I'm creating a Three.js based 3D graphing library. Similar to sigma.js, but 3D. It's called graphosaurus and the source can be found here. I'm using Three.js and using a single particle representing a single node in the graph.
This was the first task I had to deal with: given an arbitrary set of points (that each contain X,Y,Z coordinates), determine the optimal camera position (X,Y,Z) that can view all the points in the graph.
My initial solution (which we'll call Solution 1) involved calculating the bounding sphere of all the points and then scale the sphere to be a sphere of radius 5 around the point 0,0,0. Since the points will be guaranteed to always fall in that area, I can set a static position for the camera (assuming the FOV is static) and the data will always be visible. This works well, but it either requires changing the point coordinates the user specified, or duplicating all the points, neither of which are great.
My new solution (which we'll call Solution 2) involves not touching the coordinates of the inputted data, but instead just positioning the camera to match the data. I encountered a problem with this solution. For some reason, when dealing with really large data, the particles seem to flicker when positioned in front/behind of other particles.
Here are examples of both solutions. Make sure to move the graph around to see the effects:
Solution 1
Solution 2
You can see the diff for the code here
Let me know if you have any insight on how to get rid of the flickering. Thanks!
It turns out that my near value for the camera was too low and the far value was too high, resulting in "z-fighting". By narrowing these values on my dataset, the problem went away. Since my dataset is user dependent, I need to determine an algorithm to generate these values dynamically.
I noticed that in the sol#2 the flickering only occurs when the camera is moving. One possible reason can be that, when the camera position is changing rapidly, different transforms get applied to different particles. So if a camera moves from X to X + DELTAX during a time step, one set of particles get the camera transform for X while the others get the transform for X + DELTAX.
If you separate your rendering from the user interaction, that should fix the issue, assuming this is the issue. That means that you should apply the same transform to all the particles and the edges connecting them, by locking (not updating ) the transform matrix until the rendering loop is done.