Dynamic Camera Caliberation OpenCV python3 - python-3.x

Question

I am planning on making a robotic arm. I have a camera mounted on the arm. I am using Opencv with python3 to do IP.
I want the arm to detect the point on the ground and the servos to move accordingly. I have completed the part of detection and calculating the world coordinates. Also, the inverse kinematics that is required.
The problem here is that I have calibrated the camera for a certain height (20 cm). So, the correct world coordinates are received at the height of 20 cm only. I want the camera to keep correcting the reading at every 2s that it moves towards the ground (downward).
Is there a way that I can do the calibration dynamically, and give dynamic coordinates to my arm? I don't know if this is the right approach. If there is another method to do this, please help.

Answer 1

I am assuming you are using the undistort function to first undistort the image and then using the rotational vector(rcvt) and translational vector(tvct) along with distortCoeffs to get the world coordinates. The correct coordinates are only obtained at that specific height because the rvct and tvct will change according to the square size (of the chess-board) used for calibration.
A smart way to overcome this would be to eliminate the rotational vector and translational vector easily.
Since the camera calibration constants remain the same at any height/rotation, it can be used in this. Also, rather than calibrating it every 2 seconds (which would consume too much CPU), directly use the method below to get the values!
Let's say (img_x, img_y) is the image coordinate which you need to transform to world coordinate (world_x, world_y) and cameraMatrix is your camera matrix. For this method, you need to know the distance_cam, that is, the perpendicular distance of your object from the camera.
Using python, and opencv, use the following code :
import numpy as np
from numpy.linalg import inv
img_x, img_y = 20, 30 # your image coordinates go here
world_coord = np.array([[img_x], [img_y], [1]]) # create a 3x1 matrix
world_coord = inv(cameraMatrix) * world_coord # use formula cameraMatrix^(-1)*coordinates
world_coord = world_coord * distance_cam
world_x = world_coord[0][0]
world_y = world_coord[1][1]
print(world_x, world_y)
At first, we may not realise that the units in the world coordinates don't matter. After multiplying by the inverse of the camera matrix you have defined the ratio x/z which is unitless. So, you can choose the distance_cam in any unit and the end result would be in the units of distance_cam, that is, if distance_cam was in mm, then world_x, world_y would also be in mm.