I have a 3D point cloud and I would like to match different point clouds with each other for recognition purposes. Does OpenCV or Tensorflow do it for me? if yes, how?
Example:
src1 = pointCloud of object 1
src2 = pointCloud of object 2
compare(src1, src2)
Output: Both point clouds are of the same object or different objects.
I want to achieve something like this. Please help with some ideas or resources.
OpenCV Surface Matching can be used to detect and find pose of a given point cloud within another point cloud.
In Open3d there is a 3d reconstruction module, but it is used to register (find poses) of RGBD Images and reconstruct 3d object from them. But there is a sub step in which different point cloud fragments are registered (finding pose of point clouds) to combine them into a single point cloud for reconstruction. But not sure if it is useful for your task.
There are many 3d Point cloud object detection methods which use neural networks, as well, but you have to generate the data needed to train, if your objects are not available in a standard dataset.
Related
I'm trying to use face.evoLVe library, which is high-performance Face Recognition Library in PyTorch. Going through the codes, I encountered a list of coordinates named REFERENCE_FACIAL_POINTS which is :
REFERENCE_FACIAL_POINTS = [ # default reference facial points for crop_size = (112, 112); should adjust REFERENCE_FACIAL_POINTS accordingly for other crop_size
[30.29459953, 51.69630051],
[65.53179932, 51.50139999],
[48.02519989, 71.73660278],
[33.54930115, 92.3655014],
[62.72990036, 92.20410156]
]
Further down the code, these numbers are converted into numpy arrays and used heavily everywhere in align_trans.py
I have some questions:
What exactly are these numbers? Reading the comment, I'm certain they are locations for eyes, lips, etc but what exactly do they represent and how are they calculated?
It seems they are tightly coupled with the size of the input image [used in training (at the very least)]. Knowing this, how can one calculate new reference points for newer image sizes?
Are these points for frontal pose only? or will they work on profiles, etc as well? If they don't, how can we add reference points for profiles or other random facial poses?
How can I create a 3D point pattern (PP3) in spatstat package for a point cloud which does not have a cube/rectangular cube shape (for example, cone or tetrahedron)?
Currently it is only possible to use a three dimensional box as the domain of a three dimensional point pattern, and I don't expect this to change anytime soon.
I'm working on a 3D reconstruction system and want to generate a triangular mesh from the registered point cloud data using Python 3. My objects are not convex, so the marching cubes algorithm seems to be the solution.
I prefer to use an existing implementation of such method, so I tried scikit-image and Open3d but both the APIs do not accept raw point clouds as input (note that I'm not expert of those libraries). My attempts to convert my data failed and I'm running out of ideas since the documentation does not clarify the input format of the functions.
These are my desired snippets where pcd_to_volume is what I need.
scikit-image
import numpy as np
from skimage.measure import marching_cubes_lewiner
N = 10000
pcd = np.random.rand(N,3)
def pcd_to_volume(pcd, voxel_size):
#TODO
volume = pcd_to_volume(pcd, voxel_size=0.05)
verts, faces, normals, values = marching_cubes_lewiner(volume, 0)
open3d
import numpy as np
import open3d
N = 10000
pcd = np.random.rand(N,3)
def pcd_to_volume(pcd, voxel_size):
#TODO
volume = pcd_to_volume(pcd, voxel_size=0.05)
mesh = volume.extract_triangle_mesh()
I'm not able to find a way to properly write the pcd_to_volume function. I do not prefer a library over the other, so both the solutions are fine to me.
Do you have any suggestions to properly convert my data? A point cloud is a Nx3 matrix where dtype=float.
Do you know another implementation [of the marching cube algorithm] that works on raw point cloud data? I would prefer libraries like scikit and open3d, but I will also take into account github projects.
Do you know another implementation [of the marching cube algorithm] that works on raw point cloud data?
Hoppe's paper Surface reconstruction from unorganized points might contain the information you needed and it's open sourced.
And latest Open3D seems to be containing surface reconstruction algorithms like alphaShape, ballPivoting and PoissonReconstruction.
From what I know, marching cubes is usually used for extracting a polygonal mesh of an isosurface from a three-dimensional discrete scalar field (that's what you mean by volume). The algorithm does not work on raw point cloud data.
Hoppe's algorithm works by first generating a signed distance function field (a SDF volume), and then passing it to marching cubes. This can be seen as an implementation to you pcd_to_volume and it's not the only way!
If the raw point cloud is all you have, then the situation is a little bit constrained. As you might see, the Poisson reconstruction and Screened Poisson reconstruction algorithm both implement pcd_to_volume in their own way (they are highly related). However, they needs additional point normal information, and the normals have to be consistently oriented. (For consistent orientation you can read this question).
While some Delaunay based algorithm (they do not use marching cubes) like alphaShape and this may not need point normals as input, for surfaces with complex topology, it's hard to get a satisfactory result due to orientation problem. And the graph cuts method can use visibility information to solve that.
Having said that, if your data comes from depth images, you will usually have visibility information. And you can use TSDF to build a good surface mesh. Open3D have already implemented that.
I am working on a project where I am trying to extract key features of a bicycle from an overall image. I am currently investigating the use of Haar Cascades to train my computer to find certain regions of interest from said bicycles, e.g. the pedal-sprocket, seat, handle-bars. Then I will extract local features from these sub regions accordingly. The purpose is to create an overall descriptor of a particular bicycle so I can try to match it throughout a sample set of images of other bicycles.
My questions are as follows: Can I train a Haar classifier to look for a sub-component of an overall object? For example, say I want to look for the handlebars on a bicycle. How should I design the training? Should I detect the bicycle first, and then detect the handlebars within the overall bicycle region (Similar to detecting the eyes within a face in terms of facial recognition)? Since I know beforehand that all my images will contain a picture of a bicycle, I'm not sure if there is any point in detecting the bicycle to begin with and then looking for sub components.
In terms of training a Haar cascade and creating an XML that I can use (in OpenCV 3.1 and Python 3.6), could I just set up the positive and negative images with pictures of bicycles and no bicycles respectively? With the difference being that I isolate the particular area of interest by cropping the image appropriately each time (e.g. where the handlebars are)?
Also open to any recommendations about how others might solve the general problem of extracting key features for object matching. This is just one approach I am currently investigating. Thanks!
I am doing a project for stair recognition which combines point cloud library(PCL) and svmlight.
Now I am able to segment the point cloud, clustering and extract feature using fast point feature histogram(FPFH).
The problem is: How can I transform FPFH results(many FPFHSignature33 which is a feature cloud contents 33 histogram for each point) to a feature vector which can be as the input for svmlight?
I know I need to label "+1" or "-1" for pos or neg sample, but how about the feature vector or value for each data?
I'm totally confused.
Any suggestion or hint is appreciated! Thanks!
You probably want a global feature descriptor like VFH or CVFH. Here are some useful resources:
3D Descriptors for Object and Category Recognition: a Comparative Evaluation.
Tutorial: Point Cloud Library: Three-Dimensional Object Recognition and 6 DOF Pose Estimation.