I'm trying to use the matching_template function from the ArrayFire library But I don't know how to find the X and Y coordinates of the best matching value.
I was using the imageproc library to perform this function and there it has the find_extremes function that returns the coordinates to me. How would you do the same using ArrayFire lib?
My example using imageproc
let template = image::open("connect.png").unwrap().to_luma8();
let screenshot = image::open("screenshot.png").unwrap().to_luma8();
let matching_probability= imageproc::template_matching::match_template(&screenshot, &template, MatchTemplateMethod::CrossCorrelationNormalized);
let positions = find_extremes(&matching_probability);
println!("{:?}", positions);
Extremes { max_value: 0.9998113, min_value: 0.42247093,
max_value_location: (843, 696), min_value_location: (657, 832) }
My example using ArrayFire
let template: Array<u8> = arrayfire::load_image(String::from("connect.png"), true);
let screenshot: Array<u8> = arrayfire::load_image(String::from("screenshot.png"), true);
let template_gray = rgb2gray(&template, 0.2126, 0.7152, 0.0722);
let screen_gray = rgb2gray(&screenshot, 0.2126, 0.7152, 0.0722);
let matching_probability = arrayfire::match_template(&screen_gray, &template_gray, arrayfire::MatchType::LSAD);
af_print!("{:?}", matching_probability);
139569.0469 140099.2500 139869.8594 140015.7969 140680.9844 141952.5781 142602.7344 142870.7188...
from here I don't havy any idea how to get the best matching pixel coordinates.
Arrayfire doesn't provide "extremum" function, but separate min and max families of functions.
The one that provides index informations are prefixed with i.
imin_all and imax_all returns the min and max value indexes respectively wrapped in a tupple.
You can derive pixel position from value indexes and array dimensions, knowing that arrayfire is column major.
let template: Array<u8> = arrayfire::load_image(String::from("connect.png"), true);
let screenshot: Array<u8> = arrayfire::load_image(String::from("screenshot.png"), true);
let template_gray = rgb2gray(&template, 0.2126, 0.7152, 0.0722);
let screen_gray = rgb2gray(&screenshot, 0.2126, 0.7152, 0.0722);
let matching_probability = arrayfire::match_template(&screen_gray, &template_gray, arrayfire::MatchType::LSAD);
let (min, _, min_idx) = imin_all(&matching_probability);
let (max, _, max_idx) = imax_all(&matching_probability);
let dims = matching_probability.dims();
let [_, height, _, _] = dims.get();
let px_x_min = min_idx as u64 / height;
let px_y_min = min_idx as u64 % height;
let px_x_max = max_idx as u64 / height;
let px_y_max = max_idx as u64 % height;
af_print!("{:?}", matching_probability);
println!("Minimum value: {} is at pixel ({},{}).",min, px_x_min, px_y_min);
println!("Maximum value: {} is at pixel ({},{}).", max, px_x_max, px_y_max);
I am trying to position a box in an AR view and I need it to be at the same position as my AR Plane, or where the Plane Anchor is detected. My code is trying to call to the addBoxEToPlane func so that I can position BoxE the same as the plane, but it gives me the error "Value of tuple type '()' has no member 'position' ". If anybody can help me call to that function, or basically position that box the same as the AR Plane, I would really appreciate it!
func addBoxEToPlane(){
let boxE = SCNNode(geometry: SCNBox(width: 2.0, height: 2.0, length: 2.0, chamferRadius: 0))
boxE.geometry?.firstMaterial?.diffuse.contents = UIColor.red
}
func updateWith(anchor: ARPlaneAnchor) {
plane.width = CGFloat(anchor.extent.x)
plane.height = CGFloat(anchor.extent.z)
position = SCNVector3Make(anchor.center.x, 0, anchor.center.z)
if let grid = plane.materials.first as? GridMaterial {
grid.updateWith(anchor: anchor)
}
let XCoord = (plane.width)/2
let YCoord = (plane.height)/2
addBoxEToPlane().position = SCNVector3(XCoord, YCoord, anchor.center.z)
}
This code shows how to make a function mutate its input - one of the things we come to F# to avoid.
type Age = { mutable n : int }
let printInside a = printfn "Inside = %d" a.n
let inside a =
a.n <- a.n + 1
a.n
let a = {n = 1}
printInside a //a = 1
inside a
printInside a //a = 2
That being said, how do I do the same bad thing with [<Struct>] Records? I suspect that ref or byref may be involved but I just can't seem to get it to work.
type [<Struct>] Age = { mutable n : int }
let printInside a = printfn "Inside = %d" a.n
let inside a =
a.n <- a.n + 1
a.n
let a = {n = 1}
printInside a //a = 1
inside a
printInside a //a = 2
The fundamental issue is that a mutable field can only be modified if the struct itself is mutable. As you noted, we need to use byref in the declaration of Age. We also need to make sure a is mutable and lastly we need to use the & operator when calling the function inside. The & is the way to call a function with a byref parameter.
type [<Struct>] Age = { mutable n : int }
let printInside a = printfn "Inside = %d" a.n
let inside (a : Age byref) =
a.n <- a.n + 1
a.n
let mutable a = {n = 1}
printInside a //a = 1
inside &a
printInside a //a = 2
Now that I get the pattern, here is a simple example (just an int instead of a struct record) of how to mutate values passed into a function:
let mutable a = 1
let mutate (a : byref<_>) = a <- a + 1
mutate &a
a //a = 2
let prog =
"""//Kernel code:
extern "C" {
#pragma pack(1)
typedef struct {
int length;
float *pointer;
} global_array_float;
__global__ void kernel_main(global_array_float x){
printf("(on device) x.length=%d\n",x.length); // prints: (on device) x.length=10
printf("(on device) x.pointer=%lld\n",x.pointer); // prints: (on device) x.pointer=0
printf("sizeof(global_array_float)=%d", sizeof(global_array_float)); // 12 bytes just as expected
}
;}"""
printfn "%s" prog
let cuda_kernel = compile_kernel prog "kernel_main"
let test_launcher(str: CudaStream, kernel: CudaKernel, x: CudaGlobalArray<float32>, o: CudaGlobalArray<float32>) =
let block_size = 1
kernel.GridDimensions <- dim3(1)
kernel.BlockDimensions <- dim3(block_size)
printfn "(on host) x.length=%i" x.length // prints: (on host) x.length=10
printfn "(on host) x.pointer=%i" x.pointer // prints: (on host) x.pointer=21535919104
let args: obj [] = [|x.length;x.pointer|]
kernel.RunAsync(str.Stream, args)
let cols, rows = 10, 1
let a = d2M.create((rows,cols))
|> fun x -> fillRandomUniformMatrix ctx.Str x 1.0f 0.0f; x
let a' = d2MtoCudaArray a
//printfn "%A" (getd2M a)
let o = d2M.create((rows,cols)) // o does nothing here as this is a minimalist example.
let o' = d2MtoCudaArray o
test_launcher(ctx.Str,cuda_kernel,a',o')
cuda_context.Synchronize()
//printfn "%A" (getd2M o)
Here is an excerpt from the main repo that I am working on currently. I am very close to having a working F# quotations to Cuda C compiler, but I can't figure out how to pass the arguments into the function properly from the host side.
Despite the pack pragma, the NVRTC 7.5 Cuda compiler is doing some other optimization and I have no idea what it is.
Because I am working off F# quotations, I need to pass the arguments as a single struct for this to work. If I change the function from kernel_main(global_array_float x) to something like kernel_main(int x_length, float *x_pointer) then it works, but I that is not the form which the quotations system gives me upfront and I would like to avoid doing extra work to make F# more like C.
Any idea what I could try?
I've made two mistaken assumptions.
First error is assuming that let args: obj [] = [|x.length;x.pointer|] would get neatly placed on stack next to each other. In actuality these are two different arguments and the second one gets lost somewhere when passed along like in the above.
It can be fixed by making a custom struct type and rewriting the expression like so: let args: obj [] = [|CudaLocalArray(x.length,x.pointer)|].
The other mistaken assumption that I found when I rewrote it like the above is that using [<StructLayout(LayoutKind.Sequential>] does not mean the fields will be packed together. Instead, like for C, pack is a argument, so it needs to be used like so: [<StructLayout(LayoutKind.Sequential,Pack=1)>].
I'm having a problem with understanding scenekit geometery.
I have the default cube from Blender, and I export as collada (DAE), and can bring it into scenekit.... all good.
Now I want to see the vertices for the cube. In the DAE I can see the following for the "Cube-mesh-positions-array",
"1 1 -1 1 -1 -1 -1 -0.9999998 -1 -0.9999997 1 -1 1 0.9999995 1 0.9999994 -1.000001 1 -1 -0.9999997 1 -1 1 1"
Now what I'd like to do in scenekit, is get the vertices back, using something like the following:
SCNGeometrySource *vertexBuffer = [[cubeNode.geometry geometrySourcesForSemantic:SCNGeometrySourceSemanticVertex] objectAtIndex:0];
If I process the vertexBuffer (I've tried numerous methods of looking at the data), it doesn't seem correct.
Can somebody explain what "SCNGeometrySourceSemanticVertex" is giving me, and how to extract the vertex data properly? What I'd like to see is:
X = "float"
Y = "float"
Z = "float"
Also I was investigating the following class / methods, which looked promising (some good data values here), but the data from gmpe appears empty, is anybody able to explain what the data property of "SCNGeometryElement" contains?
SCNGeometryElement *gmpe = [theCurrentNode.geometry geometryElementAtIndex:0];
Thanks, assistance much appreciated,
D
The geometry source
When you call geometrySourcesForSemantic: you are given back an array of SCNGeometrySource objects with the given semantic in your case the sources for the vertex data).
This data could have been encoded in many different ways and a multiple sources can use the same data with a different stride and offset. The source itself has a bunch of properties for you to be able to decode the data like for example
dataStride
dataOffset
vectorCount
componentsPerVector
bytesPerComponent
You can use combinations of these to figure out which parts of the data to read and make vertices out of them.
Decoding
The stride tells you how many bytes you should step to get to the next vector and the offset tells you how many bytes offset from the start of that vector you should offset before getting to the relevant pars of the data for that vector. The number of bytes you should read for each vector is componentsPerVector * bytesPerComponent
Code to read out all the vertices for a single geometry source would look something like this
// Get the vertex sources
NSArray *vertexSources = [geometry geometrySourcesForSemantic:SCNGeometrySourceSemanticVertex];
// Get the first source
SCNGeometrySource *vertexSource = vertexSources[0]; // TODO: Parse all the sources
NSInteger stride = vertexSource.dataStride; // in bytes
NSInteger offset = vertexSource.dataOffset; // in bytes
NSInteger componentsPerVector = vertexSource.componentsPerVector;
NSInteger bytesPerVector = componentsPerVector * vertexSource.bytesPerComponent;
NSInteger vectorCount = vertexSource.vectorCount;
SCNVector3 vertices[vectorCount]; // A new array for vertices
// for each vector, read the bytes
for (NSInteger i=0; i<vectorCount; i++) {
// Assuming that bytes per component is 4 (a float)
// If it was 8 then it would be a double (aka CGFloat)
float vectorData[componentsPerVector];
// The range of bytes for this vector
NSRange byteRange = NSMakeRange(i*stride + offset, // Start at current stride + offset
bytesPerVector); // and read the lenght of one vector
// Read into the vector data buffer
[vertexSource.data getBytes:&vectorData range:byteRange];
// At this point you can read the data from the float array
float x = vectorData[0];
float y = vectorData[1];
float z = vectorData[2];
// ... Maybe even save it as an SCNVector3 for later use ...
vertices[i] = SCNVector3Make(x, y, z);
// ... or just log it
NSLog(#"x:%f, y:%f, z:%f", x, y, z);
}
The geometry element
This will give you all the vertices but won't tell you how they are used to construct the geometry. For that you need the geometry element that manages the indices for the vertices.
You can get the number of geometry elements for a piece of geometry from the geometryElementCount property. Then you can get the different elements using geometryElementAtIndex:.
The element can tell you if the vertices are used a individual triangles or a triangle strip. It also tells you the bytes per index (the indices may have been ints or shorts which will be necessary to decode its data.
Here is an extension method if the data isn't contiguous (the vector size isn't equal to the stride) which can be the case when the geometry is loaded from a DAE file. It also doesn't use copyByte function.
extension SCNGeometry{
/**
Get the vertices (3d points coordinates) of the geometry.
- returns: An array of SCNVector3 containing the vertices of the geometry.
*/
func vertices() -> [SCNVector3]? {
let sources = self.sources(for: .vertex)
guard let source = sources.first else{return nil}
let stride = source.dataStride / source.bytesPerComponent
let offset = source.dataOffset / source.bytesPerComponent
let vectorCount = source.vectorCount
return source.data.withUnsafeBytes { (buffer : UnsafePointer<Float>) -> [SCNVector3] in
var result = Array<SCNVector3>()
for i in 0...vectorCount - 1 {
let start = i * stride + offset
let x = buffer[start]
let y = buffer[start + 1]
let z = buffer[start + 2]
result.append(SCNVector3(x, y, z))
}
return result
}
}
}
The Swift Version
The Objective-C version and this are essentially identical.
let planeSources = _planeNode?.geometry?.geometrySourcesForSemantic(SCNGeometrySourceSemanticVertex)
if let planeSource = planeSources?.first {
let stride = planeSource.dataStride
let offset = planeSource.dataOffset
let componentsPerVector = planeSource.componentsPerVector
let bytesPerVector = componentsPerVector * planeSource.bytesPerComponent
let vectors = [SCNVector3](count: planeSource.vectorCount, repeatedValue: SCNVector3Zero)
let vertices = vectors.enumerate().map({
(index: Int, element: SCNVector3) -> SCNVector3 in
var vectorData = [Float](count: componentsPerVector, repeatedValue: 0)
let byteRange = NSMakeRange(index * stride + offset, bytesPerVector)
planeSource.data.getBytes(&vectorData, range: byteRange)
return SCNVector3Make(vectorData[0], vectorData[1], vectorData[2])
})
// You have your vertices, now what?
}
Here's a Swift 5.3 version, based on the other answers, and that also supports a bytesPerComponent different from 4 (untested for size different from 4 though):
extension SCNGeometrySource {
var vertices: [SCNVector3] {
let stride = self.dataStride
let offset = self.dataOffset
let componentsPerVector = self.componentsPerVector
let bytesPerVector = componentsPerVector * self.bytesPerComponent
func vectorFromData<FloatingPoint: BinaryFloatingPoint>(_ float: FloatingPoint.Type, index: Int) -> SCNVector3 {
assert(bytesPerComponent == MemoryLayout<FloatingPoint>.size)
let vectorData = UnsafeMutablePointer<FloatingPoint>.allocate(capacity: componentsPerVector)
defer {
vectorData.deallocate()
}
let buffer = UnsafeMutableBufferPointer(start: vectorData, count: componentsPerVector)
let rangeStart = index * stride + offset
self.data.copyBytes(to: buffer, from: rangeStart..<(rangeStart + bytesPerVector))
return SCNVector3(
CGFloat.NativeType(vectorData[0]),
CGFloat.NativeType(vectorData[1]),
CGFloat.NativeType(vectorData[2])
)
}
let vectors = [SCNVector3](repeating: SCNVector3Zero, count: self.vectorCount)
return vectors.indices.map { index -> SCNVector3 in
switch bytesPerComponent {
case 4:
return vectorFromData(Float32.self, index: index)
case 8:
return vectorFromData(Float64.self, index: index)
case 16:
return vectorFromData(Float80.self, index: index)
default:
return SCNVector3Zero
}
}
}
}
// call this function _ = vertices(node: mySceneView.scene!.rootNode)
// I have get the volume in Swift 4.2 :--- this function
func vertices(node:SCNNode) -> [SCNVector3] {
let planeSources1 = node.childNodes.first?.geometry
let planeSources = planeSources1?.sources(for: SCNGeometrySource.Semantic.vertex)
if let planeSource = planeSources?.first {
let stride = planeSource.dataStride
let offset = planeSource.dataOffset
let componentsPerVector = planeSource.componentsPerVector
let bytesPerVector = componentsPerVector * planeSource.bytesPerComponent
let vectors = [SCNVector3](repeating: SCNVector3Zero, count: planeSource.vectorCount)
let vertices = vectors.enumerated().map({
(index: Int, element: SCNVector3) -> SCNVector3 in
let vectorData = UnsafeMutablePointer<Float>.allocate(capacity: componentsPerVector)
let nsByteRange = NSMakeRange(index * stride + offset, bytesPerVector)
let byteRange = Range(nsByteRange)
let buffer = UnsafeMutableBufferPointer(start: vectorData, count: componentsPerVector)
planeSource.data.copyBytes(to: buffer, from: byteRange)
return SCNVector3Make(buffer[0], buffer[1], buffer[2])
})
var totalVolume = Float()
var x1 = Float(),x2 = Float(),x3 = Float(),y1 = Float(),y2 = Float(),y3 = Float(),z1 = Float(),z2 = Float(),z3 = Float()
var i = 0
while i < vertices.count{
x1 = vertices[i].x;
y1 = vertices[i].y;
z1 = vertices[i].z;
x2 = vertices[i + 1].x;
y2 = vertices[i + 1].y;
z2 = vertices[i + 1].z;
x3 = vertices[i + 2].x;
y3 = vertices[i + 2].y;
z3 = vertices[i + 2].z;
totalVolume +=
(-x3 * y2 * z1 +
x2 * y3 * z1 +
x3 * y1 * z2 -
x1 * y3 * z2 -
x2 * y1 * z3 +
x1 * y2 * z3);
i = i + 3
}
totalVolume = totalVolume / 6;
volume = "\(totalVolume)"
print("Volume Volume Volume Volume Volume Volume Volume :\(totalVolume)")
lbl_valume.text = "\(clean(String(totalVolume))) cubic mm"
}
return[]
}
With swift 3.1 you can extract vertices from SCNGeometry in a much faster and shorter way:
func vertices(node:SCNNode) -> [SCNVector3] {
let vertexSources = node.geometry?.getGeometrySources(for: SCNGeometrySource.Semantic.vertex)
if let vertexSource = vertexSources?.first {
let count = vertexSource.data.count / MemoryLayout<SCNVector3>.size
return vertexSource.data.withUnsafeBytes {
[SCNVector3](UnsafeBufferPointer<SCNVector3>(start: $0, count: count))
}
}
return []
}
...
Today i've noted that on osx this not going to work correct. This happens because on iOS SCNVector3 build with Float and on osx CGFloat (only apple good do smth simple so suffering). So I had to tweak the code for osx but this not gonna work as fast as on iOS.
func vertices() -> [SCNVector3] {
let vertexSources = sources(for: SCNGeometrySource.Semantic.vertex)
if let vertexSource = vertexSources.first {
let count = vertexSource.vectorCount * 3
let values = vertexSource.data.withUnsafeBytes {
[Float](UnsafeBufferPointer<Float>(start: $0, count: count))
}
var vectors = [SCNVector3]()
for i in 0..<vertexSource.vectorCount {
let offset = i * 3
vectors.append(SCNVector3Make(
CGFloat(values[offset]),
CGFloat(values[offset + 1]),
CGFloat(values[offset + 2])
))
}
return vectors
}
return []
}
For someone like me want to extract data of face from SCNGeometryElement.
Notice I only consider primtive type is triangle and index size is 2 or 4.
void extractInfoFromGeoElement(NSString* scenePath){
NSURL *url = [NSURL fileURLWithPath:scenePath];
SCNScene *scene = [SCNScene sceneWithURL:url options:nil error:nil];
SCNGeometry *geo = scene.rootNode.childNodes.firstObject.geometry;
SCNGeometryElement *elem = geo.geometryElements.firstObject;
NSInteger componentOfPrimitive = (elem.primitiveType == SCNGeometryPrimitiveTypeTriangles) ? 3 : 0;
if (!componentOfPrimitive) {//TODO: Code deals with triangle primitive only
return;
}
for (int i=0; i<elem.primitiveCount; i++) {
void *idxsPtr = NULL;
int stride = 3*i;
if (elem.bytesPerIndex == 2) {
short *idxsShort = malloc(sizeof(short)*3);
idxsPtr = idxsShort;
}else if (elem.bytesPerIndex == 4){
int *idxsInt = malloc(sizeof(int)*3);
idxsPtr = idxsInt;
}else{
NSLog(#"unknow index type");
return;
}
[elem.data getBytes:idxsPtr range:NSMakeRange(stride*elem.bytesPerIndex, elem.bytesPerIndex*3)];
if (elem.bytesPerIndex == 2) {
NSLog(#"triangle %d : %d, %d, %d\n",i,*(short*)idxsPtr,*((short*)idxsPtr+1),*((short*)idxsPtr+2));
}else{
NSLog(#"triangle %d : %d, %d, %d\n",i,*(int*)idxsPtr,*((int*)idxsPtr+1),*((int*)idxsPtr+2));
}
//Free
free(idxsPtr);
}
}
The Swift 3 version:
// `plane` is some kind of `SCNGeometry`
let planeSources = plane.geometry.sources(for: SCNGeometrySource.Semantic.vertex)
if let planeSource = planeSources.first {
let stride = planeSource.dataStride
let offset = planeSource.dataOffset
let componentsPerVector = planeSource.componentsPerVector
let bytesPerVector = componentsPerVector * planeSource.bytesPerComponent
let vectors = [SCNVector3](repeating: SCNVector3Zero, count: planeSource.vectorCount)
let vertices = vectors.enumerated().map({
(index: Int, element: SCNVector3) -> SCNVector3 in
let vectorData = UnsafeMutablePointer<Float>.allocate(capacity: componentsPerVector)
let nsByteRange = NSMakeRange(index * stride + offset, bytesPerVector)
let byteRange = Range(nsByteRange)
let buffer = UnsafeMutableBufferPointer(start: vectorData, count: componentsPerVector)
planeSource.data.copyBytes(to: buffer, from: byteRange)
let vector = SCNVector3Make(buffer[0], buffer[1], buffer[2])
})
// Use `vertices` here: vertices[0].x, vertices[0].y, vertices[0].z
}
OK, here is another Swift 5.5 version based on Oliver's answer.
extension SCNGeometry{
/**
Get the vertices (3d points coordinates) of the geometry.
- returns: An array of SCNVector3 containing the vertices of the geometry.
*/
func vertices() -> [SCNVector3]? {
let sources = self.sources(for: .vertex)
guard let source = sources.first else{return nil}
let stride = source.dataStride / source.bytesPerComponent
let offset = source.dataOffset / source.bytesPerComponent
let vectorCount = source.vectorCount
return source.data.withUnsafeBytes { dataBytes in
let buffer: UnsafePointer<Float> = dataBytes.baseAddress!.assumingMemoryBound(to: Float.self)
var result = Array<SCNVector3>()
for i in 0...vectorCount - 1 {
let start = i * stride + offset
let x = buffer[start]
let y = buffer[start + 1]
let z = buffer[start + 2]
result.append(SCNVector3(x, y, z))
}
return result
}
}
}
To use it you simply create a standard shape from which you can extract the vertex and rebuild the index.
let g = SCNSphere(radius: 1)
let newNode = SCNNode(geometry: g)
let vectors = newNode.geometry?.vertices()
var indices:[Int32] = []
for i in stride(from: 0, to: vectors!.count, by: 1) {
indices.append(Int32(i))
indices.append(Int32(i+1))
}
return self.createGeometry(
vertices:vectors!, indices: indices,
primitiveType: SCNGeometryPrimitiveType.line)
The createGeometry extension can be found here
It draws this...