I'm trying to use https://deno.land/x/opencv#v4.3.0-10 to get template matching to work in deno. I heavily based my code on the node example provided, but can't seem to work it out just yet.
By following the source code I first stumbled upon error: Uncaught (in promise) TypeError: Cannot convert "undefined" to int while calling cv.matFromImageData(imageSource).
After experimenting and searching I figured the function expects {data: Uint8ClampedArray, height: number, width: number}. This is based on this SO post and might be incorrect, hence posting it here.
The issue I'm currently faced with is that I don't seem to get proper matches from my template. Only when I set the threshold to 0.1 or lower, I get a match, but this is not correct { xStart: 0, yStart: 0, xEnd: 29, yEnd: 25 }.
I used the images provided by the templateMatching example here.
Haystack
Needle
Any input/thoughts on this are appreciated.
import { cv } from 'https://deno.land/x/opencv#v4.3.0-10/mod.ts';
export const match = (imagePath: string, templatePath: string) => {
const imageSource = Deno.readFileSync(imagePath);
const imageTemplate = Deno.readFileSync(templatePath);
const src = cv.matFromImageData({ data: imageSource, width: 640, height: 640 });
const templ = cv.matFromImageData({ data: imageTemplate, width: 29, height: 25 });
const processedImage = new cv.Mat();
const logResult = new cv.Mat();
const mask = new cv.Mat();
cv.matchTemplate(src, templ, processedImage, cv.TM_SQDIFF, mask);
cv.log(processedImage, logResult)
console.log(logResult.empty())
};
UPDATE
Using #ChristophRackwitz's answer and digging into opencv(js) docs, I managed to get close to my goal.
I decided to step down from taking multiple matches into account, and focused on a single (best) match of my needle in the haystack. Since ultimately this is my use-case anyways.
Going through the code provided in this example and comparing data with the data in my code, I figured something was off with the binary image data which I supplied to cv.matFromImageData. I solved this my properly decoding the png and passing that decoded image's bitmap to cv.matFromImageData.
I used TM_SQDIFF as suggested, and got some great results.
Haystack
Needle
Result
I achieved this in the following way.
import { cv } from 'https://deno.land/x/opencv#v4.3.0-10/mod.ts';
import { Image } from 'https://deno.land/x/imagescript#v1.2.14/mod.ts';
export type Match = false | {
x: number;
y: number;
width: number;
height: number;
center?: {
x: number;
y: number;
};
};
export const match = async (haystackPath: string, needlePath: string, drawOutput = false): Promise<Match> => {
const perfStart = performance.now()
const haystack = await Image.decode(Deno.readFileSync(haystackPath));
const needle = await Image.decode(Deno.readFileSync(needlePath));
const haystackMat = cv.matFromImageData({
data: haystack.bitmap,
width: haystack.width,
height: haystack.height,
});
const needleMat = cv.matFromImageData({
data: needle.bitmap,
width: needle.width,
height: needle.height,
});
const dest = new cv.Mat();
const mask = new cv.Mat();
cv.matchTemplate(haystackMat, needleMat, dest, cv.TM_SQDIFF, mask);
const result = cv.minMaxLoc(dest, mask);
const match: Match = {
x: result.minLoc.x,
y: result.minLoc.y,
width: needleMat.cols,
height: needleMat.rows,
};
match.center = {
x: match.x + (match.width * 0.5),
y: match.y + (match.height * 0.5),
};
if (drawOutput) {
haystack.drawBox(
match.x,
match.y,
match.width,
match.height,
Image.rgbaToColor(255, 0, 0, 255),
);
Deno.writeFileSync(`${haystackPath.replace('.png', '-result.png')}`, await haystack.encode(0));
}
const perfEnd = performance.now()
console.log(`Match took ${perfEnd - perfStart}ms`)
return match.x > 0 || match.y > 0 ? match : false;
};
ISSUE
The remaining issue is that I also get a false match when it should not match anything.
Based on what I know so far, I should be able to solve this using a threshold like so:
cv.threshold(dest, dest, 0.9, 1, cv.THRESH_BINARY);
Adding this line after matchTemplate however makes it indeed so that I no longer get false matches when I don't expect them, but I also no longer get a match when I DO expect them.
Obviously I am missing something on how to work with the cv threshold. Any advice on that?
UPDATE 2
After experimenting and reading some more I managed to get it to work with normalised values like so:
cv.matchTemplate(haystackMat, needleMat, dest, cv.TM_SQDIFF_NORMED, mask);
cv.threshold(dest, dest, 0.01, 1, cv.THRESH_BINARY);
Other than it being normalised it seems to do the trick consistently for me. However, I would still like to know why I cant get it to work without using normalised values. So any input is still appreciated. Will mark this post as solved in a few days to give people the chance to discus the topic some more while it's still relevant.
The TM_* methods of matchTemplate are treacherous. And the docs throw formulas at you that would make anyone feel dumb, because they're code, not explanation.
Consider the calculation of one correlation: one particular position of the template/"needle" on the "haystack".
All the CCORR modes will simply multiply elementwise. Your data uses white as "background", which is a "DC offset". The signal, the difference to white of anything not-white, will drown in the large "DC offset" of your data. The calculated correlation coefficients will vary mostly with the DC offset and hardly at all with the actual signal/difference.
This is what that looks like, roughly. The result of running with TM_CCOEFF_NORMED, overlaid on top of the haystack (with some padding). You're getting big fat responses for all instances of all shapes, no matter their specific shape.
You want to use differences instead. The SQDIFF modes will handle that. Squared differences are a measure of dissimilarity, i.e. a perfect match will give you 0.
Let's look at some values...
(hh, hw) = haystack.shape[:2]
(nh, nw) = needle.shape[:2]
scores = cv.matchTemplate(image=haystack, templ=needle, method=cv.TM_SQDIFF)
(sh, sw) = scores.shape # will be shaped like haystack - needle
scores = np.log10(1+scores) # any log will do
maxscore = 255**2 * (nh * nw * 3)
# maximum conceivable SQDIFF score, 3-channel data, any needle
# for a specific needle:
#maxscore = (np.maximum(needle, 255-needle, dtype=np.float32)**2).sum()
# map range linearly, from [0 .. ~8] to [1 .. 0] (white to black)
(smin, smax) = (0.0, np.log10(1+maxscore))
(omin, omax) = (1.0, 0.0)
print("mapping from", (smin, smax), "to", (omin, omax))
out = (scores - smin) / (smax - smin) * (omax - omin) + omin
You'll see gray peaks, but some are actually (close to) white while others aren't. Those are truly instances of the needle image. The other instances differ more from the needle, so they're just some reddish shapes that kinda look like the needle.
Now you can find local extrema. There are many ways to do that. You'll want to do two things: filter by absolute value (threshold) and suppress non-maxima (scores above threshold that are dominated by better nearby score). I'll just do the filtering and pretend there aren't any nearby non-maxima because the resulting peaks fall off strongly enough. If that happens to not be the case, you'd see double drawing in the picture below, boxes becoming "bold" because they're drawn twice onto adjacent pixel positions.
I'm picking a threshold of 2.0 because that represents a difference of 100, i.e. one color value in one pixel may have differed by 10 (10*10 = 100), or two values may have differed by 7 (7*7 = 49, twice makes 98), ... so it's still a very tiny, imperceptible difference. A threshold of 6 would mean a sum of squared differences of upto a million, allowing for a lot more difference.
(i,j) = (scores <= 2.0).nonzero() # threshold "empirically decided"
instances = np.transpose([j,i]) # list of (x,y) points
That's giving me 16 instances.
canvas = haystack.copy()
for pt in instances:
(j,i) = pt
score = scores[i,j]
cv.rectangle(canvas,
pt1=(pt-(1,1)).astype(int), pt2=(pt+(nw,nh)).astype(int),
color=(255,0,0), thickness=1)
cv.putText(canvas,
text=f"{score:.2f}",
org=(pt+[0,-5]).astype(int),
fontFace=cv.FONT_HERSHEY_SIMPLEX, fontScale=0.4,
color=(255,0,0), thickness=1)
That's drawing a box around each, with the logarithm of the score above it.
One simple approach to get candidates for Non-Maxima Suppression (NMS) is to cv.dilate the scores and equality-compare, to gain a mask of candidates. Those scores that are local maxima, will compare equal to themselves (the dilated array), and every surrounding score will be less. This alone will have some corner cases you will need to handle. Note: at this stage, those are local maxima of any value. You need to combine (logical and) that with a mask from thresholding the values.
NMS commonly is required to handle immediate neighbors being above the threshold, and merge them or pick the better one. You can do that by simply running connectedComponents(WithStats) and taking the blob centers. I think that's clearly better than trying to find contours.
The dilate-and-compare approach will not suppress neighbors if they have the same score. If you did the connectedComponents step, you only have non-immediate neighbors to deal with here. What to do is up to you. It's not clear cut anyway.
Related
I have a binary mask of a road, the mask is a little irregular(sometimes even more than depicted in the image).
I have tried houghLine in OpenCV to detect boundary lines, but the boundary lines are not as expected. I tried erosion and dilation to smooth out things, but no luck. Also since the path is curved it becomes even difficult to detect boundary lines using houghLines. How can I modify the code to detect lines better?
img2=cv2.erode(img2,None,iterations=2)
img2=cv2.dilate(img2,None,iterations=2)
can=cv2.Canny(img2,150,50)
lines=cv2.HoughLinesP(can,1,np.pi/180,50,maxLineGap=50,minLineLength=10)
if(lines is not None):
for x in lines:
#print(lines[0])
#mask=np.zeros(frame2.shape,dtype=np.uint8)
#roi=lines
#cv2.fillPoly(mask,roi,(255,255,255))
#cv2.imshow(mask)
for x1,y1,x2,y2 in x:
cv2.line(frame2,(x1,y1),(x2,y2),(255,0,0),2)
You say that Hough is failing but you don't say why. Why is your output "not as expected"? In my experience, Hough Line Detection’s critical points are two: 1) The edges mask you pass to it and 2) how you filter the resulting lines. You should be fine-tuning those two steps and Hough should be enough for your problem.
I don't know what kind of problems the line detector is giving you, but suppose you are interested (as your question suggests) in other methods for lane detection. There are at least two things you could try: 1) Bird's eye transform of the road – which makes line detection much easier since all your lines are now parallel lines. And 2) Contour detection (instead of lines).
Let's examine 2 and what kind of results you can obtain. Listen, man, I offer my answer in C++, but I make notes along with it. I try to highlight the important ideas, so you can implement them in your language of choice. However, if all you want is a CTRL+C and CTRL+V solution, that's ok, but this answer won't help you.
Ok, let's start by reading the image and converting it to binary. Our goal here is to first obtain the edges. Pretty standard stuff:
//Read input image:
std::string imagePath = "C://opencvImages//lanesMask.png";
cv::Mat testImage = cv::imread( imagePath );
//Convert BGR to Gray:
cv::Mat grayImage;
cv::cvtColor( testImage, grayImage, cv::COLOR_RGB2GRAY );
//Get binary image via Otsu:
cv::Mat binaryImage;
cv::threshold( grayImage, binaryImage, 0, 255, cv::THRESH_OTSU );
Now, simply pass this image to Canny's Edge detector. The parameters are also pretty standard. As per Canny's documentation, the ratios between lower and upper thresholds are related by a factor of 3:
//Get Edges via Canny:
cv::Mat testEdges;
//Setup lower and upper thresholds for edge detection:
float lowerThreshold = 30;
float upperThreshold = 3 * lowerThreshold;
cv::Canny( binaryImage, testEdges, lowerThreshold, upperThreshold );
Your mask is pretty good; these are the edges Canny finds:
Now, here's where we are trying something different. We won't use Hough's line detection, instead, let's find the contours of the mask. Each contour is made of points. What we are looking for is actually lines, straight lines that can be fitted to these points. There's more than a method for achieving that. I propose K-means, a clustering algorithm.
The idea is that the points, as you can see, can be clustered in 4 groups: The vanishing point of the lanes (those should be 2 endpoints there) and the 2 starting points of the road. If we give K-means the points of the contour and tell it to cluster the data in 4 separate groups, we should get the means (location) of those 4 points.
Let's try it out. The first step is to find the contours in the edges mask:
//Get contours:
std::vector< std::vector<cv::Point> > contours;
std::vector< cv::Vec4i > hierarchy;
cv::findContours( testEdges, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, cv::Point(0, 0) );
K-means needs a specific data type on its input. I'll use a cv::Point2f vector to store all the contour points. Let's set up the variables used by K-means:
//Set up the data containers used by K-means:
cv::Mat centers; cv::Mat labels;
std::vector<cv::Point2f> points; //the data for clustering is stored here
Next, let's loop through the contours and store each point inside the Point2f vector, so we can further pass it to K-means. Let’s use the loop to also draw the contours and make sure we are not messing things up:
//Loop thru the found contours:
for( int i = 0; i < (int)contours.size(); i++ ){
//Set a color & draw contours:
cv::Scalar color = cv::Scalar( 0, 256, 0 );
cv::drawContours( testImage, contours, i, color, 2, 8, hierarchy, 0, cv::Point() );
//This is the current vector of points that is being processed:
std::vector<cv::Point> currentVecPoint = contours[i];
//Loop thru it and store each point as a float point inside a plain vector:
for(int k = 0; k < (int)currentVecPoint.size(); k++){
cv::Point currentPoint = currentVecPoint[k];
//Push (store) the point into the vector:
points.push_back( currentPoint );
}
}
These are the contours found:
There, now, I have the contour points in my vector. Let's pass the info on to K-means:
//Setup K-means:
int clusterCount = 4; //Number of clusters to split the set by
int attempts = 5; //Number of times the algorithm is executed using different initial labels
int flags = cv::KMEANS_PP_CENTERS;
cv::TermCriteria criteria = cv::TermCriteria( CV_TERMCRIT_ITER | CV_TERMCRIT_EPS, 10, 0.01 );
//The call to kmeans:
cv::kmeans( points, clusterCount, labels, criteria, attempts, flags, centers );
And that's all. The result of K-means is in the centers matrix. Each row of the matrix should have 2 columns, denoting a point center. In this case, the matrix is of size 4 x 2. Let's draw that info:
As expected, 4 center points, each is the mean of a cluster. Very cool, now, is this approximation enough for your application? Only you know that! You could work with those points and extend both lines, but that's a possible improvement of this result.
I have a polyline(1) that is withing group that is within another group. Both of these groups are transformed in every way. I need to get the exact locations of the polyline points after the transformations but it isn't easy since there is only transform attribute at hand.
I'm trying to replicate these transformations to another polyline(2) so I would get the transformed point locations. I get the globalMatrix from the polyline(1) and with the help of this: http://svg.dabbles.info/snaptut-matrix-play and Internet, I've come to this: http://jsfiddle.net/3c4fuvfc/3/
It works if there isn't any rotation applied. If rotation is applied then all the points are a little off. Of course the scaling isn't done yet, maybe it fixes this issue?
And then there is the issue of scaling: polyline(1) is sometimes flipped around ( typically s-1,1 or s1,-1). How is scaling supposed to be implemented here?
Is it important in which order these are done when trying to replicate transformations?
Is the decomposing done right, this seems odd:
scaleX: Math.sqrt(matrix.a * matrix.a + matrix.b * matrix.b),
scaleY: Math.sqrt(matrix.c * matrix.c + matrix.d * matrix.d),
Thank you
I'm not sure if I'm misunderstanding the problem, but you seem to be doing a lot of work for information you already have.
You have the matrix ( el.matrix().globalTransform ), so why not just apply to to each point. I'm not sure what what help decomposing the matrix is giving you ?
So you could do this, iterate over the points, apply the matrix, and your polyline is flattened with the existing matrix...
var m = r1.transform().globalMatrix;
var pts = poly.attr('points');
var ptsarray = [];
for( var c = 0; c < pts.length ; c += 2 ) {
ptsarray.push( m.x( pts[c], pts[c+1] ),
m.y( pts[c], pts[c+1] ) );
}
poly.attr('points', ptsarray )
jsfiddle
Transforming a coordinate using a Snap matrix can be found here
I have a path to create a shape - eg. an octagon
pathdetail="M50,83.33 L83.33,50 L116.66,50 L150,83.33 L150,116.66 L116.66,150 L83.33,150 L50,116.66Z";
paper.path(pathdetail);
paper.path(pathdetail).transform("S3.5");
I then use this to create the shape which I know the coordinates of each corner as they are in the pathdetail.
I then rescale it using transform("S3.5") - I need to be able to get the new coordinates of each corner in the new scaled shape - is this possible to do?
Raphael provides an utility to apply matrix transforms to paths, first you need to convert the transformation to a matrix, apply the transformation and apply it to the element:
var matrix = Raphael.toMatrix(pathdetail, "S3.5");
var newPath = Raphael.mapPath(pathdetail, matrix);
octagon.path(newPath);
If I understand correctly, you want to find the transformed coordinates of each of the eight points in your octagon -- correct? If so, Raphael does not have an out of the box solution for you, but you should be able to get the information you need relatively easily using some of Raphael's core utility functions.
My recommendation would be something like this:
var pathdetail = "your path definition here. Your path uses only absolute coordinates... right?";
var pathdetail = Raphael.transformPath( pathdetail, "your transform string" );
// pathdetail will now still be a string full of path notation, but its coordinates will be transformed appropriately
var pathparts = Raphael.parsePathString( pathdetail );
var cornerList = [];
// pathparts will not be an array of path elements, each of which will be parsed into a subarray whose elements consist of a command and 0 or more parameters.
// The following logic assumes that your path string uses ONLY ABSOLUTE COORDINATES and does
// not take relative coordinates (or H/V directives) into account. You should be able to
// code around this with only a little additional logic =)
for ( var i = 0; i < pathparts.length; i++ )
{
switch( pathparts[i][0] )
{
case "M" :
case "L" :
// Capture the point
cornerList.push( { x: pathparts[i][1], y: pathparts[i][2] } );
break;
default :
console.log("Skipping irrelevant path directive '" + pathparts[i][0] + "'" );
break;
}
}
// At this point, the array cornerList should be populated with every discrete point in your path.
This is obviously an undesirable chunk of code to use inline and will only handle a subset of paths in the wild (though it could be expanded to be suitable for general purpose use). However, for the octagon case where the path string uses absolute coordinates, this -- or something much like it -- should give you exactly what you need.
How can I find the scale ratio a rotated Rect element in order fit it in a bounding rectangle (unrotated) of a specific size?
Basically, I want the opposite of getBoundingClientRect, setBoundingClientRect.
First you need to get the transform applied to the element, with <svg>.getTransformToElement, together with the result of rect.getBBox() you can calculate the actual size. Width this you can calculate the scale factor to the desired size and add it to the transform of the rect. With this I mean that you should multiply actual transform matrix with a new scale-matrix.
BUT: This is a description for a case where you are interested in the AABB, means axis aligned bounding box, what the result of getBoundingClientRect delivers, for the real, rotated bounding box, so the rectangle itself in this case, you need to calculate (and apply) the scale factor from the width and/or height.
Good luck…
EDIT::
function getSVGPoint( x, y, matrix ){
var p = this._dom.createSVGPoint();
p.x = x;
p.y = y;
if( matrix ){
p = p.matrixTransform( matrix );
}
return p;
}
function getGlobalBBox( el ){
var mtr = el.getTransformToElement( this._dom );
var bbox = el.getBBox();
var points = [
getSVGPoint.call( this, bbox.x + bbox.width, bbox.y, mtr ),
getSVGPoint.call( this, bbox.x, bbox.y, mtr ),
getSVGPoint.call( this, bbox.x, bbox.y + bbox.height, mtr ),
getSVGPoint.call( this, bbox.x + bbox.width, bbox.y + bbox.height, mtr ) ];
return points;
};
with this code i one time did a similar trick... this._dom refers to a <svg> and el to an element. The second function returns an array of points, beginning at the top-right edge, going on counter clockwise arround the bbox.
EDIT:
the result of <element>.getBBox() does not include the transform that is applied to the element and I guess that the new desired size is in absolute coordinates. So the first thing you need to is to make the »BBox« global.
Than you can calculate the scaling factor for sx and sy by:
var sx = desiredWidth / globalBBoxWidth;
var sy = desiredHeight / globalBBoxHeight;
var mtrx = <svg>.createSVGMatrix();
mtrx.a = sx;
mtrx.d = sy;
Than you have to append this matrix to the transform list of your element, or concatenate it with the actual and replace it, that depends on you implementation. The most confusion part of this trick is to make sure that you calculate the scaling factors with coordinates in the same transformation (where absolute ones are convenient). After this you apply the scaling to the transform of the <element>, do not replace the whole matrix, concatenate it with the actually applied one, or append it to the transform list as new item, but make sure that you do not insert it before existing item. In case of matrix concatenation make sure to preserve the order of multiplication.
The last steps depend on your Implementation, how you handle the transforms, if you do not know which possibilities you have, take a look here and take special care for the DOMInterfaces you need to implement this.
I think I may have stumbled onto a limitation of Flot, but I'm not sure. I'm trying to represent a single data series over time. The items' "State" is represented on the Y-Axis (there are 5 of them), and time is on the X-Axis (items can change states over time). I want the graph to have points and lines connecting those points for each data series.
In addition to tracking an item's State over time, I'd also like to represent it's "Status" at any of the particular points. This I would like to do by changing the color of the points. What this means is a single item may have different Statuses at different times, meaning for a single data series I need a line that connects different points (dots) of different colors.
The only thing I've seen so far is the ability to specify the color for all points in a given dataseries. Does anyone know if there's a way to specify colors individually?
Thanks.
There you go mate. You need to use a draw hook.
$(function () {
var d2 = [[0, 3], [4, 8], [8, 5], [9, 13]];
var colors = ["#cc4444", "#ff0000", "#0000ff", "#00ff00"];
var radius = [10, 20, 30, 40];
function raw(plot, ctx) {
var data = plot.getData();
var axes = plot.getAxes();
var offset = plot.getPlotOffset();
for (var i = 0; i < data.length; i++) {
var series = data[i];
for (var j = 0; j < series.data.length; j++) {
var color = colors[j];
var d = (series.data[j]);
var x = offset.left + axes.xaxis.p2c(d[0]);
var y = offset.top + axes.yaxis.p2c(d[1]);
var r = radius[j];
ctx.lineWidth = 2;
ctx.beginPath();
ctx.arc(x,y,r,0,Math.PI*2,true);
ctx.closePath();
ctx.fillStyle = color;
ctx.fill();
}
}
};
var plot = $.plot(
$("#placeholder"),
[{ data: d2, points: { show: true } }],
{ hooks: { draw : [raw] } }
);
});
With 3 views, it may not be worth answering my own question, but here's the solution:
My original problem was how to plot a dataseries of points and a line, but with each point being a color that I specify.
Flot only allows specifying colors of the dots at the dataseries level, meaning each color must be its own dataseries. With this in mind, the solution is to make a single dataseries for each color, and draw that dataseries with only points, and no lines. Then I must make a separate dataseries that is all of the dots I want connected by the line, and draw that one with no points, and only a line.
So if I want to show a line going through 5 points with five different colors, I need 6 dataseries: 5 for each point, and 1 for the line that connects them. Flot will simply draw everything on top of each other, and I believe there's a way to specify what gets shown on top (to make sure the dots are shown above the line).
Actually, it's not very difficult to add a feature to flot that would call back into your code to get the color for each point. It took me about an hour, and I'm not a javascript expert by any measure.
If you look at drawSeriesPoints(), all you have to do is pass a callback parameter to plotPoints() which will be used to set ctx.strokeStyle. I added an option called series.points.colorCallback, and drawSeriesPoints() either uses that, or a simple function that always returns the series.color.
One tricky point: the index you should pass to your callback probably isn't the i in plotPoints(), but rather i/ps.
Hope this helps!