I have tried to get hand mesh data from Hololens2 using MRTK V2 and Unity C#.
Now, I can get hand mesh data with turning on Hand Mesh Visualization option and referring MRTK HandTracking guide.
Unfortunately, the visualization(drawing hand CG) is heavy workload.
So, I would like to get hand mesh without turning on Hand Mesh Visualization option but OnHandMeshUpdated function is not called due to turning off Hand Mesh Visualization option.
Does anyone know how to get hand mesh data from Hololens2 without turning on Hand Mesh Visualization option?
Hand Mesh Visualization option
MRTK does not directly provide this feature. According to the source code of MRTK-Unity, check out the code line 163 of BaseHandVisualizer class, you will find the majority of jobs are processed in the OnHandMeshUpdated event handler. When the current hand mesh is updated based on the passed-in state of the hand, OnHandMeshUpdated method will be invoked with HandMeshInfo event data. Once Hand Mesh Prefab field in [InputSystem]->[Hand Tracking] is set as "None", MRTK will not instantiate handMeshFilter according to the conditional statement. But the hand mesh related data will be easily accessible from the event data. Check out the class definition of HandMeshInfo here.
Related
I'm using rust and the core-graphics crate so I don't have access to a lot of the conventional macos gui APIs. I also can't just do the drawing operations to a bitmap context and then draw that to the screen context, since I only have ownership of a handful of said operations and I want to capture everything in the context exactly as it's shown on-screen. That said:
Given a CGContextRef retrieved from NSGraphicsContext::currentContext representing content being drawn on the screen, is it possible to capture a region of that to a CGImage?
Based on my (limited) understanding of other graphics libraries I assume the general shape of the solution is to create a CGBitmapContext, draw onto it from my CGContextRef, and then call CGBitmapContext::CGBitmapContextCreateImage -- but I can't find any way to get data out of a screen-based CGContext.
I'm a big fan of PaperJS, however, the library doesn't see much activity so we're looking at other tools, like KonvaJS, Fabric, and Pixi. We'd like to replicate the example here:
http://paperjs.org/examples/path-simplification/
in KonvaJS but we're not sure which class is the most appropriate? Should we use the line, which is described as a collection of points with tension, the path tool which is what we use in Paper, or the shape class? Does KonvaJS offer the same type of access to the bezier curve tools and shape border, blue line, found in the above-mentioned paper example?
Konva.Line requires a list of x & y passed into its points property as a simple array, then draws straight lines connecting those points. The tension property can be used to make the straight line joins more curvy.
Konva.Path expects you to provide a data property that is more like a list of SVG drawing instructions, so move, lineto, arc, etc. (See supported instructions list in Konva docs for Konva.Path.data here)
There is no built-in equivalent path-editing features to those in the demo you linked to - so no automatic anchors on the path control points and no Bezier handles. You would have to DIY those. Having said that, it would all be achievable - what I mean is the drawing of the control anchors and lines, the listening for mouse and drag events, and the final passing back of the SVG drawing data to the Konva.Path shape when the path's edit mode ends are all well supported in Konva.
As at May 2022, the Konva lib is well supported, with appropriately frequent (as Goldilocks would want - not too many and not too few), no ill-thought-out breaking changes, issues are responded to, SO posts replied to, and there is a busy Discord channel.
My goal is to implement a method, that tracks persons in a single camera. For that, I'm using Scaled Yolov4 to detect persons in the scene, then I generate points inside of their bounding boxes using cv2.goodFeaturesToTrack, and track them using Lucas-Kanade Optical Flow cv2.calcOpticalFlowPyrLK.
the problem is, sometimes the points make huge jumps, and I can't tell why. The following video shows the problem I'm facing, specifically, on second 0:02, the green dots jumps in a weird manner which makes my method detects that person as a new person.
https://www.veed.io/view/37f98715-40c5-4c07-aa97-8c2242d7806c?sharingWidget=true
my question is, is it a limitation on LK optical flow, or I'm doing something wrong? And is there a recommended Optical Flow method for tracking, or an example implementation for Single Camera Multi Person Tracking using Optical Flow? because I couldn't find much literature or codes about it.
Is there an easy way to show 3D entity at all times, even when that entity is hidden behind another entity? For example, I want that lines are always shown event when they are behind mesh surface.
I use Qt3D framework.
Assuming that you are talking about the Qt3D framework I want to extend the answer of Rabbid76.
To disable depth-testing in the Qt3D framework, add a QRenderStateSet to the framegraph branch that renders things (the one that as a QViewPort for example) and add a QDepthTest to it. Then, set the depth function of the QDepthTest to always. This way, the depth test is always passed and entities in the back will also be drawn, depending on the drawing order. You can use QSortPolicy to adjust the drawing order to back-to-front.
But this won't work when the camera position changes and your entity that you always want to be drawn is in the front. I'd suggest you add another framegraph branch and use a QLayerFilter to only deactivate depth-testing for this one entity.
If your entity looks weird when deactivating depth-testing (likely for complex objects), you could replace the QDepthTest by a QClearBuffers and simply clear the depth buffer.
Have a look at my answer here, where I showed an example of a custom framegraph with depth test.
If the depth test is disabled, then the geometry (like a line) is always drawn on top of the previously drawn geometry. The depth test can be disabled by:
glDisable(GL_DEPTH_TEST)
See glEnable
As an alternative the depth test function, can be set to let a fragment always pass depth test. In Qt this can be done by the class QDepthTest, using the enumerator constant Qt3DRender::QDepthTest::Always.
In this case, you have to take care about the order in which the geometry is drawn.
You have to find a way, to render the polygons (opaque geometry) first, by using the depth test function Qt3DRender::QDepthTest::Less.
After that you have to render the lines on top, by using the depth test function Qt3DRender::QDepthTest::Always.
Is there any body of evidence that we could reference to help determine whether a person is using a device (smartphone/tablet) with their left hand or right hand?
My hunch is that you may be able to use accelerometer data to detect a slight tilt, perhaps only while the user is manipulating some sort of on screen input.
The answer I'm looking for would state something like, "research shows that 90% of right handed users that utilize an input mechanism tilt their phone an average of 5° while inputting data, while 90% of left handed users utilizing an input mechanism have their phone tilted an average of -5°".
Having this data, one would be able to read accelerometer data and be able to make informed decisions regarding placement of on screen items that might otherwise be in the way for left handed users or right handed users.
You can definitely do this but if it were me, I'd try a less complicated approach. First you need to recognize that not any specific approach will yield 100% accurate results - they will be guesses but hopefully highly probable ones. With that said, I'd explore the simple-to-capture data points of basic touch events. You can leverage these data points and pull x/y axis on start/end touch:
touchStart: Triggers when the user makes contact with the touch
surface and creates a touch point inside the element the event is
bound to.
touchEnd: Triggers when the user removes a touch point from the
surface.
Here's one way to do it - it could be reasoned that if a user is left handed, they will use their left thumb to scroll up/down on the page. Now, based on the way the thumb rotates, swiping up will naturally cause the arch of the swipe to move outwards. In the case of touch events, if the touchStart X is greater than touchEnd X, you could deduce they are left handed. The opposite could be true with a right handed person - for a swipe up, if the touchStart X is less than touchEnd X, you could deduce they are right handed. See here:
Here's one reference on getting started with touch events. Good luck!
http://www.javascriptkit.com/javatutors/touchevents.shtml
There are multiple approaches and papers discussing this topic. However, most of them are written between 2012-2016. After doing some research myself I came across a fairly new article that makes use of deep learning.
What sparked my interest is the fact that they do not rely on a swipe direction, speed or position but rather on the capacitive image each finger creates during a touch.
Highly recommend reading the full paper: http://huyle.de/wp-content/papercite-data/pdf/le2019investigating.pdf
Whats even better, the data set together with Python 3.6 scripts to preprocess the data as well as train and test the model described in the paper are released under the MIT license. They also provide the trained models and the software to
run the models on Android.
Git repo: https://github.com/interactionlab/CapFingerId