Gesture Recognition for my Grandma (Kinect) Linux - linux

I'm looking into making a project with the Kinect to allow my Grandma to control her TV without being daunted by using the remote. So, I've been looking into basic gesture recognition. The aim will be to say turn the volume of the TV up by sending the right IR code to the TV when the program detects that the right hand is being "waved."
The problem is, no matter where I look, I can't seem to find a Linux based tutorial which shows how to do something as a result of a gesture. One other thing to note is that I don't need to have any GUI apart from the debug window as this will slow my program down a fair bit.
Does anybody know of something somewhere which will allow me to in a loop, constantly check for some hand gesture and when it does, I can control something, without the need of any GUI at all, and on Linux? :/
I'm happy to go for any language but my experience revolves around Python and C.
Any help will be accepted with great appreciation.
Thanks in advance
Matt

In principle, this concept is great, but the amount of features a remote offers is going to be hard to replicate using a number of gestures that an older person can memorize. They will probably be even less incentivized to do this (learning new things sucks) if they already have a solution (remote), even though they really love you. I'm just warning you.
I recommend you use OpenNI and NITE. Note that the current version of OpenNI (2) does not have Kinect support. You need to use OpenNI 1.5.4 and look for the SensorKinect093 driver. There should be some gesture code that works for that (googling OpenNI Gesture yields a ton of results). If you're using something that expects OpenNI 2, be warned that you may have to write some glue code.
The basic control set would be Volume +/-, Channel +/-, Power on/off. But that will be frustrating if she wants to go from Channel 03 to 50.
I don't know how low-level you want to go, but a really, REALLY simple gesture recognize could look at horizontal and vertical swipes of the right hand exceeding a velocity threshold (averaged). Be warned: detected skeletons can get really wonky when people are sitting (that's actually a bit of what my PhD is on).

Related

J2ME app menu develop

I'm developing a small j2me game and i want to create a menu for this application. I imagine the menu as a vertical list of items with a cursor on the left or right side that i can move from item to item, something like this menu example but as a main menu.
What elements should i use to obtains such effects? I need only advices or links, i will develope it myself.
Thanks in advance!
import java.util.Vector;
import javax.microedition.lcdui.Canvas;
import javax.microedition.lcdui.Font;
import javax.microedition.lcdui.Graphics;
import javax.microedition.lcdui.Image;
What you plan looks doable. Can't give much links because don't recall any that could help on stuff like you're doing. Actually, most useful link for you will probably be MIDP (JSR 118) API reference - your part is going to be mostly lcdui package, and especially Graphics API.
As for advice, no problem. First thing to note is that there will be more coding and more (much more) testing/debugging than it was in your prior experiment with implicit list. If you can think of some possible deadline / timing requirements that may become a problem - just keep in mind that prior design with implicit list as a fallback. It won't look as fancy but it'll work work safe and correct.
Another important thing is to decide what kind devices you are going to target. For menu like one you are going to develop, it may be rather difficult to get consistent look and feel both at 160x200 basic phone with ITU-T keypad and on 400x600 touchscreen smartphone. Below I am going to assume you'll try to target as wide variety of devices as possible - note the narrower you can get it, the easier it will be to code and test.
When targeting lots of different devices it is helpful to use an emulator that can be configured to simulate various display sizes and resolution, presense or absence of touchscreen input etc. Keep in mind though that emulator alone won't fully simulate real device. To keep your feets on the ground, consider also some regular smoke testing of your application with real device, preferable using over-the-air (OTA) installation.
Here are some particular API tips that I can think of now.
Use Canvas.getGameAction to handle pressed key code - that is likely the most reliable/portable way to figure up/down and select actions for menu.
Use Canvas.hasPointerEvents to figure if there's touch screen support. Users with touch screen devices may get disappointed if it turns out that your fancy menu can't react when they tap on screen.
Use Font.getHeight and Font.stringWidth to figure how much space is occupied by menu item text.
Use Image.getGraphics if you want to draw something over the image object.
As I mentioned, you most likely will do a lot of stuff using lcdui.Graphics API. It's mostly rather simple, but you will probably need to understand somewhat tricky stuff about clipping. Good luck.

Touch Screen Running Windows CE

I'm starting my first project that runs on a 7 inch touch screen running Windows CE 6.0 (and NETCF 3.5).
The touch screen doesn't respond to touch too well when I use my finger. The only way for me to navigate around is by using a stylus (or similar).
Since I've never worked with Windows CE or a resistive touch screen, I'm not sure if I should expect to be able to use my finger or if the stylus method is, essentially, the only way to effectively navigate around. - or, maybe, I have a touch screen that simply isn't that good.
If you have experience with WinCE running on a touch screen, do you find that a stylus is the only way to go?
A resistive touchscreen can certainly provide feedback for a finger - I've even configured them for hands-in-gloves. It sounds like the touchscreen driver is tossing out the data samples it's getting from the panel and the key for you is going to be to figure out why.
In my experience there are really two primary reasons for your samples to be ignored.
The driver has been configured to too tight of a tolerance.
Sensitivity is often a configurable item. Maybe through recompile of the OS, maybe through the registry - depends on how your OEM implemented it. Check with the OEM and see if you can adjust it.
The panel has too much noise, causing your samples to get tossed
This one is easy to check. Drag a selection rectangle on the desktop with a stylus and hold the end point down (don't lift the stylus). Is it steady, or does it "wiggle" a lot at the final point? If so, you have noise. Grounding the panel usually helps, but it could be a hardware issue. I've done rolling-average work in touchpanel drivers to help smooth this out, but you then have to fight hysteresis.
Be aware that larger touchscreens have different resistivity properties than smaller ones, so if you just swapped panels from small to large, it's quite possible that the output range difference of the panels is not workable with the current driver settings. Again, some OEMs provide the ability to adjust these settings.
So can it work with a finger? Well there's nothing in the physical theory that would prevent using a finger. In fact if you can't use a finger, there's something wrong. Will it work in reality? Check with your OEM.

One handed coding -- tablet, special keyboard, one-handed typing?

I recently broke my finger and can now only type with my right hand. This has seriously impacted my typing speed. Since I write software for a living, this is a serious problem.
I have been doing some research, but haven't found a great solution yet. Here's what I've come up with:
Wacom tablet + hand writing recognition software. Is it possible to write code with hand writing recognition software?
one handed keyboards -- I have only found expensive (> $100) keyboards. These look like they have a steep learning curve.
one handed typing instructions: http://www.aboutonehandtyping.com/manualcompare.html. Does this really work?
What do the one handed coders out there use?
If you're a two-hand touch typist, the answer is a "mirrored" layout.
Mirroring lets you begin touch-typing with one hand almost immediately. Pretty crazy how easy it is. Based on the muscle memory you already have.
If you're typing with your right hand:
Type all right-hand keys normally.
Don't type left hand keys. Instead type the same motion (but mirrored) with your right hand.
So if you want to type:
"D" -> type "K" instead.
"W" -> type "O" instead.
"S" -> type "L" instead.
Same row of keyboard, same finger, same motion. Your muscle memory can already do this... kind of like how you're unable to pat your head and rub your belly at the same time. The wires in your brain are crossing somewhere.
Software to mirror the keyboard as described above:
Hold Spacebar to mirror:
Linux - MirrorBoard
Mac - Mirror-QWERTY
Windows - AutoHotkey version of Half-QWERTY Half-Keyboard
Predictive Text; Automatic Mirroring
Mac - One-Hand Keyboard
Windows - One-Hand Keyboard
Regarding one-handed keyboards, I've tried using a frogpad and found it ok for typing text, but unusable for coding. The symbols require several consecutive key presses and I found it impossible to use shortcuts reliably. It was too easy to hit the wrong key and get it stuck in the wrong mode.
Nobody has mentioned ENTI-key aka Coffee++ Layout yet? It is exactly designed for programming with one hand (left). And unlike qwerty, it is even optimized for speed and ergonomics. I used it some years ago for a short while and I don't know if it still works on newer systems. I think I used it for writing CSS: Typing the words with left, typing all those numbers on the numpad with right.
I can not recommend pen+handwriting. I usually use a tablet PC and handwriting code is terrible. I tried it on Windows 8 and Linux with Cellwriter, and both are not bad programs, but I still switch to onscreen keyboard whenever I can. But maybe the problem is my scratchy writing :)
I also can tell from experience that learning a new layout is not as complicated as it sounds. Especially if the layout is more logical than qwerty. I use Neo Layout since 10 years and getting the hang of it went smoothly, I was able to write a blog article after an evening of training.
"But what if you have to use qwerty on another PC?" This, also, is no problem, really. My simple trick is to never look at the keyboard when using Neo, but glimpse at it when qwerty-ing.
Good luck to anyone who wants to or has to use one hand for typing!
Now, the time to heal a broken finger will be shorter than it takes to adapt to one handed coding, not to mention the time it takes afterwards to get back to two-handed coding
Also, the time it takes to learn the methods is time you could've spend on coding (read: making a living).
Knowing this, we need a quick-fix, short term solution.
First of all, A good IDE, with code completion and similar functionality will help you a lot.
Secondly, use the shortcuts of the IDE, remember, there are Shift, Altand Ctrl keys on both sides of your keyboard.
(you might want to create a cheatsheet for those shortcuts)
In addition to helping you during your time with your injury, learning the shortcuts will also improve your coding speed when you're back up again.
Now, my comments on your proposals:
Don't, simply, Don't, it'll take even more time to fix writos (typos) beacause recognition will be flaky.
That learing curve will slow you down even more.
Won't even comment on that one...
Mirrorboard
A friend of mine broke his wrist snowboarding, and he had reasonable luck using speech recognition software (Nuance Dragon Naturally Speaking). It worked quite well for email and documentation, which would solve a part of your problem.
Another colleague, Nils Klarlund of AT&T, developed a version of emacs hooked into speech recognition. He even had a home-brewed set of foot pedals for doing shift, control, etc. He used this exclusively for years (due to bad carpal tunnel syndrome).
And maybe your feet can take up some of the burden. This is part of a parallel discussion going on in this question.
And off-topic, but extremely interesting, T.V. Raman, who's been blind since the age of 14, wrote a version of emacs that works with keyboard input and audio output. There's a chapter on it in Beautiful Code. I've seen him use it, and it's completely awesome. And of course emacs is a great interface for more than just text editing.
If you anticipate that your left hand will be out of commission for a long while, and if it's worthwhile for you to learn a new layout, then there exist one-handed Dvorak layouts.
There's some information at PC Guide: Single-Handed Dvorak Alphanumeric Layouts.
There also once was software for Qwerty Half Keyboards that used the space bar as an extra shift key that reversed the keyboard.
Good luck with your injury!
We have a developer in the office that lost mobility in his right hand and probably won't gain back full use of it. He has mainly learned to type well with his left hand and kind of fill in for his right hand. Although he lets his right hand kind of peck for things. He has gained enough speed back for it not to affect his day too greatly from what i can tell.
Only thing i can think of that might let you speed up some while typing with one hand and maybe being able to get a key or two with the other hand might be to use an IDE instead of text editor if you already don't, so you can use tab completion. Kind of a lame solution if you don't like IDEs or just don't have that option in your work environment but might help out a bit.
The same thing happened to me (I destroyed my left pinky). At the time, I didn't touch type, so my only use for my pinky was left-control, left-shift, and caps-lock.
This sounds as if it just happened to you. I promise you'll quickly learn to compensate. Remember, it's quality, not speed, that counts most.
Perhaps you should seize the opportunity and read to improve yourself as a programmer. Or spend some time debugging.

Major game components

I am in the process of developping a game, and after two months of work (not full time mind you), I have come to realise that our specs for the game are lacking a lot of details. I am not a professional game developper, this is only a hobby.
What I would like to receive help or advices for is this: What are the major components that you find in games, that have to be developped or already exists as librairies? The objective of this question is for me to be able to specify more game aspects.
Currently, we had specified pretty much only how we would work on the visual, completely forgetting everything about game logic (AI, Entities interactions, Quest logic (how do we decide whether or not a quest is completed)).
So far, I have found those points:
Physics (collision detection, actual forces, etc.)
AI (pathfinding, objectives, etc.)
Model management
Animation management
Scene management
Combat management
Inventory management
Camera (make sure not to render everything that is in the scene)
Heightmaps
Entities communication (Player with NPC, enemy, other players, etc)
Game state
Game state save system
In order to reduce the scope of this queston, I'd like it if you could specifically discuss aspects related to developping an RPG type of game. I will also point out that I am using XNA to develop this game, but I have almost no grasp of all the classes available yet (pretty much only using the Game component with some classes that are related to it such as GameTime, SpriteBatch, GraphicDeviceManager) but not much more.
You have a decent list, but you are missing storage (save load), text (text is important in RPGs : Unicode, font rendering), probably a macro system for text (something that replaces tokens like {player} with the player characters name), and most important of all content generation tools (map editor, chara editor, dialog editor) because RPGs need content (or auto generation tools if you need ). By the way have links to your work?
I do this exact stuff for a living so if you need more pointers perhaps I can help.
I don't know if this is any help, but I have been reading articles from http://www.gamasutra.com/ for many years.
I don't have a perfect set of tools from the beginning, but your list covers most of the usual trouble for RUNNING the game. But have you found out what each one of the items stands for? How much have you made already? "Inventory Management" sounds very heavy, but some games just need a simple "array" of objects. Takes an hour to program + some graphical integration (if you have your GUI Management done already).
How to start planning
When I develop games in my spare time, I usually get an idea because another game lacks this function/option. Then I start up what ever development tool I am currently using and try to see if I can make a prototype showing this idea. It's not always about fancy graphics, but most often it's more about finding out how to solve a certain problem. Green and red boxes will help you most of the way, but otherwise, use Google Images and do a quick search for prototype graphics. But remember that these images are probably copyrighted, so only use them for internal test purposes and to explain to your graphic artists what type of game/graphic you want to make.
Secondly, you'll find that you need to find/build tools to create the "maps/missions/quests" too. Today many develop their own "object script" where they can easily add new content/path to a game.
Many of the ideas we (my friends and I) have been testing started with a certain prototype of the interface, to see if its possible to generate that sort of screen output first. Then we build a quick'n'dirty map/level-editor that can supply us with test maps.
No game logic at this point, still figuring out if the game-engine in general is running.
My first game-algorithm problem
Back when I was in my teens I had a Commodore 64 and I was wondering, how do they sort 10 numbers in order for a Highscore? It took me quite a while to find a "scalable" way of doing this, but I learned a lot about programming too.
The second problem I found
How do I make a tank/cannon fire a bullet in the correct direction when I fly my helicopter around the screen?
I sat down and drew quick sketches of the actual problem, looked at the bullet lines, tried some theories of my own and found something that seemed to be working (by dividing and multiplying positions etc.) later on in school I discovered this to be more or less Pythagoras. LOL!
Years and many game attempts later
I played "Dune" and the later C&C + the new game Warcraft (v1/v2) - I remember it started to annoyed me how the lame AI worked. The path finding algorithms were frustrating for the player, I thought. They moved in direction of target position and then found a wall, but if the way was to complex, the object just stopped. Argh!
So I first sat with large amounts of paper, then I tried to draw certain scenarios where an "object" (tank/ork/soldier) would go from A to B and then suddenly there was a "structure" (building/other object) in the pathway - what then?
I learned about A-star pathfinding (after solving it first on my own in a similar way, then later reading about the reason for this working). A very "cpu heavy" way of finding a path, but I learned a lot from the process of "cracking this nut". These thoughts have helped me a lot developing other game algortimes over time.
So what I am saying is: I think you'll have to think more of:
How is the game to be played?
What does the user experience look like?
Why would the user want to come back to the game?
What requirements are needed? Broadband? 19" monitor with 1280x1024?
An RPG, yes - but will it be multi-user or single?
Do we need a fast network/server setup or do we need to develop a strong AI for the NPCs?
And much more...
I am not sure this is what you asked for, but I hope you can use it somehow?
There are hundreds of components needed to make a game, from time management to audio. You'll probably need to roll your own GUI, as native OS controls are very non-gamey. You will probably also need all kinds of tools to generate your worlds, exporters to convert models and textures into something suitable for your game etc.
I would strongly recommend that you start with one of the many free or cheap game engines that are out there. Loads of them come with the source code, so you can learn how they have been put together as you go.
When you think you are ready, you can start to replace parts of the engine you are using to better suit your needs.
I agree with Robert Gould's post , especially about tools and I'd also add
Scripting
Memory Management
Network - especially replication of game object states and match-making
oh and don't forget Localisation - particularly for text strings
Effects and effect timers (could be magical effects, could just be stuff like being stunned.)
Character professions, skills, spells (if that kind of game).
World creation tools, to make it easy for non-programmer builders.
Think about whether or not you want PvP. If so, you need to really think about how you're going to do your combat system and any limits you want on who can attack whom.
Equipment, "treasure", values of things and how you want to do the economy.
This is an older question, but IMHO now there is a better answer: use Unity (or something akin to it). It gives you 90% of what you need to make a game up front, so you can jump in and focus directly on the part you care about, which is the gameplay. When you run aground because there's something it doesn't do out of the box, you can usually find a resource in the Asset Store for free or cheap that will save you a lot of work.
I would also add that if you're not working on your game full-time, be mindful of the complexity and the time-frame of the task. If you'll try to integrate so many different frameworks into your RPG game, you can easily end up with several years worth of work; maybe it would be more advisable to start small and only develop the "core" of your game first and not bother about physics, for example. You could still add it in the second version.

Learning about low-level graphics programming

I'm interesting in learning about the different layers of abstraction available for making graphical applications.
I see a lot of terms thrown around: At the highest level of abstraction, I hear about things like C#, .NET, pyglet and pygame. Further down, I hear about DirectX and OpenGL. Then there's DirectDraw, SDL, the Win32 API, and still other multi-platform libraries like WxWidgets.
How can I get a good sense of where one of these layers ends and where the next one begins? What is the "lowest possible level" way of creating a window in Windows, in C? What about C++? (A code sample would be divine.) What about in X11? Are the Windows implementations of OpenGL and DirectX built on top of the Win32 API? Where can I begin to learn about these things?
There's another question on SO where Programming Windows is suggested. What about for Linux? Is there an equivalent such book?
I'm aware that this is very low-level, and that there are many friendlier tools available, but I would like to at least learn the basics of what's going on beneath the surface. As much as I'd like to begin slinging windows and vectors right off the bat, starting with something like pygame is too high-level for me; I really need to make the full conceptual circuit of how you draw stuff on a computer.
I will certainly appreciate suggestions for books and resources, but I think it would be stupendously cool if the answers to this question filled up with lots of different ways to get to "Hello world" with different approaches to graphics programming. C? C++? Using OpenGL? Using DirectX? On Windows XP? On Ubuntu? Maybe I ask for too much.
The lowest level would be the graphics card's video RAM. When the computer first starts, the graphics card is typically set to the 80x25 character legacy mode.
You can write text with a BIOS provided interrupt at this point. You can also change the foreground and background color from a palette of 16 distinctive colors. You can use access ports/registers to change the display mode. At this point you could say, load a different font into the display memory and still use the 80x25 mode (OS installations usually do this) or you can go ahead and enable VGA/SVGA. It's quite complicated, that's what drivers are for.
Once the card's in the 'higher' mode you'd change what's on screen by accessing the memory mapped to the video card. It's stored horizontally pixel by pixel with some 'dirty regions' of pixels that aren't mapped to screen at the end of each line which you have to compensate for. But yeah, you could copy the pixels of an image in memory directly to the screen.
For things like DirectX, OpenGL. rather than write directly to the screen, commands are sent to the graphics card and it updates its screen automatically. Commands like "Hey you, draw this image I've loaded into the VRAM here, here and here" or "Draw these triangles with this transformation matrix..." take a fraction of the time compared to pixel by pixel . The CPU will thank you.
DirectX/OpenGL is a programmer friendly library for sending those commands to the card with all the supporting functions to help you get it done smoothly. A more direct approach would only be unproductive.
SDL is an abstraction layer so without bothering to read up on it I'd guess it would have different ways of working on each system. On one it might use semi-direct screen writing, another Direct3D, etc. Whatever's fastest as long as the code stays cross-platform..able.
The GDI/GDI+ and XWindow system. They're designed specifically to draw windows. Originally they drew using the pixel-by-pixel method (which was good enough because they'd only have to redraw when a button was pressed or a window moved, etc.) but now they use Direct3D/OpenGL for accelerated drawing (and special effects). Optimizations depend on the versions and implementations of these libraries.
So if you want the most power and speed, DirectX/openGL is the way to go. SDL is certainly useful for getting the most from a cross-platform environment and integrates with OpenGL anyway. The windowing system comes last but don't underestimate it. Especially with the stuff Microsoft's coming up with lately.
Michael Abrash's Graphics Programming 'Black Book' is a great place to start. Plus you can download it for free!
If you really want to start at the bottom then drawing a line is the most basic operation. Computer graphics is simply about filling in pixels on a grid (screen), so you need to work out which pixels to fill in to get a line that goes from (x0,y0) to (x1,y1).
Check out Bresenham's algorithm to get a feel for what is involved.
To be a good graphics and image processing programmer doesn't require this low level knowledge, but i do hate to be clueless about the insides of what i'm using. I see two ways to chase this - high-level down, or bottom-level up.
Top-down is a matter of following how the action traces from a high-level graphics operation such as to draw a circle, to the hardware. Get to know OpenGL well. Then the source to Mesa (free!) provides a peek at how OpenGL can be implemented in software. The source to Xorg would be next, first to see how the action goes from API calls through the client side to the X server. Finally you dive into a device driver that interfaces with hardware.
Bottom up: build your own graphics hardware. Think of ways it could connect to a computer - how to handle massive numbers of pixels through a few byte-size registers, how DMA would work. Write a device driver, and try designing a graphics library that might be useful for app programmers.
The bottom-up way is how i learned, years ago when it was a possibility with the slow 8-bit microprocessors. The direct experience with circuitry and hardware-software interfacing gave me a good appreciation of the difficult design decisions - e.g. to paint rectangles using clever hardware, in the device driver, or higher level. None of this is of practical everyday value, but provided a foundation of knowledge to understand newer technology.
see Open GPU Documentation section:
http://developer.amd.com/documentation/guides/Pages/default.aspx
HTH
On MSWindows it is easy: you use what the API provides, whether it is the standard windows programming API or the DirectX-family API's: that's what you use, and they are well documented.
In an X windows environment you use whatever X11-libraries that are provided. If you want to understand the principles behind windowing on X, I suggest that you do this, nevermind that many others tell you not to, it will really help you to understand graphics and windowing under X. You can read the documentation on X-programming (google for it). (After this exercise you would appreciate the higher level libraries!)
Apart from the above, at the absolutely lowest level (excluding chip-level) that you can go is to call the interrupts that switch to the various graphics modes available - there are several - and then write to the screen buffers, but for this you would have to use assembler, anything else would be too slow. Going this way will not be portable at all.
Another post mentions Abrash's Black Book - an excellent resource.
Edit: As for books on programming Linux: it is a community thing, there are many howto's around; also find a forum, join it, and as long as you act civilized you will get all the help you can ever need.
Right off the bat, I'd say "you're asking too much." From what little experience I've had, I would recommend reading some tutorials or getting a book on either directX or OpenGL to start out. To go any lower than that would be pretty complex. Most of the books I've seen in OGL or DX have pretty good introductions that explain what the functions/classes do.
Once you get the hang of one of these, you could always dig in to the libraries to see what exactly they're doing to go lower.
Or, if you really, absolutely MUST learn the LOWEST level... read the book in the above post.
libX11 is the lowest level library for X11. I believe the opengl/directx talk to the driver/hardware directly (or emulate unsupported ops), so they would be the lowest level library.
If you want to start with very low level programming, look for x86 assembly code for VGA and fire up a copy of dosbox or similar.
Vulkan api is an api which gives you very low level access to most if not all features of the gpu, computational and graphical, it works on amd and Nvidia gpus (not all)
you can also use CUDA, but it only works on Nvidia gpus and has access to computational features only, no video output.

Resources