Various enterprises and personal interests, such as Man-Machine Interaction (MMI), gesture studies, signs, language, social robotics, healthcare, innovation, music, publications, etc.

Category: Gaming

Avatar Kinect

Introduction to Avatar Kinect by Microsoft.

Avatar Kinect is a new social entertainment experience on Xbox LIVE bringing your avatar to life! Control your avatar’s movements and expressions with the power of Avatar Kinect. When you smile, frown, nod, and speak, your avatar will do the same.

Ah, new developments on the Kinect front, the premier platform for Vision based human action recognition if we were to judge by frequency of geeky news stories. For a while we have been seeing various gesture recognition ‘hacks’ (such as here). In a way, you could call all interaction people have with their Xbox games using a Kinect gesture recognition. After all, they communicate their intentions to the machine through their actions.

What is new about Avatar Kinect? Well, the technology appears to pay specific attention to facial movements, and possibly to specific facial gestures such as raising your eye brows, smiling, etc. The subsequent display of your facial movements on the face of your avatar is also a new kind of application for Kinect.

The Tech Behind Avatar Kinect

So, to what extent can smiles, frowns, nods and such expressions be recognized by a system like Kinect? Well, judging from the demo movies, the movements appear to have to be quite big, even exaggerated, to be handled correctly. The speakers all use exaggerated expressions, in my opinion. This limitation of the technology would certainly not be surprising because typical facial expressions consist of small (combinations of) movements. With the current state of the art in tracking and learning to recognize gestures making the right distinctions while ignoring unimportant variation is still a big challenge in any kind of gesture recognition. For facial gestures this is probably especially true given the subtlety of the movements.

A playlist with Avatar Kinect videos.

So, what is to be expected of Avatar Kinect. Well, first of all, a lot of exaggerating demonstrators, who make a point of gesturing big and smiling big. Second, the introduction of Second Life styled gesture routines for the avatar, just to spice up your avatars behaviour (compare here and here). That would be logical. I think there is already a few in the demo movies, like the guy waving the giant hand in a cheer and doing a little dance.

Will this be a winning new feature of the Kinect? I am inclined to think it will not be, but perhaps this stuff can be combined with social media features into some new hype. Who knows nowadays?

In any case it is nice to see the Kinect giving a new impulse to gesture and face recognition, simply by showcasing what can already be done and by doing it in a good way.

Ninja Strike Airtime

Ninja Strike, a killer application for gesture recognition?

This is certainly an interesting development. Previously we have seen mobile phones using motion and acceleration sensors for gesture control (see here and here). There have also been applications where the camera was used to simply capture optical flow: something in front of the camera is moving/turning in direction A therefore the phone is moving/turning in A + 180 degrees (here). In this case the gesture recognition appears to go a step further and at least the hand appears to be extracted from the image. Or does it simply assume all movement is the hand? And then perhaps the position of the motion is categorized into left-middle-right? Maybe the velocity is calculated but I don’t think so.

Update: I do like the setup of how people can hold their phone with the camera in one hand, throw with the other and check their virtual throw on the display. The virtual throwing hand on the display is more or less in the same position as your physical hand, which I think is nice.

EyeSight is a techno start-up of 2004 from the Kingdom of Heaven (Tel Aviv) aspiring to use nothing but Air and a Camera to achieve a divine interaction between true techno-believers and their mobile phones. They prophetize that their technology will ‘offer users, including those who are less technologically-adept, a natural and intuitive way to input data, play games and use their mobile phone for new applications’. Heaven on Earth. Mind you, nothing is carved in stone these days. Besides, human nature and intuition are all too often deified these days anyway. Human nature is what usually gets us into trouble (not in the least in the Middle East).

Anyway, one of their angels called Amnon came to me in the night bearing the following message:

Hello Jeroen,
First Allow me to introduce myself. I’m Amnon Shenfeld, RND projects manager for eyeSight Mobile Technologies.
I’ve been following (and enjoying) your BLOG reports for a while, and I thought that the following news from my company, eyeSight Mobile Technologies, may make for an interesting post.
eyeSight has just launched “Ninja Strike”, an innovative mobile game featuring a unique touch free user interface technology we call eyePlay™. Allow me to provide some background information about eyeSight, eyePlay and Ninja Strike: I’m sure you are aware of the popularity and attention innovative user interfaces are getting since the introduction Apple’s IPhone and Nintendo’s Wii… My company’s vision is to bring this technology into the mobile market, and our first products are focused on changing the way mobile gamers play. Our new game, “Ninja Strike”, does exactly this.
You play a ninja warrior with Ninja Stars as your primary weapon. Your stars are thrown by making a throwing motion in front of the phone’s camera. Much like training in real life, during the game you will learn how to throw your weapon correctly, and improve your aim. Your enemies, the evil Kurai ninjas, will also gain strength as the game advances…
Looking forward to hear from you, I hope to see a new post in your blog soon, you’ve been quiet for a while… J
Amnon Shenfeld

Amnon, will you heed my calls? Have you answers to my burning questions above?

game demo

Space Invaders with Gesture Recognition

The more I think about it, the more I am convinced that the (near) future of gesture recognition lies in entertainment. Here is yet another gaming application with gestures: A multi-player, wall display remake of Space Invaders. A highly advanced gesture interface seems to allow any kind of movement at a certain spot on the baseline as ‘fire from here’. A camera tracks if a hand blots out one or several of a series of lights that represent the positions on the baseline.

YouTube: Development: Douglas Edric Stanley / This is installation was developed on-site for the Gameworld exhibition at the Laboral Art Center, Gijón, Spain. March 30 – June 30, 2007. For more information visit the responsible art centre in Spain.

Wii gestures for WWii game

Medal of Honour is a game where you play a soldier in World War II. EA created a special version for the Nintendo Wii (and the Nunchuk) and provided some animations of how you gesture to play.
Tutorial on jumping and crouching
How to jump and crouch. (source)

Here are tutorials on turning around, throwing a grenade, reloading your gun, using your bajonet, and the best one of the lot: steering your parachute.

Update June 11, 2007: Here is an impression that is a bit more realistic:

Gesture and Speech Recognition RSI

Gesture and speech recognition often promise the world a better, more natural way of interacting with computers. Often speech recognition is sold as a solution for RSI stricken computer users. And, for example, prof. Duane Varana, of the Australasian CRC for Interaction Design (ACID) believes his “gesture recognition device [unlike a mouse] will accommodate natural gestures by the user without risking RSI or strain injury”.

Gesturing: A more natural interaction style? (source)

So, it is a fairly tragic side effect of these technologies that they create new risks of physical injury. Using speech recognition may give you voice strain, which some describe as a serious medical condition affecting the vocal chords and caused by improper or excessive use of the voice.

Software coders who suffer RSI and switch to speech recognition to code are mentioned as a risk group for voice strain. Using gesture recognition, or specifically the Nintendo Wii, may cause aching backs, sore shoulders and even a Wii elbow. It comes from prolonged playing of virtual tennis or bowling when gamers appear to actually use neglected muscles for exensive periods of time…

In comparison, gamers have previously been known to develop a Nintendo thumb from thumbing a controller’s buttons. I can only say: the Wii concept is working out. It is a workout for users, and it works out commercially as well. I even saw an add on Dutch national TV just the other day.

The Wii is going mainstream. As far as injuries are concerned: If you bowl or play tennis in reality for 8 hours in a row, do you think you will stay free of injury? Just warm up, play sensibly and not the whole night. Nonsense advice for gamers, I know, but do not complain afterward.

A collection of Wii injuries (some real, some imanginary): –, devoted to Wii trouble. – What injuries do Wii risk? – Bloated Black EyeBroken TVs, and a hand cut on broken lamp (YouTube, possibly faked).

For more background see also: The Boomer effect: accommodating both aging-related disabilities and computer-related injuries.

Wii Mainstreams Gesture Recognition

Play sports games virtually but with the real movements


The Nintendo Wii controller is starting to hit the big spotlights. Interaction designers, like Matt MacQueen, are noticing the power they can bring to gaming experience. He has written a nice piece reflecting on Wii experiences sofar and projecting trends for the future.

See the huge line for the Nintendo Wii demonstrations at the E3 2006 conference

Update: here is a nice Dutch review of Wii gaming experience.

Tags: , , ,

Exergames blogs about physical gaming

Exergames is a nice-looking blog on exercise and physical gaming. They do not update much though. I guess they are too busy playing with their new Nintendo Wii‘s. Or perhaps they were seriously injured from the intense physical interaction, preventing them from typing? Wanna work out with a game? (source) By the way: Wouter has a nice post on another quirk of Nintendo Wii: The controls of enthusiastic gamers are flying out of their sweaty hands and breaking their expensive TVs or windows. Every rose has its thorn.

Powered by WordPress & Theme by Anders Norén