It seems I cannot read a newspaper or magazine these days without some story about Second Life. Somehow this virtual world has crossed a few thresholds and actually arouses common mass interest. Their site says they have almost 4.3 million registered users. If these people have virtual sex every day, which appears to be one of the more attractive features, then there is more sex in a single Second Life minute than in my entire adult life (don’t do the math please).
Anyway, I felt obliged to join in, just to check out how they did the gesturing part. And because it is free of course. Second Life is quite popular in the Netherlands, like all free things. You do have to install client software first.
I had to choose a silly name and will be known as: Jeroen Garrigus.
For a couple of days I have been trying it out for an hour or two at a time. I have not really done anything useful or funny, but I danced, had a drink, and earned a few dollars camping. I found out my avatar can make many predefined gestures, such as waving and bowing. The help on gestures explains how gestures are treated. Sounds (laughing) or chat words can be added to certain movements and are then saved together as a ‘gesture’. It is all a matter of combining things from (software) libraries. I haven’t figured out facial expressions, but I think they do not really play a role.
The question then remains how an entirely new gesture movement is created? And can I do that myself? Or is that the prerogative of the Second Life creators? Well, you can buy gestures or animations or create your own animations.
Training to Ensure Positive Outcomes. … Systematic professional training is strongly encouraged to maximize use of the unique features… The end user must be completely trained … to achieve positive outcomes. Managers of the system should … provide training for both end users and speakers … Additional time may be required to customize … Contact EnableMart for information about professional training opportunities.
This seems at first glance a fair bit of warning before you spend $6499 on an iCommunicator 5.0 kit. However, EnableMart sells the advised training for an additional $125 an hour, it is not free. I think this entire thing is a bit suspicious. I have worked with speech recognition, inlcuding the Dragon NaturallySpeaking, and it makes recognition errors (period). I have also fooled around with or seen most sign synthesis technology available today, and it is far from natural. And the same is true for speech recognition.
These technologies have yet to make good on their promises. If you ignore actual user experiences you can imagine it will solve many communication problems. But in practice, little errors cause big frustrations. Using speech recognition can be very tiring and irritating. It only works if the entire interaction is designed well and the benefits outweigh the cost.
Just imagine you are a deaf person using this iCommunicator with some teacher and a simple speech recognition error occurs: How is that error handled? Usually, when a speaker dictates to Dragon NaturallySpeaking he will spot the error and correct it. In this case your teacher will not spot the error (assuming he doesn’t monitor your screen) and the dialogue will continue with the error in place (unless there is enough context for you to spot the error and understand what the speaker actually said). A second problem is that you have to persuade people to wear your microphone to enter into a conversation with you. In a weird and cynical way you are asking them to suffer the same techno-torture as you. Not something you want to do more than twice a day, I imagine. And only with people whose affection you can afford to lose. The sign synthesis is fairly straightforward sign concatenation. A dictionary of 30.000 signs is accessed to get a video for every word. The videos are then played one by one, without any further sentence prosody. That means it looks terrible, like a gun firing signs at you. It also means it does not sign ASL, but signed English at best. Good enough, you might say, but I think the benefit of artificial signed English over typed text is not big. So, the signing is pretty much worthless. Jim the tell-sell guy further claims you can use it to improve your speaking. I do not believe speech recognition technology can give the proper feedback to improve articulation difficulties. It may be able to judge whether you pronounced something correctly (or at least similar to what it knows), but that’s about it. Although there is something in the specs about pronunciation keys, the video doesn’t show details. Well, I simply do not think a computer can reliably tell you what sort of error you made. So what does that leave? You can type text and your iCommunicator reads it out loud with text-to-speech. You can get that sort of software for the price of a cheap dinner from any of these sites.
Finally, the iCommunicator v5.0 lets you search for a word on Google with a single click. That’s pretty neat I admit. If you also think that that is worth a couple of thousand dollars, please contact me. I can supply an iBrowser v6.1 for only $2999, and will supply the necessary training for free. What the hell, I’ll even throw in a professional designer microphone v7.2 🙂 Unfortunately, the business case of the iCommunicator may actually rest on sales to hearing people who wish to reduce or entirely avoid the cost of interpreters:
HighBeam Encyclopedia: …The iCommunicator also enables government workers to provide equal access to information and services to the hearing impaired in compliance with the Americans with Disabilities Act and Section 508…
Sometimes, you can only hope the future will prove you wrong.
What happened since then? Not a lot apparently, there is little in the news. One recent newsflash does mention the Kapa o Pango and still refers to the ‘ending with a cut-throat gesture’, implying that in their view the ending has not really changed. Internally, they seem to have settled on keeping the Kapa o Pango for special test matches and sticking to the story about drawing breath instead of cutting throats.
Well, I will be looking forward to how they do this gesture at the end precisely. Fortunately for them, everybody is now so sensitive to the threatening gesture, that they hardly need to make it. Everyone will expect it and be looking for it. They can now afford to actually make a watered-down version, claim innocence and still get the message across. The only trouble is whether the All Blacks can accept giving an impression of having backed down. If some want to appear unwavering, firm, or just outright frightening (which is the purpose after all) they may well go for the throat.
There is a crazy story evolving about a deaf man, by the name of Shaun Phuprate, who landed in jail for gesturing ‘fuck you’ at the police. But did he really?
Sunderland Echo: Signing row lands deaf man in court. “A DEAF man arrested after police mistook sign language for an obscene gesture has lost his court battle for compensation. Shaun Phuprate, of Town End Farm, was handcuffed and hauled before magistrates for making a two-finger salute at officers in Sunderland. The now 26-year-old insisted he was making the sign for “I am deaf” and had not been rude.” There is also a BSL video translation from SignPost
The ‘palm-back v-sign’ (source Morris, 1979; the finger)
The policemen believed they saw a ‘Palm-back v-sign’, which is a cockney variant of giving the finger. As such it constituted an insult and, together with other misconduct, enough to land Phuprate (and his brother) in jail for the night. Initially when the case came before court in 2002 the defense claimed Phuprate was merely signing ‘I am Deaf’, which the judge accepted.
Afterwards, the brothers Phuprate launched a wrongful arrest claim, but recently jurors at Newcastle County Court found it was “more probable than not” officers believed Shaun had made an obscene hand gesture (and mouthed an insult and was drunk). Therefore the arrest was not wrongful. Nice to see how the policemen’s perception of insults is what matters in court. I am quite convinced that officers are human beings like most others, and I am also quite convinced that human beings can be oversensitive to insulting gestures. But I think we may well have another case of someone trying to pretend his insulting gesture was an innocent movement. The two gestures seem too far apart for a mix-up in the first place.
Strangely enough, in the initial hearing it was also stated that “[Phuprate] cannot possibly have understood any caution that was given or the reasons for his arrest”. Just because he was deaf he cannot understand anything? And elsewhere it says that Shaun could not have sworn since he was born deaf and ‘without speech’. As if oral education of the deaf never existed?
In this case I believe the cops can probably be trusted to have made the proper decisions about what they saw or chose to see. But unless you were there, we will never know what Phuprate really did. Ps. It appears that UK courts are more sensitive to people insulting cops than Dutch courts.
I came across a bundle of websites that is like a city full of British Sign Language (BSL). And it all seems to come from the University of Bristol, Centre for Deaf Studies (CDS). They managed to create a great signing experience in their site, in just about every page you care to look at. If you are thinking about using (video’s of) sign language on your website, you simply must check these out:
Deafresource: To find out about the Deaf community sign language and Deaf studies.
Deafstation: News and information service for BSL users. Deafstation provides a daily news programme in BSL.
Signstation: Learn about British Sign Language (BSL)? Signstation has video, interactive exercises, pictures, graphics and explanations about BSL and Deaf people at work. I would have to check how they did everything, but it is clear that a team of people with good technological skills (flash, shockwave, scripting, streaming video, etc) and good web design skills is working on these sites. They did require me to download and install a more recent shockwave player, but I guess I was due for upgrading and it’s free anyway 🙂
For any community of Deaf people sites like these can be extremely valuable, is my belief. The power of the internet can be put to use to share and enjoy without interference (by the hearing cultural majority or otherwise). But it takes skill to get the websites suited to a primarily signing audience. I think the CDS did a good job here, and I wouldn’t be surprised if these sites are already playing a large role in UK Deaf society.
Please put aside any cultural integration reservations for the length of this post and while browsing through the links. I once attended a lecture from one of Bristol’s senior researchers, Paddy Ladd, at the International Conference for the Education of the Deaf in Maastricht, 2005. It was the most memorable event of the entire conference. Mr. Ladd unleashed his anger, frustration and fears on the unsuspecting audience. He was angry about the history of abuse associated to the conference in question (at the first edition in 1880 deaf education was all but limited to oral methods for almost a century). He feared that developments in cochlear implementation would cause doctors and people in deaf education to regress into methods treating deafness as a curable medical condition. And he warned against cultural genocide taking place should sign language teaching suffer, even going so far as comparing the withholding of sign language as primary language from CI kids to child abduction. It was a great speech, arousing much passion and support from Deaf attendees. Next day the British association for the education of the Deaf (or something like that) distanced itself from Paddy Ladd’s views. His speech dominated discussions afterward.
It is not my place to take part in discussions on educational methods or the impact of CI, nor am I an expert judge on the value of Deaf sign language subculture in comparison to a potentially fuller social integration of individuals. But I will say that these websites constitute a fine part of the world wide web. They are examples of how you can make video work on the web. It would be a shame if they were exterminated.
Update 8 Mar ’07: I was so impressed I forgot thenews sparking my interest: CDS launched the worlds first sign language dictionary for mobile phones at www.mobilesign.org. Again it is a very well designed service. Two important things: it is simple enough to use with a small screen, and download size of (video) files is fairly small (and thus affordable in pay-per-bit environment like the mobile internet). Unfortunately, prior scientific work on how to reduce bit rates without harming sign language understandability were not applied as far as I can tell:
* Cavender et al. (2005). MobileASL: Intelligibility of Sign Language Video as Constrained by Mobile Phone Technology (pdf); * Parish et al. (1990). Intelligent Temporal Subsampling of American Sign Language Using Event Boundaries (pdf)
* Sperling et al. (1985). Intelligible encoding of ASL image sequences at extremely low information rates. (Might have been used? ACM, DOI). ‘
Update 9 Mar ’07: Tim Tolkt is pretty good at integrating sign video’s (in flash) as well.
I put together a playlist with YouTube movies with babies showing off their signs. Or should I say, mommies showing off their babies? Or their babies’ signs?
The first two videos are posted by a user called SmartHandCA, and constitute the most convincing but at the same time most suspect material. Why is there no real user name?
I know there are companies out there trying to make money by convincing people they should teach their babies to sign. They have everyone claiming it will boost their (language) development, succes in this life, the hereafter and then some.
Now, I am not saying it is definitely the case, merely raising a bit of doubt, but the baby in question may in fact be the child of Deaf parents, or older than the stated 12 months. This is the internet after all. The rest of the babies are all older, already talking as well or just signing ‘more’ or requesting nursing. If my distrust is unfounded then I must admit it is a neat example of a small baby picking up good vocabulary skills for his age.
All in all, it is not very funny to watch, it even got on my nerves after a bit. And, apart from the magical baby from SmartHandsCA, it seems to confirm that ‘more’ and ‘milk’ are the only frequently used signs (see my prior posts on babies signing ‘more milk’, and the fascination with nipples we share with certain apes). But perhaps I am just too biased and skeptical to see the revolution taking place in front of my eyes.
My kids are getting a bit older now, with a daughter of five and a boy of three (but a next one coming up soon). They do not seem to suffer from a lack of baby signing, which I tried half-heartedly but gave up on due to low ROI. I do shout at them a lot, and even throw books if I feel their vocabulary development is getting behind. It doesn’t seem to matter. My daughter’s most treasured words are those she picks up from her friends at school. Not always music to my ears, I must say.
Here’s a bit of world news for you: Prime Minister Blair of the UK was a bit of a rascal in his youth and student days. Well, good for him, you might think, and I hope the Britons will do the same. Specifically, it turns out that he made a crude or lewd gesture in a photo of his student club.
Can you spot Blair and his obscene gesture? (source)
The funny thing is that it was kept secret until now. Some people knew, but the picture was released with the gesture removed (by retouch), or in another case only the top half was published. As to the meaning of the gesture; I assume it can mean wanker, though that gesture is usually made with a wanking motion (which, if present, was not captured in this photo of course). But whom is he calling a wanker? I think he is making the gesture in a more general joking way. He boasts his lack of regard for decorum, saying perhaps something like “what a bunch of wankers we are all together”, or “look at me being the most daring of my company of rich brats”. Anyone else care to comment?
You can read my story here or check my profile at LinkedIn
I graduated in 1998 on speech recognition in a multi-modal user interface for a Personal Digital Assistant (PDA), and continued to work at KPN on speech recognition, mainly for phone (voice) services. Back then it was a nice time to be at KPN Research, but their place, the dr. Neherlab in Leidschendam, is not in use anymore as a research facility. KPN Research was sold by its mother company KPN Telecom and became TNO Telecom in Delft. The commercial activities on speech recognition came to an almost complete stop at the time, apart from some small projects. The research activities were the core of a new startup within TNO Telecom called Dutchear.
Later, I joined the shortlived Competence Center for Speech-driven Services (CCSD). When that Center was closed I joined Value Added Services at KPN Telecom. It was during the CCSD time that we developed and patented an idea for a Personal Call Assistant. Does anyone remember Wildfire? It seems to still be around here, and here, or listen to a demo here. It was the leading PCA at the time. They had filed original patents, to which we had to refer. ELECTRONIC CALL ASSISTANTS WITH SHARED DATABASE Alex Vieira, who works at the EPO pointed out to me that the patent had become available, since for a long time I think it was just kept ‘pending’.
The World’s Patent Databases are filled with all sorts of technological advances, that may never make it to the market. So just because we have not seen certain gesture recognition applications appear in the shops, that does not mean they were not invented. See for example this nice invention by my former employer Philips. Philips invented a new dance to catch the stars? (source Wipo)
The trouble with patents is that for most people they are hard to read. The pictures are obscure and require the text to explain them. The text itself is written to conform to certain legal standards and is full of references to prior art. The title and abstract of the example above are a nice case of such patent-language.
(WO/2007/020573) INTERACTIVE ENTERTAINMENT SYSTEM AND METHOD OF OPERATION THEREOF An interactive entertainment system comprises a plurality of devices providing an ambient environment, gesture detection means for detecting a gesture of a user, and control means for receiving an output from the gesture detection means and for communicating with at least one device. The control means is arranged to derive from the output a location in the ambient environment and to change the operation of one or more devices in the determined location, according to the output of the gesture detection means.
Are you still interested in patents? Think you can get around the lawyer-talk and see the ideas behind them? If you are willing to spend a bit of your attention I think you will be well rewarded. Below I give you the most useful links to online patent searches. The first one to spot the gesture inflatable doll gets a special mentioning.
Wipo probably offers the best search capability. They let you create an RSS-feed of any search (no account necessary). But it only contains world patents (WO). That sounds a bit strange, but it means only patents that have been applied for to be valid worldwide. And many patents are just valid for the US, or just for Europe. Those patents do not show up in these results. Search: gesture, gesture recognition, gesture synthesis.
At Esp@ceNet they discovered fossil technology called cookies. So you can search all you want, then add it to ‘MyPatents list’ which is kept as a cookie on your local computer. No account, no RSS-feeds, no alerts. You are on your own (computer). Now I use several computers and simply dislike cookie-solutions.
Esp@ce does offer a choice of coverage: Wipo, European, or ‘Worldwide’. Especially the ‘worldwide’ option is nice since it is a collection of patent applications of about 80 countries. Search worldwide for: gesture (in title or abstract); gesture recognition; gesture synthesis.