Computer aided language learning

I'm interested in what computer aided language learning (CALL) may have to offer in the following areas:

  1. making language learning fun
  2. allowing learners to practice natural language communication
  3. improving pronunciation

I think Slime Forest Adventure (lrnj.com) and the learning-through-games research by Ravi Purushotma and Dan Roy (lingualgamers.com) are nice examples of the first area, and (judging from publicity) the Tactical Language Training System (www.tacticallanguage.com) is an nice example of the second and third areas and NativeAccent ( www.carnegiespeech.com) and SRI's licensable EduSpeak library (wwww.speechatsri.com) are nice examples of the third. (I know I am leaving out a lot of good products for the sake of brevity here and I apologize.)

The rest of this page contains comments and links about these aspects of CALL, organized by topic. Be aware that I am not any kind of authority on this subject. This is a personal interest of mine and I do not work on it for ICSI.

Contents

Gameplay

Slime Forest Adventure is a role-playing game (RPG). Playing such games often involves many, many hours of repetitive combat episodes in which the user makes a series of discrete choices in order to fight simulated battles. The motivation for going through these episodes may be to advance within the game storyline or to increase the wealth and power of the user's in-game alter ego. In SFA, these episodes are language-learning drills for Japanese. The SFA web site states this design philosophy: "The ultimate target is a game people would want to play even if they have no interest in learning Japanese." The user forum for SFA contains some very enthusiastic comments such as "SFA makes learning kana [phonetic script] and basic kanji [ideograms] a breeze, and fun, to boot!" and:

Simulating conversation

www.tacticallanguage.com has a lot of information (including videos) about some very interesting software based on research at the Univ. of Southern California. There also a page about the research here and a list of research publications here.

Building language learning games out of existing games

Today's most prominent commercial games often have large production budgets, making them challenging for built-from-scratch educational software to imitate. It is possible to avoid the style of expensive titles and consider instead the style of the large number of commercial games that have been made for moderate budgets. "Casual games" and mobile games are often less expensive to build, as were console games in the past. However, another possible response to this challenge is re-use of all or part of an existing computer game.

An example of almost complete re-use, using the game Grim Fandango, has been put online by Ravi Purushotma in his master's thesis: http://www.lingualgamers.com/thesis. Ravi's work takes advantage of the fact that game developers often include translations of speech and text into multiple languages with their games in order to reach international markets. (His thesis also contains a wealth of material on other aspects of gaming and digital media in language learning.)

I find it interesting to consider how re-use could be made part of building something like Slime Forest Adventure or the Tactical Language offering.

Many game developers allow modifications ("mods") of their games to be distributed freely among owners of the game and even provide tools to assist in mod creation. Also, some game developers may be willing to license their older games for re-working at fairly low prices. I have seen games that were a few years old given away almost for free in DVD inserts in German computer and computer game magazines.

Training for tone languages

I wonder if speaking and listening ability in tone languages such as Chinese can be taught better using elements of musical training? Some references: Burnham and Brooker ICSLP 2002 paper and Alexander, Wong, and Bradlow's INTERSPEECH 2005 paper (also, there is later work also co-authored by Alexander; see her home page at Northwestern for a list). This work suggests musical training, for example pitch training approaches used for singers, might be helpful in tone language education. The Wikipedia entry for the game Karaoke Revolution says "Karaoke Revolution has been adapted in some music education classrooms as a tool to provide pitch feedback to student singers through helpful visual cues. A guide for effectively integrating Karaoke Revolution into music education can be found on http://www.cooltechschool.com". The game is fun (the Karaoke Revolution series has sold well, so has the related game Rock Band, and the similar series Singstar has sold over 10 million copies according to Wikipedia), which is good for student enthusiasm. And it gives instant individual feedback, which can be scarce in large classrooms. I guess the game only teaches level tones. But since it teaches production of many different level tones, I imagine it would help production of sloping tones as well. If I recall correctly, the Alexander et al. research strongly suggests a singing background can be helpful for learning Chinese, and (while I know nothing about singing) I suppose singer training emphasizes level tones.

Work has been done (and is ongoing last I checked) by Seneff and others at MIT on computer tone recognition for language learning which gives feedback on the quality of a student's tone production. The "Future work" section in Peabody, Seneff, and Wang's paper in InSTIL/ICALL 2004 is interesting. I think SRI has done work in this area also. I don't know if there are products on the market that do it. Hopefully they do exist. For computer-aided feedback on sloping tones, I suppose a student can also plot their own pitch as a function of time using a free speech plotting tool like WaveSurfer. I don't know how easy it is for students to judge the quality of their output. I hope that it wouldn't be hard if they have a reference to compare to.

Heterogenous sensors for speech recognition

By this I mean the use of more sensor types (beyond ordinary microphones) for speech recognition. A nose sensor could check for nasality (nasal acceloremeter?). A camera aimed at lips could check lip use (e.g., lip rounding and protrusion). Here are some comments on this by Dr. Stephen LaRocca. Here is a page on transducers in experimental phonetics from the 1984 book Experimental Phonology and Phonetics by Kate Morton. (SRI has done work on heterogeneous sensors for speech recognition in another context: speech recognition in noisy environments. There are slides about their work here.)

Articulatory feedback

Current software (NativeAccent, SRI's licensable EduSpeak, Cambridge research, Auralog's pronunciation tutor) can give pronunciation scores. It would be nice if software could give specific feedback on how to change articulator use (e.g., stick lips out more/less) or could identify phone switches (e.g., substitution of one vowel for another). This could be useful when drilling on a particular word/phrase and for identifying student weaknesses and creating targeted exercises (although perhaps listening discrimination weaknesses match production weaknesses, in which case listening tests could be used to create those). (Can Carnegie Speech's NativeAccent do things like this? Their web page suggests that it might.)

Use of TV shows or movies to create content

Here is a review of a product from EuroTalk. EuroTalk has produced a series of DVD-based language learning products with this model. From the review: " Each DVD in the series, which includes Advanced English, Spanish, German, and Italian as well as French, is based on an episode from a television program in the target language. Advanced French is based on an episode of Au coeur de la loi (At the Heart of the Law), a French television series which features a female detective and her male colleagues in Paris. " This series is now titled Movie Talk.

French newspaper Le Monde occasionally (still?) prints an English-language weekend supplement of reprinted New York Times articles, which includes a French-language sidebar which defines some English words and explains some aspects of American geography, lifestyle, etc. that relate to the articles. I haven't seen the At the Heart of the Law DVD, but if it were to be built along those lines it could include information about the French judicial system, or explanations of slang and colloquial expressions.

Some movie player software can read an external file containing subtitles, and some subtitle file formats allow powerful control over the font and placement of text. This could allow a language teacher to create on-screen notes on slang, culture, etc., for language students viewing a foreign language DVD.

(This survey of ways to learn French has interesting comments about learning from media (French in Action and Champs-Elysees). I have saved a local copy in case the link disappers.)

Links

Acknowledgements

My thanks to Ravi Purushotma, Andrew Marshall, Horacio Franco, Madelaine Plauche, Stephen LaRocca and Jen Alexander for taking the time to discuss language learning with me.