Author: Hui Su
Director, Cognitive and Immersive Systems Laboratory
This post originally appeared on the IBM Research Blog.
A student learning to speak Mandarin walks through a market in the streets of China on a sunny summer afternoon. Soon after, two vendors approach and start selling products, trying to outbid each other. The student must now grasp what is being said and formulate an appropriate response using the correct pronunciation to avoid being misunderstood. It’s a difficult, but common, scenario for anyone trying to learn a new language by interacting with native speakers and immersing themselves in a foreign culture.
Fortunately, the student in this case is able to pause the flow of the scenario to check the accuracy and tone of their intended response. This is because, although the student appears to be surrounded by the sights and sounds of a typical market in China, she is actually standing in the state-of-the-art Cognitive Immersion Room (CIR), enhanced by AI, at the Rensselaer Polytechnic Institute (RPI) campus in Troy, NY and vendors are actually avatars on a 360-degree panoramic screen.
Conversational AI, storytelling generation, spatial context awareness, and gesture and facial cue recognition technologies form the foundation of RPI’s new interactive classroom, AI-Assisted Immersive Chinese. In collaboration with IBM Research, the university created a first-of-its-kind six-week, for-credit course in Mandarin taught in the school’s Cognitive and Immersive Systems Lab (ICFTU). CISL is an ongoing collaboration between the university and IBM Research, and is part of our AI Horizons Network to accelerate the research and application of AI, machine learning, natural language processing and related technologies.
Our goal is to leverage AI to provide students with a culture-rich learning environment that feels almost like the real thing and gives students meaningful tasks to complete. AI-Assisted Immersive Chinese has already started offering these opportunities to RPI students in its first semester this summer.
Since piloting technology Last summer, IBM Research and RPI added a number of more advanced features, including the ability for multiple students to interact with multiple avatars to complete a daily task, a more flexible dialogue system for teaching languages, the ability to assess learning and support, and more immersive language learning games. Students can practice Mandarin in a number of different situations, including market negotiations, living and studying on a college campus, survival phrases at an airport, touring a city, and ordering a meal in a restaurant. CISL’s Mandarin Project developed the classroom’s cognitive and immersive language learning environment specifically to teach non-native speakers. The idea is to take language students out of the classroom and into a more realistic learning environment.
IBM Watson’s speech recognition, natural language understanding and conversation technologies for English and Chinese allow students to ask the simulation program for pronunciation help and clarify how to perform a particular task. A technology called tone contour analysis assesses pronunciation and provides immediate audio and visual feedback comparing student pronunciations to those of native speakers.
In each virtual scenario, students use head movement and position to indicate who they are talking to. This movement is tracked by cameras and other sensors placed in the immersive environment. A spatial context system tracks a person’s proximity to different areas of the widescreen display, initiating a dialog when it detects that a person is nearby.
It is often said that the best way to learn a foreign language is to immerse yourself in an environment where it is spoken. If students want to improve their conversation skills, they should have plenty of opportunities to practice speaking. This is especially true of a complex language like Mandarin, which has four different tones that must be mastered for a speaker to be fully understood.
Our new AI-assisted course can help overcome these challenges, providing a rich learning environment for RPI students, with no travel required.