Bertrand Schneider


The EarExplorer Project

One of my goals is to explore the affordances of new technologies for learning STEM disciplines (Science, Technology, Engineering and Mathematics). In collaboration with others, we are building a series of tangible interfaces for teaching science concepts. In this project, we started with the constraint of teaching a highly spatial domain where one could take advantage of the “3Dness” of physical objects. Two of the creators had been trained in neuroscience and suggested the brain as an ideal domain for designing educational TUIs. We interviewed novices after they had read a text explaining how the hearing system works and asked them various questions about how information is transformed at each step of the process. We found that novices had several misconceptions about the hearing system: first, they had trouble visualizing the different transductions happening in the ear (i.e. sound waves vibrate the ear drum with various sound pressures; the ear drum then moves the maleus to pass information as a mechanical movement; the ear bones then move the liquid contained in the cochlea and activate particular segments of the basilar membrane rolled in the cochlea). Secondly, novices struggled with the spatial mapping of different sound frequencies on the basilar membrane. High frequencies carry more energy and vibrate thicker segments of the membrane, while low frequency sounds traverse the membrane until it finds a segment thin enough to be activated. This mapping is counter-intuitive for novices, because we usually represent sounds on a number line from low (left) to high (right) frequency. On the basilar membrane, this mapping is reversed.

The design of our first prototype focused on those two aspects: the propagation and transduction of sounds through the hearing system, and the spatial mapping of sound frequencies on the basilar membrane.



Design of the System

This section describes the first prototype of EarExplorer. The system is shown below. Each tangible was created using a low-cost 3D printer. A projector displays an augmented reality layer by reflecting its image on a mirror held above the tabletop. A camera is attached to the mirror and detects the location of the fiducials on the tangibles.

When starting the activity, users are presented with three elements on the table: the outer ear, which is the starting point of the activity (top left corner, Fig. 2; the auditory cortex, which is the end point of the activity (bottom right corner, Fig. 2); and an information box (bottom left corner, Fig. 2). Eight tangibles are arranged around the projected area. Students are asked to connect the tangibles between the starting point and the ending point to let sound waves reach the auditory cortex. The 8 tangibles (in bold below) serve the following functions:

1) The speaker generates sound waves at four different frequencies (low, medium, high, very high). Those four frequencies are displayed on top of the speaker with a specific color coding (from low to high frequency: blue, green, yellow, red). By flipping the speaker, users can generate a series of sound waves to test their system.

2) The ear canal then needs to be linked to the starting point of the activity (the outer ear) and carries sound waves to the eardrum. There are two feedback showing that the tangibles are successfully connected: first, students see the sound waves follow the ear canal; second, they also see the eardrum move back and forth in the augmented reality view as the sound waves reach the end of the ear canal.

3) The ear bones need to be connected to the ear canal. As the eardrum vibrates back and forth, the ear bones will provide a similar feedback: the augmented reality view will project the shape of the ear bones on the tangible and animate them back and forth as the sound waves are reaching this part of the auditory circuit.


Figure 1: The EarExplorer Interface, after the users have connected all the tangibles in the correct sequence. They use the infobox (1) to learn about the different organs and connect them together; they then generate sounds at different frequencies with a speaker (2); sound waves travel from the emitter through the ear canal to the ear bones (3); finally, the sound reaches the basilar membrane inside the cochlea, activates a specific neuron and replays the sound if the configuration is correct (4). 

4) The snail-shaped part of the cochlea contains the basilar membrane, which react to different sounds frequencies: the base is thicker and reacts to high-energy (high frequency) sounds; the apex (i.e. tail) is thinner and react to low-energy (low frequency) sounds. When students connect the cochlea to the ear bones, they see the basilar membrane being unrolled bellow the tangible. Since understanding this step is such a crucial moment, we display a small video of a teacher reiterating that the membrane is unrolled to facilitate their task; he also instructs the students to rebuild the basilar membrane by ordering the four tangible neurons.

5-8) In this step, four neurons need to be correctly sequenced bellow the cochlea to rebuild the basilar membrane. Each neuron is associated with a particular thickness of the membrane, and a particular sound frequency. Each neuron is color-coded according to the coding scheme displayed on top of the speaker (from low to high frequencies: blue, green, yellow, red). We simplified the behavior of the system to provide an intuitive feedback when a sound wave reaches the basilar membrane: if the order is correct, users will see the part of the membrane associated with this neuron vibrate, followed by electrical potentials travelling through the neuron (Fig.2, bottom right corner, blue neuron), and the audio sound being replayed.

Each tangible can be positioned in the information box at any time of the activity to display additional information about each organ. Users can use those hints to infer the correct sequence of tangibles and learn more facts about the function of each organ.


Generalizable Design Principles

During the design process and user testing of EarExplorer, we learned important lessons that can be useful to other designers. The most critical issue was to deal with the tension between giving students enough freedom to explore a domain, and constraining the activity in a way that makes it doable for most students. We describe 5 preliminary principles that guided our design process:

1) Clear Goal: from our experience, students become frustrated with an open-ended environment if they don’t have clear directions to follow. In our system, students know exactly what they need to do (i.e. reestablish an auditory connection between the ear and the brain) and which tools they can use (i.e. tangibles). Note that this is different from rigorously scripting the entire experience.

2) Direct Feedback: each step needs to be scaffolded in such a way that students can verify the validity of their system. In our system, a feedback mechanism is built in EarExplorer: students can check that sound waves are propagated to the next organ before they move to the next step. In the early iterations of the system, we observed that without clear feedback, students tend to lose time, energy, and engagement, and are less likely to complete the task.

3) Production and agency: conceptual learning happens when students are not only consumers, but also active producers of an artifact; we leveraged this fact by having them recreate the hearing system, in a tangible way. At the end of our task, they could look at the tangible system they built and consider this artifact as their own creation.

4) “Just in time” resources: people, in general, are quite poor at dealing with lot of information out of context, that then needs to be applied all at once. Discovery learning systems need to provide affordances for learning contextual content just in time. In EarExplorer, students can easily use the “infobox” to clarify points of misunderstanding or help them discover the next step of the problem.

5) Allow for “Productive Failure”: EarExplorer does not value only the right solution, but allows students to fail in productive ways. We adopt a constructionist approach by encouraging students to explore the problem space by “fixing” this micro-world, rather than merely “ingesting” information from a textbook or lecture.


User Evaluation

We conducted a user study around this interface to explore its potential in a discovery-learning activity. The results are currently under review and will be added to this page when published.

Leave a Reply