Educators educating Educators

Jan 19

Multimodal Instruction

“We shouldn’t ask how smart you are but instead how you are smart.” ~ Howard Gardner


“All organisms develop by interacting with an environment.” ~ Piaget


“We learn by doing.” ~ John Dewey

***  Teaching Suggestions  ***

1. Start presentations multisensory, and then repeat information using one mode of the multisensory presentation separately. Re-expose students to information visually, then auditorily, then kinesthetically.

As John Medina in Brain Rules explains, "The openings moments of a lecture are cognitive hallow ground, when the most students are paying attention. There is no question that multiple cues, dished up via different senses, enhance learning. They speed up responses, increase accuracy, improve stimulation detection, and enhance coding at the moment of learning."

***  Insights ***

Taste and Smell are related. Try this!

Hold your nose and taste a jellybean.  Can you tell what flavor it is?  Try another.

Now just eat a jellybean normally.  Can you tell the difference?

Taste sensors detect only four flavors: salty, bitter, sweet, and sour.  These work with smell sensors that detect thousands of smells.  That is why food seems bland when you have a stuffy nose!


The McGurk effect

The McGurk effect is a useful introduction to how our senses interact and it illustrates the concept of sensory integration. Watch the video below before you continue.

McGurk Effect

Teaching Kids about the McGurk Effect. 7:26 minutes


The video shows a person saying “ga.” Unbeknownst to you, the scientist has turned off the sound of the original video and dubbed the sound “ba” onto it. When the scientist asks you to listen to the video with your eyes closed, you hear “ba” just fine. But if you open your eyes, your brain suddenly encounters the shape of the lips saying “ga” while your ears are still hearing “ba.” The brain has no idea what to do with this contradiction. So it makes something up. If you are like most people what you actually will hear when you open your eyes is the syllable “da.” This is the brain’s compromise between what you hear and what you see, i.e., its need to attempt integration.

McGurk Effect Chart

John Medina gives us an everyday example of the McGurk experience, using a movie theater to demonstrate this experience. When you go to a movie, the actors on the screen are not really speaking to each other at all. The actors’ voices emanate from speakers cleverly placed around the room. Even so, you believe the voices are coming from those mouths. Medina explains that “your eye observes lips moving in tandem with the words your ears are hearing, and the brain combines the experience to trick you into believing the dialogue comes from the screen. Together, these senses create the perception of someone speaking in front of you, when actually nobody is speaking in front of you.”

Several years ago researchers again demonstrated the interaction of the senses by showing that visual inputs influence auditory inputs, even with the sound turned off. People were shown a video of someone speaking but without their knowledge, the sound was completely turned off. Using fMRI, the researchers found that the area of the brain responsible for processing (auditory cortex) sound was stimulated as if the person was actually hearing the sound. Next, when the experimental group was presented with a person just simply making faces, the auditory cortex was silent. Evidently, this illustrated that “visual inputs are related to sound.”

Another experiment illustrating the phenomena called the multimodal reinforcement highlights the interplay between the senses. Researchers were shown short flashes of light near subjects’ hands, which were rigged with a tactile stimulator. At times, the stimulator was turned on when the flashes of light were occurring, sometimes not. The results clearly showed that “the visual portion of the brain always lit up the strongest when the tactile stimulator was paired with it.”


Welcome to Your Brain

Why do the senses depend on each other?

In a nerve-ending stream of sensations, external physical inputs and internal emotional inputs are incessantly bombarding your brain every waking moment. So how does your brain cope and process all of this information?

Your brain lies to you, but it doesn’t intend to explain Sandra Aamodt and Sam Wang in their book Welcome to Your Brain. Overall, it is doing a great job, working hard to help you survive. The brain tries to get quick answers rather than a perfect answer that takes time to figure out. But “your brain lies to you in your best interest but this leads to predictable mistakes, it is a reliable source of information but it’s likely to mislead you.”

Why does it lie to you? Our brain receives far more information than it can successfully process. The brain perceives many aspects of a scene but quickly forgets them. Usually these things aren’t important, so we don’t miss them. As Aamodt and Wang explain, “your brain commits lies of omission, as it discards most of the information as soon as it determines it to be unremarkable.”

The brain also decides to take shortcuts. It decides whether speed vs. accuracy in a particular situation is more important. Most of the time, the brain favors speed, interpreting events speedily but not logically. The rest of the time, it uses a slow, careful approach such as in solving math/logic problems.

David Hahnerman won the Nobel Prize in economics for demonstrating these rules and how they influence real-life behavior. Logical thinking takes a lot of effort and most of the time we take mental shortcuts unless we are cued that we should be using logic , and frequently the intuitive answer is good enough to get by.


Margaret Thatcher

Margaret Thatcher

As an example of the above, examine below the face of Margaret Thatcher. (Note. This example is also used in vision.)


The photos at the top look fairly normal to most people-except for being upside down. The bottom pictures are the same images turned right side up. Now you can see that the one on the right is really weird! Both the eyes and the mouth have been turned upside down within the face. But you probably didn’t notice that when looking at the top right picture.


Richard Mayer

To further understand the interaction between the senses and how we can capitalize upon this phenomenon in our classrooms, we turn to Richard Mayer, a cognitive psychologist from University of California, Santa Barbara, who has done more than anyone to explore the link between multimedia exposure and learning.

Mayer demonstrated the advantages of including as many senses as possible in the learning process.

One of his experiments that undoubtedly exemplify this point involves three groups of people. One group receives information delivered via one sense (hearing), another the same information from another sense (sight), and the third group the same information as a combination of the first two senses. His results consistently confirm that multisensory groups always do better than the unisensory groups. They have superior and more accurate recall and have better resolution, which lasts longer. In fact, in one experiment this is evident 20 years later.

In another study, the multisensory group generated more than 50% more creative solutions on a problem-solving test, and in another experiment, the result was 75% more creative solutions.

The benefits are not just confined to a combination of sight and sound. When touch is combined with visual information, recognition learning leaps by almost 30%, as compared with touch alone.

These experiments clearly illustrate that the positive contributions of multisensory presentations are greater than the sum of their parts. The technical term for this process is supra-additive integration.

Why is this so? It has to do with working memory, formerly short-term memory, where information is held for a short time. Research has emphatically shown that extra information given at the time of learning makes learning better. Naturally, multisensory experiences are more elaborate which helps our working memories store more information. As stated in technical terms by Medina, “this means the extra cognitive processing of information that helps the learner to integrate the new material with prior information.”


The rules of multimedia presentation

In Brain Rules, John Medina explains Myer’s rules of multimedia presentation, which Mayer developed and illustrated for the first time in Cambridge Handbook of Multimedia Learning.

Multimedia principle: Students learn better from words and pictures than from words alone.

Temporal contiguity principle: Students learn better when corresponding words and pictures are presented simultaneously rather than successively.

Spatial contiguity principle: Students learn better when corresponding words and pictures are presented near to each other rather than far from each on the page or screen.

Coherence principal: Students learn better when extraneous material is excluded rather than included.

Modality principle: Students learn better from animation and narration than from animation and on-screen text.

Richard Mayer's Book

Mayer's Graph

Multimedia Learning


What about the other senses such as smell?

Think for a moment about your favorites smells. The smell of freshly cut grass or a live Christmas tree, baking bread, the ocean, a fall bonfire, sour beer… all can conjure up an involuntary memory of a special moment, perhaps from childhood or of a once familiar place, which just for a moment is poignantly and clearly remembered.

This occurs because olfactory information has a direct connection with the amygdala, which enables animals to experience and express passion, desires and emotion, it adds the emotional spice to life, both positive and negative.

All sensory signals must go to the thalamus before being directed to other parts of the brain. This is true for all senses except one, namely smell. Odors enter the nose and proceed to the olfactory region right between the eyes, which is the size of a postage-stamp. The nerves then proceed directly to the amygdala and to the orbitofrontal cortex, the decision-making area of the brain, bypassing the thalamus.

It is as if odor is saying, “ My signal is so important, what are you going to do about it?” How much of a hurry is the sense of smell? So much so that nature decided the olfactory receptor cells didn’t need a protective barrier! The cornea guards the visual receptor neurons and the eardrum shelters the hearing neurons. But there is no protective guardian between the olfactory region and the amygdala.



Can you recall the smell of freshly baked muffins or cake?

Proust Image

The Proust Effect

The occurrence of smell evoking memories is called the Proust effect, something scientists have known about for years. Marcel Proust, the French author, presented his theory a century ago about smells and their ability to elicit long-lost memories and their unusual ability to enhance retrieval.

Neuroscientists call involuntary aroma-triggered memories the ‘Proust Effect’, in homage to the description in Remembrance of Things Past by Proust, when Marcel eats a madeleine (a small rich cake, typically baked in a shell-shaped mold and often decorated with coconut and jam). Not all of Proust’s reveries are sweetly romantic. Later on in Remembrance of Things Past, Proust talks, less poetically, of the stink of urinals on the Champs-Elysées.

The memory stirred by smelling and eating a madeleine is the significant starting point of what many people view as the greatest novel of the 20th century. It is certainly the greatest literary description of the link between smell and memory.

John Medina describes one experiment developed to demonstrate the Proust effect involves two groups of people assigned to watch a movie together. After the movie is over, the control group goes to an un-manipulated room, while the experimental group enters a room flooded with the smell of popcorn. Both groups are given a memory test, which records the number of events recalled, the accuracy of events recalled, and specific characteristics of the event. Some researchers reported that the experimental groups accurately retrieved twice as many memories as the control group and other researchers report a 10% to 20% improvement.

Congruent smells deliver the best results. For instance, the movie test would not get the same results if gasoline were used with the experimental group.

Some types of memory are sensitive to smells and other types are nearly impenetrable. Odors are helpful when students are asked to retrieve the emotional details of a memory and autobiographical memories. Conversely, odors are not good at retrieving declarative memory although you can get smell to boost declarative memory but only if the subjects are emotionally aroused, usually stressed, before the experiment begins. Recently, experiments show that smell can improve declarative memory recall during sleep.


How to hear better on a cell phone in a loud room and why it works?

Aamodt and Wang provide an interesting example of how hearing can be improved in a noisy room. It is called the “cocktail party situation” and takes advantage of the brain’s ability to separate different signals from each other, which are skills used in crowded and confusing situations.

At a cocktail party, you have to make out one voice/noise and separate the noise from others. In other words we have multiple voices and sounds coming from different directions and is called source separation. It turns out that technology can’t do this task very well; but the brain does it effortlessly.

But a telephone allows sounds from the room to mix with the signal it gets from the phone, which is hard for the brain to unmix. When you cover the mouthpiece, you stop the mixing from happening and re-create the “cocktail party situation.”

Why do telephones do that? The mixing of the caller’s own voice with the signal give more of a feeling of talking live that is called “full duplex.”


Survival insights

There is an evolutionary rationale for why multisensory situations are effective and successful in capturing our attention. In our East African home, our ancestors were encountering a multisensory world and were already champions at experiencing it. It is the same in the classroom. The more the learning environment is optimized, the more multisensory the environment becomes. But the opposite is true; learning is less effective in a unisensory environment.




“If we can control the attention of the child, we solve the problems of education.” Maria Montessori

This month Ed Tip will examine how to improve students' learning by activating their attention.