Munson Team’s “Zoom” and Creativity Add Dimension and Scope to Speech Perception Research

Professor Benjamin Munson, chair of the Department of Speech-Language-Hearing Sciences, has transformed his currently funded NIH grant to have an entire arm that studies audiovisual speech perception over Zoom. He and postdoctoral research associate Dr. Alayo Tripp took a few moments to answer some questions about this “making lemonade” out of COVID-19 lemons reality.

What were some of the early results/expectations from your NIH grant?

Our NIH grant examines how seeing someone speak changes how easily we can understand what they are saying, particularly when listening is hard (like in a noisy environment). Most previous work on this topic has used white talkers and has found that people are easier to understand when you can see them talk. However, there is a startling lack of research on audiovisual speech perception that examines the perception of people of color, and some research suggesting that the audiovisual benefit might differ when the talker and the listener don’t share race/ethnicity.  

Our project closes that gap. We had planned to spend the spring semester recording people in the lab under highly controlled conditions, with high-quality audio equipment, and a consistent video recording environment. Back in February, we assumed that by now we would be ready to use the first-ever multiracial database of audiovisual speech recordings. 

Social distancing put an end to that!  Fortunately, our university’s pivot to conducting most of its meetings on Zoom immediately gave us a tool to record people in their own home environments, and that was Zoom itself. We mustered all of our creativity—to say nothing of years of experience working with the IRB (Institutional Review Board), and the fast responses of the IRB itself—to revamp our protocol to collect data on Zoom, from participants’ homes.

How will the research be approached differently to incorporate Zoom factors?

The most obvious one is that we have reduced control over recording settings and equipment that people are using. We had to adjust our expectations of the kinds of listening experiments that we could do with our recordings given the reduced range of frequencies that we will be able to record through a teleconferencing tool like Zoom. We had to develop a protocol that would help us help our participants find the best possible recording location in their homes. We also had to develop a protocol that maximized participants’ privacy. That means making sure that we were crystal-clear in the directions we gave about where in people’s homes people should be when we made the recordings. We had to develop a new protocol for ending data recording sessions if someone should unintentionally “photobomb” the recording session.  

Those negatives aside, though, this work opens up our work for people who are themselves socially distancing. All that a person needs to participate is a smartphone, a tablet, or a computer. We have high hopes that this will allow us to recruit a more diverse set of people than might have participated if we had limited participation to people who could come to our on-campus lab. We made a recruitment video, and we hope that people will share it far and wide to folks who might be interested in participating

What does the research "look" like along a timeline?

Our data collection is proceeding quite nicely. In the week that we have been recording people, we have had ten sessions. At this pace, we will make our year-one recruitment goals by the end of June. By the end of summer, we should be able to start our listening studies and have an answer to many of our research questions. Those listening studies will look different, given the different quality of the stimuli that we have recorded. 

At the same time, we feel like our project has gained an even-greater relevance, given how frequently we all communicate via teleconferencing. We are also very grateful for the opportunity that this has given us to think flexibly and creatively about research and to stay productive during challenging times.