Using psychology to help develop the next generation of hearing aids

For people who have hearing loss, listening to speech in noisy environments can be a highly demanding task. And according to Ryerson University professor of psychology Frank Russo, hearing aid manufacturers are motivated to find new approaches to the development of devices to address this challenge.

That’s why the Natural Sciences and Engineering Research Council of Canada (NSERC) has named professor Russo the NSERC/Sonova Senior Industrial Research Chair in Auditory Cognitive Neuroscience. The award provides professor Russo with funding to pioneer the development of research tools that could help the industry find a breakthrough, and to conduct a major five-year study in collaboration with Sonova, the world’s leading manufacturer of hearing aids.

The multifaceted project will be the largest-ever brain-imaging study to investigate listening effort among hearing aid users. The ultimate goal of the research is to advance our basic understanding of the neural processes involved when listening to speech in challenging environments, and to provide Sonova with a novel method for analyzing listening effort in their product development.

“We will all experience or be affected by hearing loss at some points in our lives − either we have family or friends that are affected, or it affects us personally,” said professor Russo. “There have been incredible improvements in how technology can support this but we seem to have hit a bit of a glass ceiling, and I think this is an excellent time to use the tools of psychology and neuroscience to generate new insights that will enable new engineering solutions.”

The new award is worth almost $4.4 million in funding and in-kind contributions from NSERC, Sonova and Ryerson. Uniquely placed to lead the project, professor Russo is one of only a small number of researchers around the world who have expertise in psychology, neuroscience, and hearing technology.

Ten lines of research

From his Ryerson base at the Science of Music, Auditory Research, and Technology (SMART) Lab (external link) , professor Russo will lead 10 lines of research that will each include at least one study. One large part of the project will look at why people with hearing loss find it difficult to understand emotion in speech. Another study will use a car driving simulator to examine listening effort in challenging environments. In total, approximately 400 participants will be involved.

Professor Russo will harness a relatively new neuroimaging method called functional near-infrared spectroscopy (fNIRS), which is uniquely suited to this type of research as it is silent, entirely compatible with hearing instruments, and resilient to “artifacts” – distortions in the signals recorded from the brain – arising from the hearing instrument itself as well as from head movements present during natural conversation. The fNIRS technique works by shining light into the scalp. The light is absorbed in a distinctive way by oxygenated blood, which fuels brain activity. More oxygenated blood in the prefrontal cortex suggests that more effort is required to complete the listening task.

The research team will play recordings of speech to participants while they wear fNIRS headgear. Brain activity will be monitored across the cortex, but the researchers will be particularly interested in two regions found in the prefrontal cortex: the dorsolateral prefrontal cortex and the inferior frontal gyrus. A subset of the planned studies will involve simultaneous fNIRS and EEG measurements, providing an important opportunity to validate the interpretation of fNIRS results.

To explore how people who have hearing loss understand emotion in speech, the research team will employ the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) (external link) , the largest and most popular validated database of emotional speech in the English language, which was developed in the SMART Lab.

“Our past work has shown that people with hearing impairments have difficulty understanding speech emotion, which is quite surprising to some people,” said professor Russo. “Even more remarkable is that, after restoring their hearing to ‘normal’ using hearing aids, there’s no improvement in their understanding of speech emotion, so we want to understand that a little better.”

Among the aims of the research is to identify patterns of relevant brain activity that can be observed by both the SMART Lab’s cutting-edge equipment and simpler brain-imaging devices. The use of simpler devices will allow Sonova to eventually integrate brain imaging as part of hearing aid fittings with customers.

Professor Russo’s ongoing relationship with Sonova

Sonova, a Swiss-based company with a large research headquarters in Toronto, has worked with professor Russo since 2010. Their relationship stemmed from projects that aimed to optimize hearing aids to support music perception. Several of professor Russo’s former students have also gone on to work for the company.

“Congratulations to professor Russo on being named an Industrial Research Chair,” said Steven N. Liss, vice-president, research and innovation at Ryerson. “Collaborating with leading industrial partners on cutting-edge projects is a crucial part of our aspiration to conduct research that meets societal needs. This funding means professor Russo and Sonova will be able to strengthen their unique partnership and continue the development of devices that have the potential to improve the lives of people around the world.”

Thanks to the new NSERC funding, professor Russo’s lab will gain new fNIRS equipment. Professor Russo will also be able to form a new research team to help run the five-year study, including graduate students, research staff, and postdoctoral fellows.