Research News

A new publication from the neurosciences laboratory of Murat Yildirim, PhD, introduces a high-precision, deep learning framework for neurological testing. The framework uses artificial intelligence (AI) to analyze videos and connect facial movements with predicted brain activity.

Outlined in a new article in Cell Reports Methods and available for any researcher to download and use, the framework developed by the Yildirim Lab significantly improves upon existing programs. The team’s work also answers a needed call in the field for better tools that can efficiently extract and process big datasets.

Building on a known connection

Scientists already know that there are links between subtle facial movements and brain activity. The brain controls facial movements by sending signals through the facial nerves. Because of that link, even the most subtle facial motions can reflect what’s happening in the nervous system. For example:

• Eye movements can reflect attention or alertness
• Pupil size (measured through exams called pupillometry) can change with stress and cognitive effort
• Facial tension or expressions can correlate with emotion or discomfort

Clinically, researchers want to know whether these facial movements could someday help diagnose neurological disease, psychiatric conditions, fatigue or cognitive decline, or even treatment response. To do so, they need a tool that can process and analyze a lot of data.

Although some AI frameworks exist that allow researchers to upload videos, track facial features and predict brain activity, these systems are widely considered to be slow and inconsistent, and require manual adjustment. Even small sets of video recordings would take up to 15 hours to process.

‘I know there’s a better way’

First author Kemal Ozdemirli, an undergraduate researcher in Dr. Yildirim’s lab studying mechanical engineering at Case Western Reserve University, was confident that the programs could be improved. He and his team members—with contributions from some of his classmates and other researchers at Cleveland Clinic, including Bruce Trapp, PhD; Ignacio Mata, PhD; and Justin Lathia, PhD—spent two years constructing a better framework.

Building on the team’s existing pupil-tracking work, their AI-powered predictive model more efficiently processes large datasets. Since deep learning (which is a subfield of AI that is also based in machine learning) requires more data points to improve accuracy for brain activity prediction, the approach is ideal for neuroscience researchers who have expansive video libraries.

After a researcher uploads raw videos:

• Those videos are processed, frame by frame.
• Facial features are automatically detected and tracked over time (a significant improvement over previous programs, which required features to be manually labeled).
• The model optimizes using a small amount of labeled data and applies itself across different videos and conditions.
• Movements are converted into numerical data.
• Data is used to predict brain activity.

From the numerical predictions, researchers can then map the data onto between 20 and 30 brain regions—much more than what older frameworks allowed.

“We created a framework that can handle hundreds to thousands of videos and reduced processing time to two and a half hours,” Ozdemirli says. “In addition, we trained it with multiple datasets, which means it can be used to analyze data across a variety of conditions and experimental contexts.”

An open invitation to download and apply

“We’re eager to see how other researchers in the neurosciences community might use this framework to retrain customized AI models with their own datasets,” Dr. Yildirim says. The team has assembled resources available to any researcher, including tutorial videos, web apps and code to download and install.

Early collaborators using the framework and sharing data collected include John Carroll University and Akron Children’s Hospital.

“This publication represents an important milestone for the Yildirim Lab. It’s our first contribution to the growing field of cancer neuroscience, which investigates how cancers such as glioblastoma interact with neural circuits and behavior,” Dr. Yildirim says. “Our team hopes these AI-driven behavioral approaches will eventually help identify noninvasive biomarkers of tumor progression and therapeutic response.”

To collaborate with the Yildirim Lab and download the framework, please contact the principal investigator.