Software uses selfies to detect early symptoms of Parkinson’s disease

Machine learning lets Rochester researchers accurately identify signs of the neurological disease by analyzing facial muscles.

Every day, millions of people take selfies with their smartphones or webcams to share online. And they almost invariably smile when they do so.

To Ehsan Hoque and his collaborators at the University of Rochester, those pictures are worth far more than the proverbial “thousand words.” Computer vision software—based on algorithms that the computer scientist and his lab have developed—can analyze the brief videos, including the short clips created while taking selfies, detecting subtle movements of facial muscles that are invisible to the naked eye.

The software can then predict with remarkable accuracy whether a person who takes a selfie is likely to develop Parkinson’s disease—as reliably as expensive, wearable digital biomarkers that monitor motor symptoms. The researchers’ technology is described in Nature Digital Medicine.*

“Parkinson’s is the fastest growing neurological disorder,” says Hoque, an associate professor of computer science. “What if, with people’s permission, we could analyze those selfies and give them a referral in case they are showing early signs?”

Though ethical and technological considerations still need to be addressed, the Gordon and Betty Moore Foundation has agreed to fund this novel research through a $500,000 grant, effective November of 2021.

“The foundation wants us to validate the feedback that we would give people if they did, indeed, show early signs of Parkinson’s—especially if they are performing the test at home,” Hoque says. “The challenge is not only validating the accuracy of our algorithms but also translating the raw machine-generated output in a language that is humane, assuring, understandable, and empowering to the patients.”

Software analyzes facial expressions, hand movements

Smiles are not the only behaviors that Hoque and his lab can analyze for early symptoms of Parkinson’s disease or related disorders.

In collaboration with Ray Dorsey—a leading expert in Parkinson’s disease and the David M. Levy Professor of Neurology at Rochester—and the University’s Morris K. Udall Parkinson Disease Research Center, the researchers have developed a five-pronged test that neurologists could administer to patients sitting in front of their computer webcams hundreds of miles away.

This could be transformative for patients who are quarantined, immobile, or living in underdeveloped areas where access to a neurologist is limited, Hoque says.

In addition to making the biggest smile, and alternating it with a neutral expression three times, patients taking the test are also asked to:

• Read aloud a complex written sentence
• Touch their index finger to their thumb 10 times as quickly as possible
• Make the most disgusted look possible, alternating with a neutral expression, three times
• Raise their eyebrows as high as possible, then lower them as far as they can, three times slowly

Using machine learning algorithms, the computer program shows—within minutes—a percentage likelihood from each of the tests whether the patient is showing symptoms of Parkinson’s disease or related disorders.

What exactly does the program look for? When patients are making a smile, the software can detect whether they show less control over their facial muscles while doing so, a symptom of Parkinson’s that clinicians refer to as “modularity.”

“One thing about Parkinson’s is that you don’t show all the symptoms all the time, and not every symptom is shown in every part of your body,” says Rafayet Ali ’20, lead author of the paper. “For example, you may not have hand tremors, but you may show a significant level of deviation in your smile.”

Hence the importance of testing other expressions and movements, according to Ali, a former postdoctoral associate in Hoque’s lab who now is an associate data scientist at Sysco.

From pen-and-paper evaluations to ‘objective, digital assessments’

Both Hoque and Ali have personal stakes in helping people with Parkinson’s. Their mothers both have suffered from the disorder. Hoque’s late mother in Bangladesh, for example, was put on leovodopa, a leading medication for the disorder, after finally finding one of the country’s few neurologists. The tremors went away. “We were so happy,” Hoque says. Unfortunately, it was difficult to make follow-up appointments, and the tremors eventually returned.

That prompted Hoque to email Dorsey, “just to casually chat.” When they finally met in 2016, Hoque recalls, Dorsey “took a big document and just threw it on the table.” The pamphlet contained forms physicians need to fill out as part of the Movement Disorders Society–Unified Parkinson’s Disease Rating Scale (MDS-UPDRS).

“That’s the gold standard for measuring Parkinson’s,” Dorsey told him. “Everything we do is pen and paper. Any automation, any data analytics that you can bring into this would be a contribution. And he immediately helped us see how we could do that,” Hoque says.

“Objective, digital assessments of Parkinson’s disease can help us diagnose people with the condition and evaluate new therapies for the condition faster,” says Dorsey, an author of Ending Parkinson’s Disease (2020).

Progress toward FDA approval

It will be a while yet, however, before Hoque and his researchers can start seeking permission to analyze people’s selfies, or even before neurologists can deploy the five-pronged test that the researchers have developed.

“An algorithm will never be 100 percent accurate,” Hoque says. “What if it makes a mistake? We want to be very careful and follow guidance from the FDA if we want anybody from any part of the world to try this and get an assessment.”

Moreover, there is a whole family of movement disorders that are closely related to Parkinson’s disease, including ataxia, Huntington’s disease, progressive supranuclear palsy, and multiple dystrophy.

“They all share similar symptoms of tremor, but the tremors are very different in nature,” Hoque says. “However, even expert neurologists find it very, very difficult to distinguish among them.”

The researchers have made great progress in detecting Parkinson’s disease by automatically analyzing expressions, voice and motor movements. Yet further work is needed to develop algorithms to differentiate how these involuntary tremors differ across other movement disorders, including Ataxia and Huntington’s.

“We can’t tell that just yet,” Hoque says. “But we are in a pursuit of differentiating those tremors using AI to prevent the potential harm of misdiagnosis while maximizing benefit.”

And speaking of timely Parkinson’s disease diagnosis . . .

Turns out it’s not just selfies and videos that can help with diagnosing Parkinson’s disease.

More and more, people are using speech-activated smart devices, such as Alexa, Apple Watch, and Google Voice Assistant, to accomplish everyday tasks. Could these devices analyze our speech and voices to alert us if we show early warning signs of Parkinson’s disease?

Recent work by Rochester researchers suggests it’s entirely possible. Wasifur Rahman, Sangwu Lee, Md. Saiful Islam, and other students in Hoque’s lab published findings in the Journal of Medical Internet Research that show how an online tool can be used to help screen almost anyone anywhere for Parkinson’s disease remotely using video- or audio-enabled speech tasks.

Taken together, the Rochester researchers’ efforts are contributing to a future in which equity and access to neurological care is as ubiquitous as owning a smart phone or other internet-enabled device.

*Editor’s note: As of February 2023, the authors have retracted the article published in npj Digital Medicine that established the relationship between smiles and the early onset of Parkinson’s disease because the classification described in the paper was performed on an inaccurate use of a data pre-processing tool. Please see the authors’ retraction note for more information.

Read more