Real-Time Assistance for Persons with Dementia during Handwashing using a POMDP

Motivation

Older adults living with cognitive disabilities (such as Alzheimer's disease or other forms of dementia) have difficulty completing activities of daily living (ADLs). They forget the proper sequence of tasks that need to be completed, or they lose track of the steps that they have already completed. The current solution is to have a human caregiver assisting the patients at all times, who prompts them for tasks or reminds them of their situation. The dependence on a caregiver is difficult for the patient, and can lead to anger and helplessness, particularly for private ADLs such as using the washroom.

Here we present our real-time system for assist persons with dementia during handwashing. Assistance is given in the form of verbal and/or visual prompts, or through the enlistment of a human caregiver's help. The system uses only video inputs, and combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision theoretic framework for computing policies of action -- specifically a partially observable Markov decision process (POMDP). A key element of the system is the ability to estimate and adapt to user states, such as awareness, responsiveness and overall dementia level.

This project is part of the COACH project.

Overall System

The overall system works as follows as shown to the right. Video is grabbed by an overhead Point Grey Research Dragonfly II IEEE-1394 camera, and fed to a hand and towel tracker. The tracker reports the positions of the hands and towel to a belief monitor that tries to estimate where in the task the user is currently: what have they managed to do so far, and what is their internal mental state. The belief about where the user's state is then passed to the policy. The policy maps belief states into actions: audio-visual prompts or calls for human assistance.

Talks, Videos

You can browse the ICVS talk, which should link to videos, but if not here they are again. Each video looks as follows (still shot):

On the left, you see video taken from an independent video camera showing the whole scene. This video is not used by the system.
On the right, you see the video from the overhead camera
In the middle, you see the belief state in the planstep (PS), the awareness (AW), the responsiveness (RE) and the overall dementia level (DL).

Scenario B actor trial - person needs some assistance, but is responsive to audio prompts.

Scenario C actor trial - person needs assistance for most steps, and is only responsive to video prompts. The system learns this after an initial attempt with audio prompts, and then switches to using video only.

Papers

More details can be found by reading the following paper, presented at ICVS 2007:

Assisting Persons with Dementia during Handwashing Using a Partially Observable Markov Decision Process

Jesse Hoey, Axel von Bertoldi, Pascal Poupart, and Alex Mihailidis,
In Proceedings of the International Conference on Vision Systems ICVS 2007, Biefeld, Germany, March 2007.
Winner: Best Paper Award!
you can retrieve the paper (pdf 1.7Mb)

Details on the hand tracker can be found in the following paper, presented at BMVC 2006

Tracking using Flocks of Features, with Application to Assisted Handwashing

Jesse Hoey
In Proceedings of British Machine Vision Conference (BMVC) 2006, Edinburgh, Scotland
you can retrieve the paper gzipped ps(9.5Mb) or pdf (1.8Mb)

Still more papers on this topic can be found in my publications page, or through the IATSL web page

Currently

The system is currently being tested in clinical trials in Toronto. For collaborating researchers, results and example videos from real trials can be found by clicking on this button - you will need a password to access the videos. Email Jesse Hoey for more information. Alternatively, you can check out a video of two professional actors playing the roles of a user and a carer in this video

Future

We plan to extend this system in a number of directions

Applying it to other tasks, such as mobility (e.g. powered wheelchairs), emergency response (e.g. non-invasive community alarms), and other ADL in the washroom, such as toothbrushing or toileting. See the following paper for a general overview of these projects and how the same technology could be used.
POMDP models for Assistive Technology
Jesse Hoey, Pascal Poupart, Craig Boutilier and Alex Mihailidis,
In Proceedings of the AAAI Fall Symposium on Caring Machines: AI in Eldercare 2005
Making the system adaptive over time, in both the short and long terms. This involves offline learning from data, and online learning. More to come here.
Exporting the system as a commercial device. Partners wishing to be involved should contact Jesse Hoey (UK and Europe) or Alex Mihailidis (U.S. and Canada).