Task Prediction in cooking activities using Hierarchical state space Markov Chain and Object based Task Grouping

Publication Type:

Conference Paper


P. Lade, N.C. Krishnan, S. Panchanathan


CEA Workshop, International Symposium on Multimedia (ISM 2010) (2010)



Cooking activities are complex activities consisting of multiple steps or tasks. These tasks can be associated with one another based on two properties - the temporal structure that defines the sequence of occurrence of tasks and the objects that are used in the activity. This paper develops cooking activity models for the purpose of task prediction based on these two properties. The temporal structure of the sequence of tasks is captured by the novel hierarchical state space Markov chain (HMC) and the object usage is represented using the object based task group (OTG) models. A probabilistic task prediction algorithm that fuses the HMC and OTG models has been developed to predict the next most probable task, given that a sequence of tasks has been completed. The proposed models and algorithms have been evaluated on two complex cooking activities - making brownies and making eggs, achieving a subject independent accuracy of 68.5% for predicting tasks, which is an improvement by an average of 6% in comparison to a Markov chain. The work done in the paper is first of its kind as it focuses on task prediction rather than task recognition. The task prediction framework described in the paper can be easily adapted to any complex activity supporting various annotation schemes and activity models.


Prasanth Lade

Prasanth Lade

Ph.D. Student Researcher

Narayanan Chatapuram Krishnan

Narayanan Chatapuram Krishnan

Ph.D Student Researcher

Dr. Sethuraman "Panch" Panchanathan

Dr. Sethuraman "Panch" Panchanathan

Director, National Science Foundation


Traditional approaches to human activity recognition relying on vision as the primary sensory medium have met with little success. The emergence of the ubiquitous and pervasive paradigm of computing has ushered in new low bandwidth wearable, unobtrusive, inexpensive and…