JOINT AMI/PASCAL/IM2/M4 Workshop on Multimodal Interaction and Related Machine Learning Algorithms

S-SEER: A Multimodal Office Activity Recognition System with Selective Perception

author: Nuria Oliver, Tel Aviv University

Description

I will present the use of layered probabilistic representations for modeling the activities of people in a system named S-SEER. I will describe how we use the representation to do sensing, learning and inference at multiple levels of temporal granularity and abstraction. The approach centers on the use of a cascade of Hidden Markov Models (HMMs) named Layered Hidden Markov Models (LHMMs) to diagnose states of a user's activity based on real-time streams of evidence from video, audio and computer (keyboard and mouse) interactions.

Categories

Top: Computer Science: Machine Learning: Markov Processes
Top: Computer Science: Computer Vision

You might be experiencing some problems with Your Video player.

*Slides*
0:03	S-Seer: A Selective Perception System for Multimodal Office Activity Recognition
0:29	Overview of the Talk
1:40	Background and Motivation
2:57	Sensing in Multimodal Systems
3:56	Seer Office Awareness System (ICMI 2002, CVIU 2004 (to appear))
4:37	Multimodal Inputs
5:33	HMMs for Behavior Recognition
6:15	Several Limitations of HMMs for Multimodal Reasoning
7:47	Seer Explored Layered HMMs (LHMMs)
8:23	Seer: Multi-Scale Activity Recognition
9:08	TITLE
11:17	SEER’s Architecture
13:00	Value of LHMMs for Seer Task
14:14	HMMs Inference in Seer
15:43	LHMMs Inference in Seer
16:59	Selective Perception Policies (ICMI’03)
17:15	Related work
17:39	Policy 1: Expected Value of Information (EVI)
18:35	Subsets of Features: Example
18:57	Criticality of Utility Model
19:58	Considering Outcome of Making an Observation
20:41	Balancing Costs and Benefits
21:48	Hypothetico-Deductive Cycle
22:31	EVI with HMMs
22:47	EVI in HMMs
22:59	Policy 2: Heuristic Rate-based Perception
23:25	Policy 3: Random Selection
23:29	S-Seer
24:00	Experiments with Selective Perception
24:07	Comparison of the Selective Perception Policies
24:55	Richer Utility and Cost Models
25:22	Context-sensitive Cost Models
26:06	Video
33:46	Summary
34:31	Future Work
35:27	Thank you

Lecture rating

People found this lecture:
Worth seeing
because it is:
Valuable and informative
Well presented
Easily understandable
Acceptably recorded
You need to login to cast your vote.

Report a problem or upload files

If you have found a problem with this lecture or would like to send us extra material, articles, exercises, etc., please use our ticket system to describe your request and upload the data.
Enter your e-mail into the 'Cc' field, and we will keep you updated with your request's status.