Multimodal Interactive Systems Management

This book provides a synthesis of the multifaceted field of interactive multimodal information management. The subjects treated include spoken language processing, image and video processing, document and handwriting analysis, identity information and interfaces. The book concludes with an overview of the highlights of the progress of the field during the past ten years, as well as the problems that are now under investigation and that offer the most promising results for the future. The book is addressed to the graduate student/postdoc level, but much of the book will be accessible to all those with a general background in information processing.

Interactive Multimodal Information Management: Shaping the Vision
Meeting capture, analysis and access
The IM2 Swiss National Center of Competence in Research
Related international projects and consortia

Human-Computer Interaction and Human Factors
Human Factors in Multimodal Information Management
Role of human factors
Prominent research topics in human factors
Methodological approach in human factors
Empirical studies
Discussion and implications
User Attention During Mobile Video Consumption
Modeling user behavior
Data acquisition experiment
Data processing and results
Conclusions
Wizard of Oz Evaluations of the Archivus Meeting Browser
The Archivus meeting browser
Multimodal Wizard of Oz evaluation
Conclusions
Document-Centric and Multimodal Meeting Assistants
The Smart Meeting Minutes application
Document centric meeting browsing
Cross-meeting and ego-centric browsing
Multimodal user interfaces prototyping for online meeting assistants
The Communication Board application
Conclusion
Semantic Meeting Browsers and Assistants
The JFerret framework and browser
TQB: a transcript-based query and browsing interface
Evaluation of meeting browsers
Automatic meeting browsers and assistants
Conclusions and perspectives
Multimedia Information Retrieval
Introduction
Multimedia information retrieval: from information to user satisfaction
Interaction log mining: from user satisfaction to improved information retrieval
Multimedia information retrieval in a wider context

Visual and Multimodal Analysis of Human Appearance and Behavior
Face Recognition for Biometrics
Introduction
Face processing in a nutshell
From face detection to face recognition
Statistical generative models for face recognition
Cross-pollination to other problems
Open data and software
Conclusion and future work
Facial Expression Analysis
Introduction and state-of-the-art
Recognizing action units
Modeling human perception of static facial expressions
Conclusion
Software for Automatic Gaze and Face/Object Tracking
Gaze tracking
Face tracking in real environments
Application to autism spectrum disorder
Conclusion
Learning to Learn New Models of Human Activities in Indoor Settings
Introduction
Related work
Proposed approach
Activity tracking for unusual event detection
Knowledge transfer for unusual event learning
Experiments
Conclusion
Nonverbal Behavior Analysis
Introduction: a brief history of nonverbal behavior research in IM2
VFOA recognition for communication analysis in meeting rooms and beyond
Social signal processing
Behavioral analysis of video blogging
Final remarks
Multimodal Biometric Person Recognition
Introduction
Biometric classification with quality measures
Modeling reliability with Bayesian networks
A-stack: biometric recognition in the score-age-quality classification space
Conclusions
Medical Image Annotation
Introduction
Multiple cues for image annotation
Exploiting the hierarchical structure of data: confidence-based opinion fusion
Facing the class imbalance problem: virtual examples
Experiments
Conclusions

Speech, Language, and Document Processing
Speech Processing
Methods for automatic speech recognition
Front-end processing of speech
Posterior-based automatic speech recognition
The Juicer decoder
Conclusions
Research Trends in Speaker Diarization
Goals and applications of speaker diarization
A state-of-the-art speaker diarization system
Research problems in speaker diarization
Conclusions and perspectives
Speaker Diarization of Large Corpora
Two-stage cross-meeting diarization
Speaker linking
Experimental results
Conclusions
Language Processing in Dialogues
Objectives of language analysis in meetings
Dialogue acts
Discourse particles
Thematic episodes and hot spots
Semantic cross-modal alignment
Conclusion and perspectives
Offline Handwriting Recognition
Introduction
Offline word recognition
From word to text recognition
From text to documents
Conclusions
Online Handwriting Analysis and Recognition
Introduction
Database acquisition
Online mode detection
Online handwriting recognition
Writer identification
Conclusion
Analysis of Printed Documents
Extracting and reorganizing digital content from printable documents
Tagging the information extracted from digital documents . .
Video recording alignment and the temporal dimension of printable documents
Aligning digital documents with audio recordings
From printable documents to cross-media alignment and indexing
Conclusion

Assessments
It was Worth it! Assessment of the Impact of IM2
Motivation and procedure
The assessment questions
Synthesis of the interviews
Conclusion
Technology Transfer: Turning Science into Products
Visual recognition on mobile devices: kooaba AG
Joining capture and webcast: Klewel SA
Business experience meets technology: KeyLemon SA
XED and Dolores for ebooks: sugarcube Information Technology Sarl
From speech to text: Koemei SA
Look me in the eye: Pomelo SARL
The Association for Interactive Multimodal Information Management
The International Create Challenge
Conclusion and Perspectives
Looking back on the initial motivations of IM2
Scientific achievements of IM2
Structural achievements
Technology transfer achievements
Perspectives