Exploratory Vision : The Active Eye

Advances in sensing, signal processing, and computer technology during the past half century have stimulated numerous attempts to design general-purpose ma­ chines that see. These attempts have met with at best modest success and more typically outright failure. The difficulties encountered in building working com­ puter vision systems based on state-of-the-art techniques came as a surprise. Perhaps the most frustrating aspect of the problem is that machine vision sys­ tems cannot deal with numerous visual tasks that humans perform rapidly and effortlessly. In reaction to this perceived discrepancy in performance, various researchers (notably Marr, 1982) suggested that the design of machine-vision systems should be based on principles drawn from the study of biological systems. This "neuro­ morphic" or "anthropomorphic" approach has proven fruitful: the use of pyramid (multiresolution) image representation methods in image compression is one ex­ ample of a successful application based on principles primarily derived from the study of biological vision systems. It is still the case, however, that the perfor­ of computer vision systems falls far short of that of the natural systems mance they are intended to mimic, suggesting that it is time to look even more closely at the remaining differences between artificial and biological vision systems.

I Active Human Vision.- 1 Moveo Ergo Video: Natural Retinal Image Motion and its Effect on Vision.- 1.1 Prologue.- 1.2 Introduction.- 1.3 Relation Between Eye Movement and Visual Acuity Circa 1900.- 1.4 The Marshall-Talbot Dynamic Theory of Visual Acuity.- 1.5 Empirical Tests of the Marshall-Talbot Theory.- 1.6 The Phone Rang.- 1.7 The Phone Rang Again.- 1.8 Retinal Image Slip Following Adaptation of the VOR.- 1.9 An Intramural Phone Call.- 1.10 References.- 2 Cogito Ergo Moveo: Cognitive Control of Eye Movement.- 2.1 Introduction.- 2.2 Example 1: Selection of the Target for Smooth Eye Movements.- 2.2.1 Smooth eye movements in the presence of visual backgrounds.- 2.2.2 The role of selective attention.- 2.3 Example 2: Predicting the Future Position of Targets.- 2.3.1 The effect of expectations on smooth eye movements.- 2.3.2 Past history vs. cognitive expectations of future target motion.- 2.4 Example 3: Planning Sequences of Saccades.- 2.5 Example 4: Saccades to Selected Targets in the Presence of Irrelevant Visual Backgrounds.- 2.6 Summary and Conclusions.- 2.7 References.- II Solving Visual Problems.- 3 Robust Computational Vision.- 3.1 Introduction.- 3.2 Vision Problems.- 3.3 Vision Methods.- 3.4 Robust Methods.- 3.5 Applications.- 3.5.1 Surface reconstruction.- 3.5.2 Image flow.- 3.5.3 Dynamic stereo.- 3.6 Discussion.- 3.6.1 Comparison with other paradigms.- 3.6.2 Improving performance.- 3.6.3 Computational resources.- 3.6.4 Further work.- 3.7 References.- 4 Eye Movements and the Complexity of Visual Processing.- 4.1 Introduction.- 4.2 Visual Task Performance.- 4.2.1 Detection.- 4.2.2 Masking.- 4.2.3 Localization.- 4.2.4 Multidimensional tasks.- 4.2.5 Speed-accuracy tradeoff.- 4.3 Task Complexity.- 4.3.1 Theory of complexity.- 4.3.2 Capacity of constrained parallel machines.- 4.3.3 Sequential machines.- 4.3.4 Theoretical speed-accuracy tradeoff.- 4.4 Translation Invariance.- 4.5 Conclusion.- 4.6 References.- 5 Exploratory Vision: Some Implications for Retinal Sampling and Reconstruction.- 5.1 Introduction.- 5.2 From Scene to Sensor to Code.- 5.3 Linear Reconstruction and the Sampling Theorem.- 5.4 Linear Reconstruction and Aliasing.- 5.5 Nonlinear Constraints on Possible Images.- 5.6 Irregular Sampling Arrays and Aliasing.- 5.7 Linear Reconstruction and Movement.- 5.8 Linear Reconstruction with Multiple Sampling Arrays.- 5.9 Ideal Arrays.- 5.10 Visual Representation and Transformational Constancy.- 5.11 Conclusion.- 5.12 References.- 6 Calibration of a Visual System with Receptor Drop-out.- 6.1 Introduction.- 6.1.1 Retinal degeneration and bisection judgments.- 6.1.2 Cone position calibration models.- 6.2 The Learning Algorithms.- 6.2.1 The visual system model.- 6.2.2 The delta rule.- 6.2.3 The TI rule.- 6.2.4 Inadequate sampling.- 6.2.5 A new rule.- 6.2.6 A final example.- 6.3 Discussion.- 6.3.1 Known translations.- 6.3.2 The interpolated image.- 6.3.3 Two views from two eyes.- 6.3.4 Partial damage.- 6.4 Conclusions.- 6.5 References.- 7 Peripheral Visual Field, Fixation and Direction of Heading.- 7.1 Introduction.- 7.2 Retinal Flow in a Rigid 2-D Universe.- 7.2.1 Calculating retinal flow.- 7.2.2 Level sets of retinal flow.- 7.3 Retinal Flow in a Rigid 3-D Universe.- 7.3.1 Calculating retinal flow.- 7.3.2 Points with zero flow in the 3-D universe.- 7.4 Latitudinal and Longitudinal Flow.- 7.4.1 Calculating latitudinal and longitudinal flow.- 7.4.2 Points with zero longitudinal flow in the 3-D universe.- 7.4.3 Points with zero latitudinal flow in the 3-D universe.- 7.5 A Systematic Pattern at the Periphery.- 7.6 Experiment I: Simulated Image Sequence.- 7.7 Experiment II: Servoing to a Target.- 7.8 Conclusion.- 7.9 References.- 8 Local Qualitative Shape from Active Shading.- 8.1 Introduction.- 8.2 Local Qualitative Shape.- 8.3 Diffuse Shading.- 8.3.1 A model of diffuse shading.- 8.3.2 An example.- 8.3.3 Diffuse shading in concavities.- 8.4 Point Source Shading.- 8.5 Active Shading.- 8.6 Conclusion.- 8.7 References.- III Robots that Explore.- 9 The Synthesis of Vision and Action.- 9.1 Prolegomena.- 9.2 Marr’s Theory and Its Drawbacks.- 9.3 The Architecture.- 9.3.1 The modules of the system.- 9.3.2 Outline of the approach.- 9.4 The Competences.- 9.4.1 Computational principles.- 9.4.2 Biological hierarchy.- 9.4.3 A hierarchy of models for navigational competences.- 9.4.4 Motion-based competences.- 9.4.5 A look at the motion pathway.- 9.4.6 Form-based competences.- 9.4.7 Spatial understanding.- 9.5 Conclusions.- 9.6 References.- 10 A Framework for Information Assimilation.- 10.1 Introduction.- 10.2 Information Assimilation: Formal Framework.- 10.2.1 Perceptual cycle.- 10.2.2 Sensor fusion and information assimilation.- 10.2.3 Environment Model.- 10.2.4 Input information tracks.- 10.2.5 Task modeling.- 10.2.6 Information assimilation.- 10.2.7 Knowledge caching for assimilation.- 10.3 Example Application: Autonomous Outdoor Navigation.- 10.3.1 Design.- 10.3.2 System architecture.- 10.3.3 Information assimilation module.- 10.4 Applications of Information Assimilation.- 10.5 Conclusion.- 10.6 References.- 11 Task-Oriented Vision.- 11.1 Introduction.- 11.2 Systems Description.- 11.2.1 Rock sampling system.- 11.2.2 Bin picking system.- 11.3 System Analysis.- 11.3.1 Rock-sampling system.- 11.3.2 Bin-picking system.- 11.4 Task-Oriented Approach.- 11.5 Conclusion.- 11.6 References.- IV Human and Machine: Telepresence and Virtual Reality.- 12 Active Vision and Virtual Reality.- 12.1 Introduction.- 12.1.1 Virtual reality and telepresence.- 12.1.2 Active vision.- 12.1.3 Active telepresence.- 12.2 Generating Views.- 12.2.1 Camera calibration.- 12.2.2 Digitization.- 12.2.3 Active estimation of surface depth.- 12.2.4 Registration.- 12.2.5 Warping.- 12.2.6 Integration of views.- 12.2.7 Distortions.- 12.3 Results.- 12.3.1 Simulations.- 12.3.2 Digitized Image Interpolation.- 12.4 Discussion.- 12.5 References.- 13 A Novel Environment for Situated Vision and Behavior.- 13.1 Introduction.- 13.2 The “Looking at People” Domain.- 13.3 Attention and Intention.- 13.4 Action Selection with Time-Varying Goals.- 13.5 Routines for Looking at People.- 13.5.1 Domain constraints.- 13.5.2 Figure-ground processing.- 13.5.3 Scene projection and calibration.- 13.5.4 Hand tracking.- 13.5.5 Gesture interpretation.- 13.6 An Example Implementation: ALIVE.- 13.7 Conclusion.- 13.8 References.- Author Index.