Image Sequence Analysis

The processing of image sequences has a broad spectrum of important applica­ tions including target tracking, robot navigation, bandwidth compression of TV conferencing video signals, studying the motion of biological cells using microcinematography, cloud tracking, and highway traffic monitoring. Image sequence processing involves a large amount of data. However, because of the progress in computer, LSI, and VLSI technologies, we have now reached a stage when many useful processing tasks can be done in a reasonable amount of time. As a result, research and development activities in image sequence analysis have recently been growing at a rapid pace. An IEEE Computer Society Workshop on Computer Analysis of Time-Varying Imagery was held in Philadelphia, April 5-6, 1979. A related special issue of the IEEE Transactions on Pattern Anal­ ysis and Machine Intelligence was published in November 1980. The IEEE Com­ puter magazine has also published a special issue on the subject in 1981. The purpose of this book is to survey the field of image sequence analysis and to discuss in depth a number of important selected topics. The seven chap­ ters fall into two categories. Chapters 2, 3, and 7 are comprehensive surveys on, respectively, the whole field of image sequence analysis, efficient coding of image sequences, and the processing of medical image sequences. In Chapters 1, 4, 5, and 6 the authors present mainly results of their own research on, respectively, motion estimation, noise reduction in image sequences, moving object extraction, and occlusion.

I Introduction and Survey.- 1. Image Sequence Analysis: Motion Estimation.- 1.1 Outline of Book.- 1.2 Estimation of Two-Dimensional Translation.- 1.2.1 The Fourier Method.- 1.2.2 Matching.- 1.2.3 The Method of Differentials.- 1.3 Estimation of General Two-Dimensional Motion.- 1.4 Estimation of Three-Dimensional Motion: A Two-Step Method.- 1.4.1 Estimating Image-Space Shifts.- 1.4.2 Determining Motion Parameters ― The Case of Three-Dimensional Translation.- 1.4.3 Determining Motion Parameters ― The General Three-Dimensional Case.- 1.5 Estimation of Three-Dimensional Motion: A Direct Method.- 1.6 Summary.- References.- 2. Image Sequence Analysis: What Can We Learn from Applications?.- 1. Introduction.- 1.1 Long-Range Implications of Image Sequence Analysis.- 1.2 Scope of this Contribution.- 2. Application-Oriented Review.- 2.1 Coding of Image Sequences.- 2.1.1 Coarse Attributes of Broadcast TV-Frame Sequences.- 2.1.2 Predefined Frame Segmentation.- 2.1.3 Towards Variable Spatial Segmentation.- 2.1.4 Spatial Segmentation Based on Temporal Characteristics.- 2.1.5 Reduction of Spatial Bandwidth in Moving Subimages.- 2.1.6 Interframe Coding Based on Movement Compensation.- 2.1.7 Coding of Color Video Sequences.- 2.1.8 Discussion.- 2.2 Image Sequences from Airborne and Satellite Sensors.- 2.2.1 Horizontal Wind Velocities Derived from Image Sequences in the Visual Channel.- 2.2.2 Image Sequences Including the Infrared Channel.- 2.2.3 Formation and Refinement of Meteorological and Geological Knowledge.- 2.2.4 Registration of Images and Production of Mosaics.- 2.2.5 Change Detection.- 2.2.6 Cover-Type Mapping Based on Time-Varying Imagery.- 2.2.7 Discussion.- 2.3 Medicine: Image Sequences of the Human Body.- 2.3.1 Preprocessing of Image Sequences.- 2.3.2 Blood Circulation Studies.- 2.3.3 Delineating Images of the Heart for the Study of Dynamic Shape Variations.- 2.3.4 Isolation of Organs Based on Spectral and Temporal Pixel Characteristics.- 2.3.5 Quantitative Description, Categorization, and Modeling of Organ Functions.- 2.3.6 Body Surface Potential Maps.- 2.3.7 Studying the Pupil of the Human Eye.- 2.4 Biomedical Applications.- 2.5 Behavioral Studies.- 2.6 Object Tracking in Outdoor Scenes.- 2.6.1 Traffic Monitoring.- 2.6.2 Target Tracking.- 2.7 Industrial Automation and Robotics.- 2.8 Spatial Image Sequences.- 2.8.1 No Explicit Models: Presentation of Images from Spatial Slices.- 2.8.2 Isolation, Tracking, and Representation of Linelike Features in 3-D Space.- 2.8.3 Object Surfaces Derived from Contour Measurements in a Series of Slices.- 2.8.4 Surface Detection in Samples on a 3-D Grid.- 2.8.5 Volume Growing.- 2.8.6 Deriving Descriptions Based on Volume Primitives.- 2.8.7 Estimating Parameters of Spatial Models by Statistical Evaluation of Planar Sections (Stereology).- 2.8.8 Discussion.- 3. Modeling Temporal Variations of Image Functions Caused by Moving Objects.- 3.1 Estimating the Translation for Video Images of Moving Objects.- 3.2 Including Image Plane Rotation and Scale Changes into the Displacement Characteristic.- 3.3 Discussion.- 4. Conclusions.- 5. Acknowledgements.- 6. References.- 7. Author Index.- II Image Sequence Coding, Enhancement, and Segmentation.- 3. Image Sequence Coding.- 3.1 Overvi ew.- 3.2 The Television Signal.- 3.2.1 The Digital Television Signal.- a) Scanning.- b) Spectrum of Scanned Signal.- c) Sampling.- 3.2.2 Characterization of the Sampled Video Signal.- 3.3 Some Relevant Psychovisual Properties of the Viewer.- 3.3.1 Spatiotemporal Response of the Human Visual System.- 3.3.2 Perception in Moving Areas.- 3.3.3 Temporal Masking.- 3.3.4 Exchange of Spatial, Temporal, and Amplitude Resolution.- 3.4 Predictive Coding.- 3.4.1 Philosophy of Predictive Coding.- 3.4.2 Predictor Design.- a) Linear Predictors.- b) Nonlinear Predictors.- 3.4.3 Quantization.- 3.4.4 Code Assignment.- a) Variable-Word-Length Coding.- b) Run-Length Coding.- 3.5 Movement-Compensated Prediction.- 3.5.1 General.- 3.5.2 Block-Structured Movement-Compensated Coders.- a) Displacement Estimation.- b) Results.- 3.5.3 Pel-Recursive Movement-Compensated Coders.- a) Pel-Recursive Displacement Estimation.- b) Coder Operation.- 3.5.4 Code Assignment.- 3.6 Transform Coding.- 3.6.1 General.- 3.6.2 Coding of the Transform Coefficients.- 3.6.3 Types of Transforms.- 3.6.4 Adaptive Coding of Transform Coefficients.- 3.6.5 Hybrid Transform/DPCM Coding.- 3.7 Multimode Coders.- 3.7.1 Overview.- 3.7.2 Techniques Used in Multimode Coding.- a) Subsampling.- b) Temporal Filtering.- c) Change of Thresholds.- d) Switched Quantizers.- 3.7.3 Choice and Ordering of Modes of Operation.- 3.7.4 Multimode Coder Example.- 3.8 Color Coding.- 3.8.1 The NTSC Composite Video Signal.- 3.8.2 Three-Dimensional Spectrum of the NTSC Composite Signal.- 3.8.3 Predictive Coding.- 3.9 Concluding Remarks.- Appendix A: A Digital Television Sequence Store (DVS).- A. 1 Capabi Titles.- A. 2 The System.- A.3 Software.- References.- 4. Image Sequence Erihaneement.- 4.1 Temporal Filtering.- 4.1.1 Straight Temporal Filtering.- 4.1.2 Motion-Compensated Temporal Filtering.- 4.2 Temporal Filtering with Motion Compensation by Matching.- 4.2.1 Motion Estimation by Matching.- 4.2.2 Experiment Results of Filtering.- 4.2.3 Discussions.- 4.3 Temporal Filtering with Motion Compensation by the Method of Differentials.- 4.3.1 Motion Estimation by the Method of Differentials.- 4.3.2 Various Factors Influencing Motion Estimation.- 4.3.3 Experimental Results of Filtering.- 4.3.4 Discussions.- 4.4 Summary.- References.- 5. Image Region Extraction of Moving Objects.- 5.1 Overview.- 5.1.1 Symbolic Description.- 5.1.2 Sequences.- 5.1.3 Planning.- 5.2 Vector Field.- 5.2.1 Sampli ng.- 5.2.2 Noise.- 5.2.3 Motion Effects.- 5.2.4 Plane Equation.- 5.3 Region Extraction.- 5.3.1 Node Consistency.- 5.3.2 Arc Consistency.- 5.3.3 Region Attributes.- 5.3.4 Example.- 5.4 Sequences.- 5.4.1 Similarity.- 5.4.2 Identity.- 5.4.3 Simple Sequences.- 5.4.4 Compound Sequences.- 5.5 Planning.- 5.6 Resume.- 5.6.1 Hierarchy.- 5.6.2 Outlook.- References.- 6. Analyzing Dynamic Scenes Containing Multiple Moving Objects.- 6.1 Occlusion in General.- 6.1.1 Arbitrary Images.- 6.1.2 Scene Domain Imposed Constraints.- 6.1.3 Occlusion in Image Sequences.- 6.2 Dot Pattern Analysis.- 6.2.1 Combined Motion and Correspondence Processes.- 6.2.2 Separate Correspondence Determination.- 6.2.3 Motion Analysis Given Dot Correspondence.- 6.3 Edge and Boundary Analysis.- 6.3.1 Straight Edge Domain.- 6.3.2 Curvilinear Boundary Domain.- 6.4 Conclusion.- References.- III Medical Applications.- 7. Processing of Medical Image Sequences.- 7.1 Extraction of Measurements from Image Time Sequences.- 7.1.1 Left Ventricular Shape-Versus-Time Determination.- a) Determination of Approximate Ventricular Boundaries by Motion Extraction.- b) Threshold.- c) Boundary Extraction.- 7.1.2 Determination of Precise Ventricle Boundaries Using Prediction Techniques.- a) Absolute Gradient Maximum.- b) Local Gradient Maximum ».- c) Four-Feature Majority Voting.- d) Special Condition to Ignore Outer Heart Wall.- e) Postprocessing.- 7.1.3 Results.- 7.1.4 Videodensitometry.- 7.2 Functional Images.- 7.3 Image Enhancement.- 7.3.1 Motion Deblurring.- 7.3.2 Long-Term Change Detection.- 7.4 Spatial Sequence.- 7.4.1 Electron and Light Micrograph Series.- 7.4.2 Series of Ultrasonic Data.- 7.4.3 Stacks of Computerized Tomograms.- 7.5 Frequency Series.- 7.6 Summary.- References.- Additional References.