3D Computer Vision - Efficient Methods and Applications

Preface

Acknowledgements

Contents

Part I: Methods of 3D Computer Vision

Chapter 1: Triangulation-Based Approaches to Three-Dimensional Scene Reconstruction

1.1 The Pinhole Model

1.2 Geometric Aspects of Stereo Image Analysis

1.2.1 Euclidean Formulation of Stereo Image Analysis

1.2.2 Stereo Image Analysis in Terms of Projective Geometry

1.2.2.1 De nition of Coordinates and Camera Properties

1.2.2.2 The Essential Matrix

1.2.2.3 The Fundamental Matrix

1.2.2.4 Projective Reconstruction of the Scene

1.3 The Bundle Adjustment Approach

1.4 Geometric Calibration of Single and Multiple Cameras

1.4.1 Methods for Intrinsic Camera Calibration

1.4.2 The Direct Linear Transform (DLT) Method

1.4.3 The Camera Calibration Method by Tsai (1987)

1.4.4 The Camera Calibration Method by Zhang (1999a)

1.4.5 The Camera Calibration Toolbox by Bouguet (2007)

1.4.6 Self-calibration of Camera Systems from Multiple Views of a Static Scene

1.4.6.1 Projective Reconstruction: Determination of the Fundamental Matrix

1.4.6.2 Metric Self-calibration

The Basic Equations for Self-calibration and Methods for Their Solution

1.4.6.3 Self-calibration Based on Vanishing Points

1.4.7 Semi-automatic Calibration of Multiocular Camera Systems

1.4.7.1 The Calibration Rig

1.4.7.2 Existing Algorithms for Extracting the Calibration Rig

1.4.7.3 A Graph-Based Rig Extraction Algorithm

Outline of the Rig Finding Algorithm

De nition of the Graph

Extraction of Corner Candidates

Candidate Filter and Graph Construction

Non-bidirectional Edge Elimination

Edge Circle Filter

Edge Length Filter

Corner Enumeration

Notch Direction Detector

Rig Direction

1.4.7.4 Discussion

1.4.8 Accurate Localisation of Chequerboard Corners

1.4.8.1 Different Types of Calibration Targets and Their Localisationin Images

1.4.8.2 A Model-Based Method for Chequerboard Corner Localisation

1.4.8.3 Experimental Evaluation

1.4.8.4 Discussion

1.5 Stereo Image Analysis in Standard Geometry

1.5.1 Image Recti cation According to Standard Geometry

1.5.2 The Determination of Corresponding Points

1.5.2.1 Correlation-Based Blockmatching Stereo Vision Algorithms

1.5.2.2 Feature-Based Stereo Vision Algorithms

General Overview

A Contour-Based Stereo Vision Algorithm

1.5.2.3 Dense Stereo Vision Algorithms

1.5.2.4 Model-Based Stereo Vision Algorithms

1.5.2.5 Spacetime Stereo Vision and Scene Flow Algorithms

General Overview

Local Intensity Modelling

1.6 Resolving Stereo Matching Errors due to Repetitive Structures Using Model Information

1.6.1 Plane Model

1.6.1.1 Detection and Characterisation of Repetitive Structures

1.6.1.2 Determination of Model Parameters

1.6.2 Multiple-plane Hand-Arm Model

1.6.3 Decision Feedback

1.6.4 Experimental Evaluation

1.6.5 Discussion

Chapter 2: Three-Dimensional Pose Estimation and Segmentation Methods

2.1 Pose Estimation of Rigid Objects

2.1.1 General Overview

2.1.1.1 Pose Estimation Methods Based on Explicit Feature Matching

2.1.1.2 Appearance-Based Pose Estimation Methods

Methods Based on Monocular Image Data

Methods Based on Multiocular Image Data

2.1.2 Template-Based Pose Estimation

2.2 Pose Estimation of Non-rigid and Articulated Objects

2.2.1 General Overview

2.2.1.1 Non-rigid Objects

2.2.1.2 Articulated Objects

2.2.2 Three-Dimensional Active Contours

2.2.2.1 Active Contours

2.2.2.2 Three-Dimensional Multiple-View Active Contours

2.2.2.3 Experimental Results on Synthetic Image Data

2.2.3 Three-Dimensional Spatio-Temporal Curve Fitting

2.2.3.1 Modelling the Hand-Forearm Limb

2.2.3.2 Principles and Extensions of the CCD Algorithm

Step 1: Learning Local Probability Distributions

Step 2: Re nement of the Estimate (MAP Estimation)

2.2.3.3 The Multiocular Extension of the CCD Algorithm

Step 1: Extraction and Projection of the Three-Dimensional Model

Step 2: Learning Local Probability Distributions from all Nc Images

Step 3: Re nement of the Estimate (MAP Estimation)

2.2.3.4 The Shape Flow Algorithm

Step 1: Projection of the Spatio-Temporal Three-Dimensional Contour Model

Step 2: Learn Local Probability Distributions from all Nc Images

Step 3: Re ne the Estimate (MAP Estimation)

2.2.3.5 Veri cation and Recovery of the Pose Estimation Results

Pose Veri cation

Pose Recovery on Loss of Object

2.3 Point Cloud Segmentation Approaches

2.3.1 General Overview

2.3.1.1 The k-Means Clustering Algorithm

2.3.1.2 Agglomerative Clustering

2.3.1.3 Mean-Shift Clustering

2.3.1.4 Graph Cut and Spectral Clustering

2.3.1.5 The ICP Algorithm

2.3.1.6 Photogrammetric Approaches

2.3.2 Mean-Shift Tracking of Human Body Parts

2.3.2.1 Clustering and Object Detection

2.3.2.2 Target Model

2.3.2.3 Image-Based Mean-Shift

2.3.2.4 Point Cloud-Based Mean-Shift

2.3.3 Segmentation and Spatio-Temporal Pose Estimation

2.3.3.1 Scene Clustering and Model-Based Pose Estimation

2.3.3.2 Estimation of the Temporal Pose Derivatives

2.3.4 Object Detection and Tracking in Point Clouds

2.3.4.1 Motion-Attributed Point Cloud

2.3.4.2 Over-Segmentation for Motion-Attributed Clusters

2.3.4.3 Generation and Tracking of Object Hypotheses

Chapter 3: Intensity-Based and Polarisation-Based Approaches to Three-Dimensional Scene Reconstruction

3.1 Shape from Shadow

3.1.1 Extraction of Shadows from Image Pairs

3.1.2 Shadow-Based Surface Reconstruction from Dense Sets of Images

3.2 Shape from Shading

3.2.1 The Bidirectional Re ectance Distribution Function (BRDF)

3.2.2 Determination of Surface Gradients

3.2.2.1 Photoclinometry

3.2.2.2 Single-Image Approaches with Regularisation Constraints

3.2.3 Reconstruction of Height from Gradients

3.2.4 Surface Reconstruction Based on Partial Differential Equations

3.3 Photometric Stereo

3.3.1 Photometric Stereo: Principle and Extensions

3.3.2 Photometric Stereo Approaches Based on Ratio Images

3.3.2.1 Ratio-Based Photoclinometry of Surfaces with Non-uniform Albedo

3.3.2.2 Ratio-Based Variational Photometric Stereo Approach

3.4 Shape from Polarisation

3.4.1 Surface Orientation from Dielectric Polarisation Models

3.4.2 Determination of Polarimetric Properties of Rough Metallic Surfaces for Three-Dimensional Reconstruction Purposes

Chapter 4: Point Spread Function-Based Approaches to Three-Dimensional Scene Reconstruction

4.1 The Point Spread Function

4.2 Reconstruction of Depth from Defocus

4.2.1 Basic Principles

4.2.2 Determination of Small Depth Differences

4.2.3 Determination of Absolute Depth Across Broad Ranges

4.2.3.1 De nition of the Depth-Defocus Function

4.2.3.2 Calibration of the Depth-Defocus Function

Stationary Camera

Moving Camera

4.2.3.3 Determination of the Depth Map

Stationary Camera

Moving Camera

4.2.3.4 Estimation of the Useful Depth Range

4.3 Reconstruction of Depth from Focus

Chapter 5: Integrated Frameworks for Three-Dimensional Scene Reconstruction

5.1 Monocular Three-Dimensional Scene Reconstruction at Absolute Scale

5.1.1 Combining Motion, Structure, and Defocus

5.1.2 Online Version of the Algorithm

5.1.3 Experimental Evaluation Based on Tabletop Scenes

5.1.3.1 Evaluation of the Of ine Algorithm

Cuboid Sequence

Bottle Sequence

Lava Stone Sequence

5.1.3.2 Evaluation of the Online Algorithm

5.1.3.3 Random Errors vs. Systematic Deviations

5.1.4 Discussion

5.2 Self-consistent Combination of Shadow and Shading Features

5.2.1 Selection of a Shape from Shading Solution Based on Shadow Analysis

5.2.2 Accounting for the Detailed Shadow Structure in the Shape from Shading Formalism

5.2.3 Initialisation of the Shape from Shading Algorithm Based on Shadow Analysis

5.2.4 Experimental Evaluation Based on Synthetic Data

5.2.5 Discussion

5.3 Shape from Photopolarimetric Re ectance and Depth

5.3.1 Shape from Photopolarimetric Re ectance

5.3.1.1 Global Optimisation Scheme

5.3.1.2 Local Optimisation Scheme

5.3.2 Estimation of the Surface Albedo

5.3.3 Integration of Depth Information

5.3.3.1 Fusion of SfPR with Depth from Defocus

5.3.3.2 Integration of Accurate but Sparse Depth Information

5.3.4 Experimental Evaluation Based on Synthetic Data

5.3.5 Discussion

5.4 Stereo Image Analysis of Non-Lambertian Surfaces

5.4.1 Iterative Scheme for Disparity Estimation

5.4.2 Qualitative Behaviour of the Specular Stereo Algorithm

5.5 Combination of Shape from Shading and Active Range Scanning Data

5.6 Three-Dimensional Pose Estimation Based on Combinations of Monocular Cues

5.6.1 Photometric and Polarimetric Information

5.6.2 Edge Information

5.6.3 Defocus Information

5.6.4 Total Error Optimisation

5.6.5 Experimental Evaluation Based on a Simple Real-World Object

5.6.6 Discussion

Part II: Application Scenarios

Chapter 6: Applications to Industrial Quality Inspection

6.1 Inspection of Rigid Parts

6.1.1 Object Detection by Pose Estimation

Comparison with Other Pose Estimation Methods

6.1.2 Pose Re nement

Comparison with Other Pose Re nement Methods

6.2 Inspection of Non-rigid Parts

6.3 Inspection of Metallic Surfaces

6.3.1 Inspection Based on Integration of Shadow and Shading Features

6.3.2 Inspection of Surfaces with Non-uniform Albedo

6.3.3 Inspection Based on SfPR and SfPRD

6.3.3.1 Results Obtained with the SfPR Technique

6.3.3.2 Results Obtained with the SfPRD Technique

6.3.4 Inspection Based on Specular Stereo

6.3.4.1 Qualitative Discussion of the Three-Dimensional Reconstruction Results

6.3.4.2 Comparison to Ground Truth Data

6.3.4.3 Self-consistency Measures for Three-Dimensional Reconstruction Accuracy

6.3.4.4 Consequences of Poorly Known Re ectance Parameters

6.3.5 Inspection Based on Integration of Photometric Image Information and Active Range Scanning Data

6.3.6 Discussion

Chapter 7: Applications to Safe Human-Robot Interaction

7.1 Vision-Based Human-Robot Interaction

7.1.1 Vision-Based Safe Human-Robot Interaction

7.1.2 Pose Estimation of Articulated Objects in the Context of Human-Robot Interaction

7.1.2.1 The Role of Gestures in Human-Robot Interaction

7.1.2.2 Recognition of Gestures

7.1.2.3 Including Context Information: Pointing Gestures and Interactions with Objects

7.1.2.4 Discussion in the Context of Industrial Safety Systems

7.2 Object Detection and Tracking in Three-Dimensional Point Clouds

7.3 Detection and Spatio-Temporal Pose Estimation of Human Body Parts

7.4 Three-Dimensional Tracking of Human Body Parts

7.4.1 Image Acquisition

7.4.2 Data Set Used for Evaluation

7.4.3 Fusion of the ICP and MOCCD Poses

7.4.4 System Con gurations Regarded for Evaluation

Con guration 1: Tracking Based on the MOCCD

Con guration 2: Tracking Based on the Shape Flow Method

Con guration 3: ICP-Based Tracking

Con guration 4: Fusion of ICP and MOCCD

Con guration 5: Fusion of ICP, MOCCD, and SF

7.4.5 Evaluation Results

7.4.6 Comparison with Other Methods

7.4.7 Evaluation of the Three-Dimensional Mean-Shift Tracking Stage

7.4.8 Discussion

7.5 Recognition of Working Actions in an Industrial Environment

Chapter 8: Applications to Lunar Remote Sensing

8.1 Three-Dimensional Surface Reconstruction Methodsfor Planetary Remote Sensing

8.1.1 Topographic Mapping of the Terrestrial Planets

8.1.1.1 Active Methods

8.1.1.2 Shadow Length Measurements

8.1.1.3 Stereo and Multi-image Photogrammetry

8.1.1.4 Photoclinometry and Shape from Shading

8.1.2 Re ectance Behaviour of Planetary Regolith Surfaces

8.2 Three-Dimensional Reconstruction of Lunar Impact Craters

8.2.1 Shadow-Based Measurement of Crater Depth

8.2.2 Three-Dimensional Reconstruction of Lunar Impact Craters at High Resolution

8.2.3 Discussion

8.3 Three-Dimensional Reconstructionof Lunar Wrinkle Ridges and Faults

8.4 Three-Dimensional Reconstruction of Lunar Domes

8.4.1 General Overview of Lunar Domes

8.4.2 Observations of Lunar Domes

8.4.2.1 Spacecraft Observations of Lunar Mare Domes

8.4.2.2 Telescopic CCD Imagery

8.4.3 Image-Based Determination of Morphometric Data

8.4.3.1 Construction of DEMs

8.4.3.2 Error Estimation

8.4.3.3 Comparison to Other Height Measurements

8.4.4 Discussion

Chapter 9: Conclusion

References