Advanced Signal Processing and Digital Noise Reduction

Stochastic signal processing plays a central role in telecommunication and information processing systems, and has a wide range of applications in speech technology, audio signal processing, channel equalisation, radar signal processing, pattern analysis, data forecasting, decision making systems etc. The theory and application of signal processing is concerned with the identification, modelling, and utilisation of patterns and structures in a signal process. The observation signals are often distorted, incomplete and noisy. Hence, noise reduction and the removal of channel distortions is an important part of a signal processing system. The aim of this book is to provide a coherent and structured presentation of the theory and applications of stochastic signal processing and noise reduction methods. This book is organised in fourteen chapters. Chapter 1 begins with an introduction to signal processing, and provides a brief review of the signal processing methodologies and applications. The basic operations of sampling and quantisation are reviewed in this chapter. Chapter 2 provides an introduction to the theory and applications of stochastic signal processing. The chapter begins with an introduction to random signals, stochastic processes, probabilistic models and statistical measures. The concepts of stationary, non-stationary and ergodic processes are introduced in this chapter, and some important classes of random processes such as Gaussian, mixture Gaussian, Markov chains, and Poisson processes are considered. The effects of transformation of a signal on its distribution are considered.

1 Introduction.- 1.1 Signals and Information.- 1.2 Signal Processing Methods.- 1.2.1 Non-parametric Signal Processing.- 1.2.2 Model-based Signal Processing.- 1.2.3 Bayesian Statistical Signal Processing.- 1.2.4 Neural Networks.- 1.3 Applications of Digital Signal Processing.- 1.3.1 Adaptive Noise Cancellation and Noise Reduction.- 1.3.2 Blind Channel Equalisation.- 1.3.3 Signal Classification and Pattern Recognition.- 1.3.4 Linear Prediction Modelling of Speech.- 1.3.5 Digital Coding of Audio Signals.- 1.3.6 Detection of Signals in Noise.- 1.3.7 Directional Reception of Waves: Beamforming.- 1.4 Sampling and Analog to Digital Conversion.- 1.4.1 Time-Domain Sampling and Reconstruction of Analog Signals.- 1.4.2 Quantisation.- 2 Stochastic Processes.- 2.1 Random Signals and Stochastic Processes.- 2.1.1 Stochastic Processes.- 2.1.2 The Space or Ensemble of a Random Process.- 2.2 Probabilistic Models of a Random Process.- 2.3 Stationary and Nonstationary Random Processes.- 2.3.1 Strict Sense Stationary Processes.- 2.3.2 Wide Sense Stationary Processes.- 2.3.3 Nonstationary Processes.- 2.4 Expected Values of a Stochastic Process.- 2.4.1 The Mean Value.- 2.4.2 Autocorrelation.- 2.4.3 Autocovariance.- 2.4.4 Power Spectral Density.- 2.4.5 Joint Statistical Averages of Two Random Processes.- 2.4.6 Cross Correlation and Cross Covariance.- 2.4.7 Cross Power Spectral Density and Coherence.- 2.4.8 Ergodic Processes and Time-averaged Statistics.- 2.4.9 Mean-ergodic Processes.- 2.4.10 Correlation-ergodic Processes.- 2.5 Some Useful Classes of Random Processes.- 2.5.1 Gaussian (Normal) Process.- 2.5.2 Multi-variate Gaussian Process.- 2.5.3 Mixture Gaussian Process.- 2.5.4 A Binary-state Gaussian Process.- 2.5.5 Poisson Process.- 2.5.6 Shot Noise.- 2.5.7 Poisson-Gaussian Model for Clutters and Impulsive Noise.- 2.5.8 Markov Processes.- 2.6 Transformation of a Random Process.- 2.6.1 Monotonic Transformation of Random Signals.- 2.6.2 Many-to-one Mapping of Random Signals.- Summary.- 3 Bayesian Estimation and Classification.- 3.1 Estimation Theory: Basic Definitions.- 3.1.1 Predictive and Statistical Models in Estimation.- 3.1.2 Parameter Space.- 3.1.3 Parameter Estimation and Signal Restoration.- 3.1.4 Performance Measures.- 3.1.5 Prior, and Posterior Spaces and Distributions.- 3.2 Bayesian Estimation.- 3.2.1 Maximum a Posterior Estimation.- 3.2.2 Maximum Likelihood Estimation.- 3.2.3 Minimum Mean Squared Error Estimation.- 3.2.4 Minimum Mean Absolute Value of Error Estimation.- 3.2.5 Equivalence of MAP, ML, MMSE and MAVE.- 3.2.6 Influence of the Prior on Estimation Bias and Variance.- 3.2.7 The Relative Importance of the Prior and the Observation.- 3.3 Estimate-Maximise (EM) Method.- 3.3.1 Convergence of the EM algorithm.- 3.4 Cramer-Rao Bound on the Minimum Estimator Variance.- 3.4.1 Cramer-Rao Bound for Random Parameters.- 3.4.2 Cramer-Rao Bound for a Vector Parameter.- 3.5 Bayesian Classification.- 3.5.1 Classification of Discrete-valued Parameters.- 3.5.2 Maximum a Posterior Classification.- 3.5.3 Maximum Likelihood Classification.- 3.5.4 Minimum Mean Squared Error Classification.- 3.5.5 Bayesian Classification of Finite State Processes.- 3.5.6 Bayesian Estimation of the Most Likely State Sequence.- 3.6 Modelling the Space of a Random Signal.- 3.6.1 Vector Quantisation of a Random Process.- 3.6.2 Design of a Vector Quantiser: K-Means Algorithm.- 3.6.3 Design of a Mixture Gaussian Model.- 3.6.4 The EM Algorithm for Estimation of Mixture Gaussian Densities.- Summary.- 4 Hidden Markov Models.- 4.1 Statistical Models for Nonstationary Processes.- 4.2 Hidden Markov Models.- 4.2.1 A Physical Interpretation of Hidden Markov Models.- 4.2.2 Hidden Markov Model As a Bayesian Method.- 4.2.3 Parameters of a Hidden Markov Model.- 4.2.4 State Observation Models.- 4.2.5 State Transition Probabilities.- 4.2.6 State-Time Trellis Diagram.- 4.3 Training Hidden Markov Models.- 4.3.1 Forward-Backward Probability Computation.- 4.3.2 Baum-Welch Model Re-Estimation.- 4.3.3 Training Discrete Observation Density HMMs.- 4.3.4 HMMs with Continuous Observation PDFs.- 4.3.5 HMMs with Mixture Gaussian pdfs.- 4.4 Decoding of Signals Using Hidden Markov Models.- 4.4.1 Viterbi Decoding Algorithm.- 4.5 HMM-based Estimation of Signals in Noise.- 4.5.1 HMM-based Wiener Filters.- 4.5.2 Modelling Noise Characteristics.- Summary.- 5 Wiener Filters.- 5.1 Wiener Filters: Least Squared Error Estimation.- 5.2 Block-data Formulation of the Wiener Filter.- 5.3 Vector Space Interpretation of Wiener Filters.- 5.4 Analysis of the Least Mean Squared Error Signal.- 5.5 Formulation of Wiener Filter in Frequency Domain.- 5.6 Some Applications of Wiener Filters.- 5.6.1 Wiener filter for Additive Noise Reduction.- 5.6.2 Wiener Filter and Separability of Signal and Noise.- 5.6.3 Squared Root Wiener Filter.- 5.6.4 Wiener Channel Equaliser.- 5.6.5 Time-alignment of Signals.- 5.6.6 Implementation of Wiener Filters.- Summary.- 6 Kalman and Adaptive Least Squared Error Filters.- 6.1 State-space Kalman Filters.- 6.2 Sample Adaptive Filters.- 6.3 Recursive Least Squares (RLS) Adaptive Filters.- 6.4 The Steepest Descent Method.- 6.5 The LMS Adaptation Method.- Summary.- 7 Linear Prediction Models.- 7.1 Linear Prediction Coding.- 7.1.1 Least Mean Squared Error Predictor.- 7.1.2 The Inverse Filter: Spectral Whitening.- 7.1.3 The Prediction Error Signal.- 7.2 Forward, Backward and Lattice Predictors.- 7.2.1 Augmented Equations for Forward and Backward Predictors.- 7.2.2 Levinson-Durbin Recursive Solution.- 7.2.3 Lattice Predictors.- 7.2.4 Alternative Formulations of Least Squared Error Predictors.- 7.2.5 Model Order Selection.- 7.3 Short-term and Long-term Predictors.- 7.4 MAP Estimation of Predictor Coefficients.- 7.5 Signal Restoration Using Linear Prediction Models.- 7.5.1 Frequency Domain Signal Restoration.- Summary.- 8 Power Spectrum Estimation.- 8.1 Fourier Transform, Power Spectrum and Correlation.- 8.1.1 Fourier Transform.- 8.1.2 Discrete Fourier Transform (DFT).- 8.1.3 Frequency Resolution and Spectral Smoothing.- 8.1.4 Energy Spectral Density and Power Spectral Density.- 8.2 Non-parametric Power Spectrum Estimation.- 8.2.1 The Mean and Variance of Periodograms.- 8.2.2 Averaging Periodograms (Bartlett Method).- 8.2.3 Welch Method ¡Averaging Periodograms from Overlapped and Windowed Segments.- 8.2.4 Blackman-Tukey Method.- 8.2.5 Power Spectrum Estimation from Autocorrelation of Overlapped Segments.- 8.3 Model-based Power Spectrum Estimation.- 8.3.1 Maximum Entropy Spectral Estimation.- 8.3.2 Autoregressive Power Spectrum Estimation.- 8.3.3 Moving Average Power Spectral Estimation.- 8.3.4 Autoregressive Moving Average Power Spectral Estimation.- 8.4 High Resolution Spectral Estimation Based on Subspace Eigen Analysis.- 8.4.1 Pisarenko Harmonic Decomposition.- 8.4.2 Multiple Signal Classification (MUSIC) Spectral Estimation.- 8.4.3 Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT).- Summary.- 9 Spectral Subtraction.- 9.1 Spectral Subtraction.- 9.1.1 Power Spectrum Subtraction.- 9.1.2 Magnitude Spectrum Subtraction.- 9.1.3 Spectral Subtraction Filter: Relation to Wiener Filters.- 9.2 Processing Distortions.- 9.2.1 Effect of Spectral Subtraction on Signal Distribution.- 9.2.2 Reducing the Noise Variance.- 9.2.3 Filtering Out the Processing Distortions.- 9.3 Non-linear Spectral Subtraction.- 9.4 Implementation of Spectral Subtraction.- 9.4.1 Application to Speech Restoration and Recognition.- Summary.- 10 Interpolation.- 10.1 Introduction.- 10.1.1 Interpolation of a Sampled Signal.- 10.1.2 Digital Interpolation by a Factor of I.- 10.1.3 Interpolation of a Sequence of Lost Samples.- 10.1.4 Factors that Affect Interpolation.- 10.2 Polynomial Interpolation.- 10.2.1 Lagrange Polynomial Interpolation.- 10.2.2 Newton Interpolation Polynomial.- 10.2.3 Hermite Interpolation Polynomials.- 10.2.4 Cubic Spline Interpolation.- 10.3 Statistical Interpolation.- 10.3.1 Maximum a Posterior Interpolation.- 10.3.2 Least Squared Error Autoregressive Interpolation.- 10.3.3 Interpolation Based on a Short-term Prediction Model.- 10.3.4 Interpolation Based on Long-term and Short-term Correlations.- 10.3.5 LSAR Interpolation Error.- 10.3.6 Interpolation in Frequency-Time Domain.- 10.3.7 Interpolation using Adaptive Code Books.- 10.3.8 Interpolation Through Signal Substitution.- Summary.- 11 Impulsive Noise.- 11.1 Impulsive Noise.- 11.1.1 Autocorrelation and Power Spectrum of Impulsive Noise.- 11.2 Stochastic Models for Impulsive Noise.- 11.2.1 Bernoulli-Gaussian Model of Impulsive Noise.- 11.2.2 Poisson-Gaussian Model of Impulsive Noise.- 11.2.3 A Binary State Model of Impulsive Noise.- 11.2.4 Signal to Impulsive Noise Ratio.- 11.3 Median Filters.- 11.4 Impulsive Noise Removal Using Linear Prediction Models.- 11.4.1 Impulsive Noise Detection.- 11.4.2 Analysis of Improvement in Noise Detectability.- 11.4.3 Two-sided Predictor.- 11.4.4 Interpolation of Discarded Samples.- 11.5 Robust Parameter Estimation.- 11.6 Restoration of Archived Gramophone Records.- Summary.- 12 Transient Noise.- 12.1 Transient Noise Waveforms.- 12.2 Transient Noise Pulse Models.- 12.2.1 Noise Pulse Templates.- 12.2.2 Autoregressive Model of Transient Noise.- 12.2.3 Hidden Markov Model of a Noise Pulse Process.- 12.3 Detection of Noise Pulses.- 12.3.1 Matched Filter.- 12.3.2 Noise Detection Based on Inverse Filtering.- 12.3.3 Noise Detection Based on HMM.- 12.4 Removal of Noise Pulse Distortions.- 12.4.1 Adaptive Subtraction of Noise pulses.- 12.4.2 AR-based Restoration of Signals Distorted by Noise Pulses.- Summary.- 13 Echo Cancellation.- 13.1 Telephone Line Echoes.- 13.1.1 Telephone Line Echo Suppression.- 13.2 Adaptive Echo Cancellation.- 13.2.1 Convergence of Line Echo Canceller.- 13.2.2 Echo Cancellation for Digital Data Transmission over Subscriber’s Loop.- 13.3 Acoustic Feedback Coupling.- 13.4 Sub-band Acoustic Echo Cancellation.- Summary.- 14 Blind Deconvolution and Channel Equalisation.- 14.1 Introduction.- 14.1.1 The Ideal Inverse Channel Filter.- 14.1.2 Equalisation Error, Convolutional Noise.- 14.1.3 Blind Equalisation.- 14.1.4 Minimum and Maximum Phase Channels.- 14.1.5 Wiener Equaliser.- 14.2 Blind Equalisation Using Channel Input Power Spectrum.- 14.2.1 Homomorphic Equalisation.- 14.2.2 Homomorphic Equalisation using a Bank of High Pass Filters.- 14.3 Equalisation Based on Linear Prediction Models.- 14.3.1 Blind Equalisation Through Model Factorisation.- 14.4 Bayesian Blind Deconvolution and Equalisation.- 14.4.1 Conditional Mean Channel Estimation.- 14.4.2 Maximum Likelihood Channel Estimation.- 14.4.3 Maximum a Posterior Channel Estimation.- 14.4.4 Channel Equalisation Based on Hidden Markov Models.- 14.4.5 MAP Channel Estimate Based on HMMs.- 14.4.6 Implementations of HMM-Based Deconvolution.- 14.5 Blind Equalisation for Digital Communication Channels.- 14.6 Equalisation Based on Higher-Order Statistics.- 14.6.1 Higher-Order Moments.- 14.6.2 Higher Order Spectra of Linear Time-Invariant Systems.- 14.6.3 Blind Equalisation Based on Higher Order Cepstrum.- Summary.- Frequently used Symbols and Abbreviations.