Pattern Classification

This book provides a unified approach for developing a fuzzy classifier and explains the advantages and disadvantages of different classifiers through extensive performance evaluation of real data sets. It thus offers new learning paradigms for analyzing neural networks and fuzzy systems, while training fuzzy classifiers. Function approximation is also treated and function approximators are compared.

I. Pattern Classification.- 1. Introduction.- 1.1 Development of a Classification System.- 1.2 Optimum Features.- 1.3 Classifiers.- 1.3.1 Neural Network Classifiers.- 1.3.2 Conventional Fuzzy Classifiers.- 1.3.3 Fuzzy Classifiers with Learning Capability.- 1.4 Evaluation.- 1.5 Data Sets Used in the Book.- 2. Multilayer Neural Network Classifiers.- 2.1 Three-layer Neural Networks.- 2.2 Synthesis Principles.- 2.3 Training Methods.- 2.4 Training by the Back-propagation Algorithm.- 2.5 Training by Solving Inequalities.- 2.5.1 Setting of Target Values.- 2.5.2 Formulation of Training by Solving Inequalities.- 2.5.3 Determination of Weights by Solving Inequalities.- 2.6 Performance Evaluation.- 2.6.1 Iris Data.- 2.6.2 Numeral Data.- 2.6.3 Thyroid Data.- 2.6.4 Blood Cell Data.- 2.6.5 Hiragana Data.- 2.6.6 Discussions.- 3. Support Vector Machines.- 3.1 Support Vector Machines for Pattern Classification.- 3.1.1 Conversion to Two-class Problems.- 3.1.2 The Optimal Hyperplane.- 3.1.3 Mapping to a High-dimensional Space.- 3.2 Performance Evaluation.- 3.2.1 Iris Data.- 3.2.2 Numeral Data.- 3.2.3 Thyroid Data.- 3.2.4 Blood Cell Data.- 3.2.5 Hiragana Data.- 3.2.6 Discussions.- 4. Membership Functions.- 4.1 One-dimensional Membership Functions.- 4.1.1 Triangular Membership Functions.- 4.1.2 Trapezoidal Membership Functions.- 4.1.3 Bell-shaped Membership Functions.- 4.2 Multi-dimensional Membership Functions.- 4.2.1 Extension to Multi-dimensional Membership Functions.- 4.2.2 Rectangular Pyramidal Membership Functions.- 4.2.3 Truncated Rectangular Pyramidal Membership Functions.- 4.2.4 Polyhedral Pyramidal Membership Functions.- 4.2.5 Truncated Polyhedral Pyramidal Membership Functions.- 4.2.6 Bell-shaped Membership Functions.- 4.2.7 Relations between Membership Functions.- 5. Static Fuzzy Rule Generation.- 5.1 Classifier Architecture.- 5.2 Fuzzy Rules.- 5.2.1 Fuzzy Rules with Pyramidal Membership Functions.- 5.2.2 Polyhedral Fuzzy Rules.- 5.2.3 Ellipsoidal Fuzzy Rules.- 5.3 Class Boundaries.- 5.3.1 Fuzzy Rules with Pyramidal Membership Functions.- 5.3.2 Ellipsoidal Fuzzy Rules.- 5.3.3 Class Boundaries for the Iris Data.- 5.4 Training Architecture.- 5.4.1 Fuzzy Rule Generation by Preclustering.- 5.4.2 Fuzzy Rule Generation by Postclustering.- 6. Clustering.- 6.1 Fuzzy c-means Clustering Algorithm.- 6.2 The Kohonen Network.- 6.3 Minimum Volume Clustering Algorithm.- 6.4 Fuzzy Min-max Clustering Algorithm.- 6.5 Overlap Resolving Clustering Algorithm.- 6.5.1 Approximation of Overlapping Regions.- 6.5.2 Extraction of Data from the Overlapping Regions.- 6.5.3 Clustering Algorithm.- 7. Tuning of Membership Functions.- 7.1 Problem Formulation.- 7.2 Direct Methods.- 7.2.1 Tuning of Slopes.- 7.2.2 Tuning of Locations.- 7.2.3 Order of Tuning.- 7.3 Indirect Methods.- 7.3.1 Tuning of Slopes Using the Least-squares Method.- 7.3.2 Tuning by the Steepest Descent Method.- 7.4 Performance Evaluation.- 7.4.1 Performance Evaluation of the Fuzzy Classifier with Pyramidal Membership Functions.- 7.4.2 Performance Evaluation of the Fuzzy Classifier with Polyhedral Regions.- 7.4.3 Performance Evaluation of the Fuzzy Classifier with Ellipsoidal Regions.- 8. Robust Pattern Classification.- 8.1 Why Robust Classification Is Necessary?.- 8.2 Robust Classification.- 8.2.1 The First Stage.- 8.2.2 The Second Stage.- 8.2.3 Tuning Slopes near Class Boundaries.- 8.2.4 Upper and Lower Bounds Determined by Correctly Classified Data.- 8.2.5 Range of the Interclass Tuning Parameter that Resolves Misclassification.- 8.3 Performance Evaluation.- 8.3.1 Classification Performance without Outliers.- 8.3.2 Classification Performance with Outliers.- 9. Dynamic Fuzzy Rule Generation.- 9.1 Fuzzy Min-max Classifiers.- 9.1.1 Concept.- 9.1.2 Approximation of Input Regions.- 9.1.3 Fuzzy Rule Extraction.- 9.1.4 Performance Evaluation.- 9.2 Fuzzy Min-max Classifiers with Inhibition.- 9.2.1 Concept.- 9.2.2 Fuzzy Rule Extraction.- 9.2.3 Fuzzy Rule Inference.- 9.2.4 Performance Evaluation.- 10. Comparison of Classifier Performance.- 10.1 Evaluation Conditions.- 10.2 Iris Data.- 10.3 Numeral Data.- 10.4 Thyroid Data.- 10.5 Blood Cell Data.- 10.6 Hiragana Data.- 10.7 Discussions.- 11. Optimizing Features.- 11.1 Scaling Features.- 11.1.1 Scale and Translation Invariance of the Fuzzy Classifier with Pyramidal Membership Functions.- 11.1.2 Invariance of the Fuzzy Classifiers with Hyperbox Regions.- 11.1.3 Invariance of the Fuzzy Classifier with Ellipsoidal Regions.- 11.2 Feature Extraction.- 11.2.1 Principal Component Analysis.- 11.2.2 Discriminant Analysis.- 11.2.3 Neural-network-based Feature Extraction.- 11.3 Feature Selection.- 11.3.1 Criteria for Feature Selection.- 11.3.2 Monotonicity of the Selection Criteria.- 11.3.3 Selection Search.- 11.4 Performance Evaluation of Feature Selection.- 11.4.1 Evaluation Conditions.- 11.4.2 Selected Features.- 11.4.3 Iris Data.- 11.4.4 Numeral Data.- 11.4.5 Thyroid Data.- 11.4.6 Blood Cell Data.- 11.4.7 Discussions.- 12. Generation of Training and Test Data Sets.- 12.1 Resampling Techniques.- 12.1.1 Estimation of Classification Performance.- 12.1.2 Effect of Resampling on Classifier Performance.- 12.2 Division of Data by Pairing.- 12.3 Similarity Measures.- 12.3.1 Relative Difference of Center Norms.- 12.3.2 Relative Difference of Covariance Matrix Norms.- 12.4 Performance Evaluation.- 12.4.1 Evaluation Conditions.- 12.4.2 Blood Cell Data.- 12.4.3 Hiragana-50 Data.- 12.4.4 Discussions.- II. Function Approximation.- 13. Introduction.- 13.1 Function Approximators.- 13.1.1 Neural Networks.- 13.1.2 Conventional Fuzzy Function Approximators.- 13.1.3 Fuzzy Function Approximators with Learning Capability.- 14. Fuzzy Rule Representation and Inference.- 14.1 Fuzzy Rule Representation.- 14.2 Defuzzification Methods.- 15. Fuzzy Rule Generation.- 15.1 Fuzzy Rule Generation by Preclustering.- 15.1.1 Clustering of Input Space.- 15.1.2 Clustering of Input and Output Spaces.- 15.2 Fuzzy Rule Generation by Postclustering.- 15.2.1 Concept.- 15.2.2 Rule Generation for FACG.- 15.2.3 Rule Generation for FALC.- 15.3 Fuzzy Rule Tuning.- 15.3.1 Concept.- 15.3.2 Evaluation Functions.- 15.3.3 Tuning of Parameters.- 15.4 Performance Evaluation.- 15.4.1 Mackey-Glass Differential Equation.- 15.4.2 Water Purification Plant.- 15.4.3 Discussions.- 16. Robust Function Approximation.- 16.1 Introduction.- 16.2 Fuzzy Function Approximator with Ellipsoidal Regions.- 16.2.1 Fuzzy Rule Representation.- 16.2.2 Fuzzy Rule Generation.- 16.3 Robust Parameter Estimation.- 16.3.1 Robust Estimation by the Least-median-of-squares Method.- 16.3.2 Robust Estimation by the Least-squares Method with Partial Data.- 16.4 Performance Evaluation.- 16.4.1 Determination of the Multiplier.- 16.4.2 Performance Comparison.- 16.5 Discussions.- III. Appendices.- A. Conventional Classifiers.- A.1 Bayesian Classifiers.- A.2 Nearest Neighbor Classifiers.- A.2.1 Classifier Architecture.- A.2.2 Performance Evaluation.- B. Matrices.- B.1 Matrix Properties.- B.2 Least-squares Method and Singular Value Decomposition.- B.3 Covariance Matrix.- References.