Statistical Methods in Software Engineering : Reliability and Risk

This preface pertains to three issues that we would like to bring to the attention of the readers: our objectives, our intended audience, and the nature of the material. We have in mind several objectives. The first is to establish a framework for dealing with uncertainties in software engineering, and for using quantitative measures for decision making in this context. The second is to bring into perspective the large body of work having statistical content that is relevant to software engineering, which may not have appeared in the traditional outlets devoted to it. Connected with this second objective is a desire to streamline and organize our own thinking and work in this area. Our third objective is to provide a platform that facilitates an interface between computer scientists and statisticians to address a class of problems in computer science. It appears that such an interface is necessary to provide the needed synergism for solving some difficult problems that the subject poses. Our final objective is to serve as an agent for stimulating more cross-disciplinary research in computer science and statistics. To what extent the material here will meet our objectives can only be assessed with the passage of time. Our intended audience is computer scientists, software engineers, and reliability analysts, who have some exposure to probability and statistics. Applied statisticians interested in reliability problems are also a segment of our intended audience.

1 Introduction and Overview.- 1.1 What is Software Engineering?.- 1.2 Uncertainty in Software Production.- 1.2.1 The Software Development Process.- 1.2.2 Sources of Uncertainty in the Development Process.- 1.3 The Quantification of Uncertainty.- 1.3.1 Probability as an Approach for Quantifying Uncertainty.- 1.3.2 Interpretations of Probability.- 1.3.3 Interpreting Probabilities in Software Engineering.- 1.4 The Role of Statistical Methods in Software Engineering.- 1.5 Chapter Summary.- 2 Foundational Issues: Probability and Reliability.- 2.0 Preamble.- 2.1 The Calculus of Probability.- 2.1.1 Notation and Preliminaries.- 2.1.2 Conditional Probabilities and Conditional Independence.- 2.1.3 The Calculus of Probability.- 2.1.4 The Law of Total Probability, Bayes’ Law, and the Likelihood Function.- 2.1.5 The Notion of Exchangeability.- 2.2 Probability Models and Their Parameters.- 2.2.1 What is a Software Reliability Model?.- 2.2.2 Some Commonly Used Probability Models.- 2.2.3 Moments of Probability Distributions and Expectation of Random Variables.- 2.2.4 Moments of Probability Models: The Mean Time to Failure.- 2.3 Point Processes and Counting Process Models.- 2.3.1 The Nonhomogeneous Poisson Process Model.- 2.3.2 The Homogeneous Poisson Process Model.- 2.3.3 Generalizations of the Point Process Model.- 2.4 Fundamentals of Reliability.- 2.4.1 The Notion of a Failure Rate Function.- 2.4.2 Some Commonly Used Model Failure Rates.- 2.4.3 Covariates in the Failure Rate Function.- 2.4.4 The Concatenated Failure Rate Function.- 2.5 Chapter Summary.- Exercises for Chapter 2.- 3 Models for Measuring Software Reliability.- 3.1 Background: The Failure of Software.- 3.1.1 The Software Failure Process and Its Associated Randomness.- 3.1.2 Classification of Software Reliability Models.- 3.2 Models Based on the Concatenated Failure Rate Function.- 3.2.1 The Failure Rate of Software.- 3.2.2 The Model of Jelinski and Moranda (1972).- 3.2.3 Extensions and Generalizations of the Model by Jelinski and Moranda.- 3.2.4 Hierarchical Bayesian Reliability Growth Models.- 3.3 Models Based on Failure Counts.- 3.3.1 Time Dependent Error Detection Models.- 3.4 Models Based on Times Between Failures.- 3.4.1 The Random Coefficient Autoregressive Process Model.- 3.4.2 A Non-Gaussian Kalman Filter Model.- 3.5 Unification of Software Reliability Models.- 3.5.1 Unification via the Bayesian Paradigm.- 3.5.2 Unification via Self-Exciting Point Process Models.- 3.5.3 Other Approaches to Unification.- 3.6 An Adaptive Concatenated Failure Rate Model.- 3.6.1 The Model and Its Motivation.- 3.6.2 Properties of the Model and Interpretation of Model Parameters.- 3.7 Chapter Summary.- Exercises for Chapter 3.- 4 Statistical Analysis of Software Failure Data.- 4.1 Background: The Role of Failure Data.- 4.2 Bayesian Inference, Predictive Distributions, and Maximization of Likelihood.- 4.2.1 Bayesian Inference and Prediction.- 4.2.2 The Method of Maximum Likelihood.- 4.2.3 Application: Inference and Prediction Using Jelinski and Moranda’s Model.- 4.2.4 Application: Inference and Prediction Under an Error Detection Model.- 4.3 Specification of Prior Distributions.- 4.3.1 Standard of Reference-Noninformative Priors.- 4.3.2 Subjective Priors Based on Elicitation of Specialist Knowledge.- 4.3.3 Extensions of the Elicitation Model.- 4.3.4 Example: Eliciting Priors for the Logarithmic-Poisson Model.- 4.3.5 Application: Failure Prediction Using Logarithmic-Poisson Model.- 4.4 Inference and Prediction Using a Hierarchical Model.- 4.4.1 Application to NTDS Data: Assessing Reliability Growth.- 4.5 Inference and Predictions Using Dynamic Models.- 4.5.1 Inference for the Random Coefficient Exchangeable Model.- 4.5.2 Inference for the Adaptive Kalman Filter Model.- 4.5.3 Inference for the Non-Gaussian Kalman Filter Model.- 4.6 Prequential Prediction, Bayes Factors, and Model Comparison.- 4.6.1 Prequential Likelihoods and Prequential Prediction.- 4.6.2 Bayes’ Factors and Model Averaging.- 4.6.3 Model Complexity Occam’s Razor.- 4.6.4 Application: Comparing the Exchangeable, Adaptive, and Non-Gaussian Models.- 4.6.5 An Example of Reversals in the Prequential Likelihood Ratio.- 4.7 Inference for the Concatenated Failure Rate Model.- 4.7.1 Specification of the Prior Distribution.- 4.7.2 Calculating Posteriors by Markov Chain Monte Carlo.- 4.7.3 Testing Hypotheses About Reliability Growth or Decay.- 4.7.4 Application to System 40 Data.- 4.8 Chapter Summary.- Exercises for Chapter 4.- 5 Software Productivity and Process Management.- 5.1 Background: Producing Quality Software.- 5.2 A Growth-Curve Model for Estimating Software Productivity.- 5.2.1 The Statistical Model.- 5.2.2 Inference and Prediction Under the Growth-Curve Model.- 5.2.3 Application: Estimating Individual Software Productivity.- 5.3 The Capability Maturity Model for Process Management.- 5.3.1 The Conceptual Framework.- 5.3.2 The Probabilistic Approach for Hierarchical Classification.- 5.3.3 Application: Classifying a Software Developer.- 5.4 Chapter Summary.- Exercises for Chapter 5.- 6 The Optimal Testing and Release of Software.- 6.1 Background: Decision Making and the Calculus of Probability.- 6.2 Decision Making Under Uncertainty.- 6.3 Utility and Choosing the Optimal Decision.- 6.3.1 Maximization of Expected Utility.- 6.3.2 The Utility of Money.- 6.4 Decision Trees.- 6.4.1 Solving Decision Trees.- 6.5 Software Testing Plans.- 6.6 Examples of Optimal Testing Plans.- 6.6.1 One-Stage Testing Using the Jelinski-Moranda Model.- 6.6.2 One-and Two-Stage Testing Using the Model by Goel and Okumoto.- 6.6.3 One-Stage Lookahead Testing Using the Model by Goel and Okumoto.- 6.6.4 Fixed-Time Lookahead Testing for the Goel–Okumoto Model.- 6.6.5 One-Bug Lookahead Testing Plans.- 6.6.6 Optimality of One-Stage Look Ahead Plans.- 6.7 Application: Testing the NTDS Data.- 6.8 Chapter Summary.- Exercises for Chapter 6.- 7 Other Developments: Open Problems.- 7.0 Preamble.- 7.1 Dynamic Modeling and the Operational Profile.- 7.1.1 Martingales, Predictable Processes, and Compensators: An Overview.- 7.1.2 The Doob-Meyer Decomposition of Counting Processes.- 7.1.3 Incorporating the Operational Profile.- 7.2 Statistical Aspects of Software Testing: Experimental Designs.- 7.2.1 Inferential Issues in Random and Partition Testing.- 7.2.2 Comparison of Random and Partition Testing.- 7.2.3 Design of Experiments in Software Testing.- 7.2.4 Design of Experiments in Multiversion Programming.- 7.2.5 Concluding Remarks.- 7.3 The Integration of Module and System Performance.- 7.3.1 The Protocols of Control Flow and Data Flow.- 7.3.2 The Structure Function of Modularized Software.- Appendices.- Appendix A Statistical Computations Using the Gibbs Sampler.- A.1 An Overview of the Gibbs Sampler.- A.2 Generating Random Variates The Rejection Method.- A.3 Examples: Using the Gibbs Sampler.- A.3.1 Gibbs Sampling the Jelinski-Moranda Model.- A.3.2 Gibbs Sampling the Hierarchical Model.- A.3.3 Gibbs Sampling the Adaptive Kalman Filter Model.- A.3.4 Gibbs Sampling the Non-Gaussian Kalman Filter Model.- Appendix B The Maturity Questionnaire and Responses.- B. 1 The Maturity Questionnaire.- B.2 Binary (Yes, No) Responses to the Maturity Questionnaire.- B.3 Prior Probabilities and Likelihoods.- References.- Author Index.