Computational Intelligence for Missing Data Imputation, Estimation, and Management: Knowledge Optimization Techniques

Descriptions and implementations for using these missing data estimation process follow (Bishop, 1995; Marwala, 2007). These methods are implemented in both the Bayesian and maximum-likelihood framework (Marwala, 2001).

It is still very difficult to know beforehand which of these computational intelligence methods are ideal for missing data imputation. For this reason, hybrid methods are also introduced and implemented in this book for missing data imputation. In particular, the ensemble methods that use more than one learning algorithm are considered (Perrone & Cooper, 1993). Some of these methods are computationally intensive, and as a result, the book introduces methods that are computationally efficient, such as the principal component analysis (Adams et al., 2002) and dynamic programming method (Bellman, 1957; Bertsekas, 2000).

In this book, many optimization methods are used. For example, to train multi-layer perceptrons, a scaled conjugate gradient optimization method (Møller, 1993) is used. Other optimization methods used are:

It is difficult to know in advance which optimization method to use for missing data estimation process and, therefore, this book also explores various hybrid optimization techniques. Some of the hybrid optimization techniques that are considered in this book include the hybrid of genetic algorithms and particle swarm optimization.

Traditional missing data imputation methods have been largely based on static models. Even computational intelligence methods are traditionally constructed in a static manner. These methods are static in the sense that they are the same over time. For situations where the concepts are drifting and, therefore, the data are non-stationary, these methods fail (Kubat, & Widmer, 1996). Many engineering problems have to model systems that are continuously changing because of aging. Therefore, for many engineering problems, data imputation methods are required that are immune or at best can handle these changes in the character of the systems. This problem, therefore, requires missing data models that evolve with the systems on which they are based. Evolutionary methods have been successful in designing learning machines that evolve with systems. These evolutionary methods include genetic algorithms (Goldberg, 2002), fuzzy maps (Carpenter et al., 1992), particle swarm optimization (Kennedy & Eberhart, 1995) and are described in detail in this book.

Throughout this book, examples from the literature and case studies are used to illustrate the effectiveness of the presented missing data estimation methods. Some of the case studies used include the artificial taster, HIV and a mechanical system.

SUMMARY OF THE BOOK

In Chapter 1, traditional missing data issues, such as missing data patterns and mechanisms, are described. Attention is paid to the best models to deal with particular missing data mechanisms. A review of traditional missing data imputation methods is conducted, and the methods reviewed include case deletion and prediction rules (Acork, 2005). The case deletion methods reviewed are list-wise and pair-wise deletion. The prediction rule imputation techniques reviewed are mean substitution, hot-deck, regression and decision trees. Two missing data examples are studied, namely, the Sudoku puzzle and a mechanical system.

Missing data estimation processes requires mathematical models that capture interrelationships amongst the variables. In Chapter 2, a method is presented that is aimed at approximating missing data and, thereby, capturing variables’ interrelationships by combining genetic algorithms and autoassociative neural networks. The neural network architectures implemented are the multi-layer perceptron and the radial basis function neural networks (Russell & Norvig, 1995). The proposed procedures are tested and then compared for missing data imputation.

The ability to identify a model which captures the interrelationships between the variables is very important. Different models bring unique perspectives to the missing data problem and one way to maximize the performance of the missing data procedure is to hybridize different methods. In Chapter 3, the hybrid of autoassociative neural networks models are developed and used in conjunction with genetic algorithms (Goldberg, 2002) to estimate missing data. One hybrid technique combines three neural networks to form a hybrid autoassociative network, while the other merges principal component analysis and neural networks. These procedures are compared to the Bayesian auto-associative neural network (Bishop, 1995) and the genetic algorithm approach.

In Chapter 4, two techniques, i.e., Gaussian mixture models trained using the Expectation Maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977) and the combined auto-associative neural networks and particle swarm optimization methods are implemented for missing data estimation and then compared. Of a particular interest is the nature of the data in the analysis that suits each of these methods.

Chapter 5 investigates an imputation technique based on rough sets computation (Wu, Mi, & Zhang, 2003). The characteristic relations are introduced to describe incompletely specified decision tables and then used for missing data estimation. Empirical results obtained using real data are given and insights into the problem of missing data are derived.

In Chapter 6, autoassociative neural networks, principal components analysis and support vector regression (Marivate, Nelwamondo, & Marwala, 2008) are all combined with genetic algorithms, and then used to impute missing variables. The impact of using the principal component analysis on the overall performance of the autoassociative network and support vector regression is then assessed.

In Chapter 7, a committee of networks is introduced for missing data estimation. This committee of networks consists of a multi-layer perceptron, support vector machines and radial basis functions. It is constructed through a weighted combination of the three networks. The networks committee is implemented collectively with a hybrid of the genetic algorithm and the particle-swarm optimization method for missing data estimation, and is then tested and assessed. Furthermore, evolutionary methods are used to evolve a committee of networks. The results of this committee are compared to the results from a traditional committee and stand-alone networks.

The use of inferential sensors is common in on-line fault detection systems in various control applications. A problem arises when sensors fail while the system is designed to make a decision based on the data from those sensors. Various techniques to handle missing data are discussed in Chapter 8. Firstly, a novel algorithm that classifies and regresses in the presence of missing data is proposed. The algorithm is tested for both classification and regression problems. Secondly, an estimation algorithm that uses an ensemble of regressors within the context of the boosting mechanism is proposed. Hybrid genetic algorithms and fast simulated annealing are used to predict missing values and the results are compared.

In Chapter 9, a classifier method is presented that is based on a missing data estimation framework, and which uses auto-associative multi-layer perceptron neural networks and genetic algorithms. The method is tested and compared to conventional feed-forward neural network using classification accuracies and the area under the receiver operating characteristics curve.

In Chapter 10, various optimization methods are compared with the aim of optimizing the missing data estimation equation, which is made out of the auto-associative neural networks with missing values as design variables. These optimization techniques are the genetic algorithm, particle swarm optimization, hill climbing and simulated annealing. They are tested and the results obtained are compared.

In implementing solutions to the missing data estimation problem, using optimization techniques, the definition of variable bounds is of critical importance. Chapter 11 introduces a novel paradigm to impute missing data that combines decision trees with an auto-associative neural network and principal component analysis. This is designed to answer the crucial question on whether the optimization bounds actually matter in the estimation of missing data. In the model, a decision tree is used to predict search bounds for a hybrid simulated annealing and genetic algorithm that minimizes an error function derived from the respective models. The results obtained are compared.

Chapter 12 presents a control mechanism to assess the effect of a demographic variable, education level, on the HIV risk of individuals. This is intended to assist for understanding the extent to which the spread of HIV can be controlled by using the variable education level. This control mechanism is based on missing data frameworks where the missing data are the set points for control. An inverse neural network model and a missing data approximation model, based on an auto-associative neural network and the genetic algorithm, are used for the control mechanism and the results obtained are then compared.

In Chapter 13, a computational intelligence approach to predicting missing data in the presence of concept drift is presented, using an en¬semble of multi-layered feed-forward neural networks. An algorithm that detects concept drift is presented. Six in¬stances prior to the occurrence of missing data are used to approximate the missing values. The algorithm is applied to simulated time-series data set resembling the non-stationary data from a sensor. Secondly, an algorithm that uses dynamic programming and neural networks to solve the problem of missing data imputation is presented, tested and the results are assessed. Thirdly, the impact of missing data estimation on fault classification in mechanical systems is studied. The missing data estimation method is based on auto-associative neural networks where the network is trained to recall the input data through some non-linear neural network mapping using genetic algorithm. The classification methods used are extension neural networks and Gaussian mixture models.

TARGET AUDIENCE OF THIS BOOK

This book is intended for researchers and practitioners who use data analysis to build decision support systems. In particular the target audience includes engineers, scientists and statisticians. The areas of engineering where decision support tools are becoming widely used (the target audience of this book) are aerospace, mechanical, civil, biomedical and electrical engineering. Furthermore, researchers in statistics and social science will also find the techniques introduced in this book to be highly applicable to their work. This book is carefully written to give a good balance between theory and application of various missing data estimation techniques. The applications selected reflect the target audience of this book and include examples from various branches of engineering.