Intelligent Decision Support Systems Based on Machine Learning and Multicriteria Decision-MakingView this Special Issue
Failure Prediction of Aircraft Equipment Using Machine Learning with a Hybrid Data Preparation Method
There is a large amount of information and maintenance data in the aviation industry that could be used to obtain meaningful results in forecasting future actions. This study aims to introduce machine learning models based on feature selection and data elimination to predict failures of aircraft systems. Maintenance and failure data for aircraft equipment across a period of two years were collected, and nine input and one output variables were meticulously identified. A hybrid data preparation model is proposed to improve the success of failure count prediction in two stages. In the first stage, ReliefF, a feature selection method for attribute evaluation, is used to find the most effective and ineffective parameters. In the second stage, a K-means algorithm is modified to eliminate noisy or inconsistent data. Performance of the hybrid data preparation model on the maintenance dataset of the equipment is evaluated by Multilayer Perceptron (MLP) as Artificial Neural network (ANN), Support Vector Regression (SVR), and Linear Regression (LR) as machine learning algorithms. Moreover, performance criteria such as the Correlation Coefficient (CC), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE) are used to evaluate the models. The results indicate that the hybrid data preparation model is successful in predicting the failure count of the equipment.
Reliability and availability of aircraft components have always been an important consideration in aviation. Accurate prediction of possible failures will increase the reliability of aircraft components and systems. The scheduling of maintenance operations help determine the overall maintenance and overhaul costs of aircraft components. Maintenance costs constitute a significant portion of the total operating expenditure of aircraft systems.
There are three main types of maintenance for equipment: corrective maintenance, preventive maintenance, and predictive maintenance . Corrective maintenance helps manage repair actions and unscheduled fault events, such as equipment and machine failures. When aircraft equipment fails while it is in use, it is repaired or replaced. Preventive maintenance can reduce the need for unplanned repair operations. It is implemented by periodic maintenance to avoid equipment failures or machinery breakdowns. Tasks for this type of maintenance are planned to prevent unexpected downtime and breakdown events that would lead to repair operations. Predictive maintenance, as the name suggests, uses some parameters which are measured while the equipment is in operation to guess when failures might happen. It intends to interfere with the system before faults occur [1, 2] and help reduce the number of unexpected failures by providing the maintenance personnel with more reliable scheduling options for preventive maintenance. Assessing system reliability is important to choose the right maintenance strategy.
Machine learning is a rising technology that is supposed to develop in the future. Machine learning methods are applied in prediction/preventive systems, communications, security, energy management, and so on . The data preparing level is the core module of machine learning and the decision making system. It manages the data to make it useful for decision. The decision making depends on future forecasting, failure event, and availability of equipment . Data mining is a way of classifying and clamping data into comprehensible information. It comprehends the applicable models from a mass of information and adopts different approaches to uncover secret data. Data mining can be defined as knowledge derivation from raw data .
Feature selection is a fundamental issue in data mining and machine learning algorithms that focus on the features which are the most relevant to the intended prediction . Features collected from the observation of a circumstance are not all equivalently significant. Normally, operational data tend to be incomplete, insufficient, or partially meaningful or not meaningful at all. Some of them may be noisy, redundant, or irrelevant. Feature selection aims to choose a feature set that is relevant to a specific duty. This problem is a complex and multidimensional issue . Hsu  proposed a novel feature selection algorithm based on the correlation coefficient clustering method. It focused on reducing noisy, repeated, or redundant features. The performance in the computational speed and the classification accuracy can be improved through the removal of the irrevelant features. Methods of data processing helps improve the quality of the data and increase the accuracy of data mining, thereby making it more efficient . Data quality is important for the process of information discovery, checking data anomalies, and predicting and analyzing for decision making . Predicting equipment failures are essential to reduce repair and equipment costs and to assess equipment availability .
Mass data can be useful for businesses and can guide systems to follow right paths. To boost performance in machine learning algorithms, it is critical that meaningful information be gathered from the dataset. To eliminate noisy and irrevelant data during data preparation, we used K-means clustering algorithm, which is one of the popular unsupervised machine learning algorithms. It defines k number of centroids and allocates every case to the nearest cluster while keeping the centroids small . The “means” in the K-means refers to the averaging of the dataset to find the centroid. This algorithm assigns each case to only a single set. The purpose is to accomplish a high level of similarity within the clusters and low similarly across them . It is used for more effective and better clustering with decreased complexity.
There are many studies on maintenance data and forecasting failure rates. Data preparation is a critical step in the feature selection process, and it has a major effect on the success of a machine learning algorithm. Gurbuz et al.  applied various techniques of preprocessing along with feature selection on 15 datasets of a Turkish airline company to understand and clean the dataset and to find the relationships between input and output features. They came up with 15 rules for creating failure alerts, and these were found useful by the experts of the aviation company. Classification algorithm was used to extract patterns within equipment data. Kutlylowska  proposed the application of artificial neural networks to failure rate modelling. Data from a water utility in Poland were used to estimate the output values of failure frequency. The results showed that artificial networks could be an option to assess the frequency of problems in the systems of water supply. Ramos et al.  carried out a study to predict malfunctions and to do predictive maintenance practices in a piece of manufacturing equipment. In this study, ARIMA forecasting methods were successfully compared with neural network models. The results indicated that both models were good at forecasting defibrator disc replacement, but the ARIMA was much better in the forecasting the distance between the discs. Trani et al.  introduced a basic method to estimate aircraft fuel consumption through the use of an artificial neural network. A fuel consumption model supported by a neural network was created by using the data given in the performance manual of the aircraft. The results from the neural network model were compared with analytical models. The results revealed that the proposed three-layer ANN with nonlinear transfer functions could correctly estimate fuel consumption in different stages of a flight. Ming et al.  investigated the use of the ANN method in vibration analysis by using the integrated data from the devices of vibration to predict equipment failures. The ANN model was applied to diagnose the faults in a mill. The results lent support to the efficiency of this methodology. Kozik and Sep  applied ANN forecasting to identify the demand for spare parts to be replaced during aircraft engine overhaul. The results indicated that the forecasting method that is composed of the engine’s hardtime calculation should be a power in the implementation of lean manufacturing in MRO (maintenance, repair, and overhaul) facilities. Altay et al.  used 532 failures of 60 aircraft to model an artificial neural network to forecast failures. The proposed model produced high correlation rates of prediction between the actual and target failure times of aircraft. Benkedjouh et al.  proposed a method to guess the useful life (RUL) in machinery with bearings. For this purpose, the researchers used the isometric feature mapping reduction technique (ISOMAP) and support vector regression (SVR). Moura et al.  presented an analysis to comparatively assess the SVM effectiveness in predicting failure time. The performance of SVM regression is compared with other learning methods.
This paper discusses the feature selection of variables in the maintenance data obtained from an aviation company in Turkey. The proposed system will help companies to collect, extract, and create data to improve the maintenance actions through more accurate predictions. This study proposes a hybrid data preparation method for maintenance data and predicting failure counts of equipment by comparing the results of three different algorithms. The feature selection method (ReliefF algorihm in the present study) is used for selecting attributes, and the modified K-means algorithm is used to eliminate the redundant data. Three methods for predicting equipment failure counts are introduced and compared using MLP as an ANN algorithm, SVR algorithm, and LR. The next section presents an overview of the materials and methodology, followed by experimental results and conclusions.
2. Materials and Methodology
The context in which the present case study was carried out was an avitation company in Ankara, Turkey. The maintenance data were collected from the records of the maintenance department. They included removal of equipment, repair activities, experience of the operators, flight hours of the equipment, and other information relevant to the case study.
2.1. Dataset Acquisition
In the ERP platform, a program is developed to collect data and to format the dataset for analysis through machine algorithms. The variables were grouped as the input variables, while the failure count was considered to be the output variable. Selected parameters are evaluated by the feature selection ReliefF method to find the most influential parameters that have a share in the failures.
Flow logic of the developed program is presented in Figure 1. Firstly, the selected materials’ serial-numbered equipment used in the landing gear system was selected. Their maintenance and operational data were identified. Attributes of the maintenance and failure data were identified in cooperation with maintenance personnel and technicians. Each instance of a no fault found (NFF) status was examined to find confirmed failure data. The total flight hours for each piece of equipment across different aircraft were calculated for a given time period. Nine input variables that affect the failure of the equipment were determined. The failure count was calculated as an output variable. These nine input variables and an output variables were used for modelling the machine learning algorithms used in the present study (MLP, SVR, and LR).
2.2. Feature Selection and ReliefF Algorithm
Feature selection is a technique to obtain the relevant features by removing irrelevant and noisy data from the original dataset. It is the process of selecting a subset of features that could adequately depict all the datasets. The main objective of feature selection is to mine the data to obtain the minimum number of features to achieve maximum accuracy. Feature selection methods are used in data mining and machine learning, as well as in artificial intelligence. They reduce model complexity and let algorithms operate faster. Relief is a feature selection algorithm used for random selection of instances for feature weight calculation. The Relief algorithm is proposed by Kira and Rendell in 1992 [20, 21]. It estimates feature weights iteratively, according to their ability to make a distinction between neighboring models. Relief was extended to deal with noisy, irrevelant, and missing data to address multiclass issues. Kononenko  proposed an extension to Relief called ReliefF to address the multiclass problems. ReliefF is an extension of the Relief algorithm, which fails to remove irrelevant or incomplete features in two-class classification problems. The ReliefF algorithm finds one near miss for each different class and averages their value to revise feature weights.
2.3. Data Elimination and Modified K-Means Algorithm
K-means, a widely used algorithm in a wide range of applications, was first developed in 1967 by MacQueen . It allows each data point to be a member of a single set. It has limitation fields: fixed K value and an initial centroid. It uses the distance criterion as the Euclidean distance measurement. Fahad and Alam  proposed a method by using modified K-means algorithm, which proved less time consuming yet more efficient in clustering. The quality of the resulting clusters depends on the selection of the initial centroid. K-means algorithm makes it possible to create a new data cluster by eliminating the smallest class value represented in the cluster. Yilmaz et al.  proposed a system using modified K-means algorithm to eliminate noisy and irrevelant data. In this study, we used modified K-means algorithm as in  and developed the pseudocode given in Algorithm 1.
2.4. Prediction Methods
In recent years, research on machine learning algorithms and data mining has been carried out to study failure prediction applications. In this study, the MLP, SVR, and LR algorithms were examined to model maintenance data and predict the failure count.
2.4.1. Multilayer Perceptron as an Artificial Neural Network
An Artificial Neural Network (ANN) is a mathematical model based on a biological interconnected group of artificial neurons. ANN imitates the brain’s ability to process the information approach to computation [24, 25]. Neural networks are a nonlinear statistical data modelling and machine learning method. They can be used to model complex nonlinear relationships between inputs and outputs in the data. They also describe patterns or relationships in the data, and they help forecast output values with the help of training, learning, and testing processes. A cell in a neural network is called a neuron, and a fixed number of neurons build a layer. Neurons connect to other neurons in other layers by a weight factor. ANN algorithms compute weights for input values, hidden layer, and output layer neurons by a feed forward approach [26, 27]. Weights in an ANN are calculated by using a training algorithm as the most popular backpropagation algorithm. Backpropagation is a learning algorithm that seeks to minimize the difference between the real and target outputs. The weights are updated, so that the total error is distributed to the various neurons in the neural network. The error remains at a low level through feeding forward and backpropagating . The predictive capability of neural networks comes from their multilayered structure. Neural networks have an input layer, one or more hidden layers, and an output layer. MLP algorithms are comprised of the activation function of the neurons . In this study, multilayer perceptron (MLP) feed forward neural networks were used with a backpropagation learning algorithm.
2.4.2. Support Vector Regression
Support Vector Machines (SVM) algorithm was introduced by Cortes and Vapnik in 1995 . It is a linear model used to address classification and regression problems. The SVM algorithm produces a hyperplane that classifies the data. There are two distinct classes separated by a linear plane. The training in the algorithm involves the process of identifying the parameters . Support Vector Regression (SVR) is a regression algorithm that uses a similar method of SVM to carry out regression analysis. SVR is a supervised machine learning algorithm and an effective method which can be used for prediction and data mining and is successfully adopted for regression problems.
2.4.3. Linear Regression
Linear regression is defined as a machine learning algorithm that is based on supervised learning, involving a regression task. It is used to model the linear relationship among dependent variables or independent variables. It helps determine the relationship between variables and prediction. Schuld et al.  proposed a prediction algorithm on a quantum computer, based on a linear regression model with least-squares optimization. Its scheme focused on the machine learning task of assuming the output corresponding to a new input. The prediction result can be used for further quantum information processing routines.
2.5. Evaluation Performance Measures
In this study, the mean absolute error (MAE), root mean square error (RMSE), and correlation coefficient (CC) criteria were used to evaluate the success of the all the models. There are many error measurement techniques, and they are most commonly used to quantify error measures. The error parameters, adopted from , are presented in the following equations, respectively.where N is the number of data; Xi is the observed value; Yi is the predicted value; is the mean of the observed data, and is the mean of the observed data and predicted data values. CC measures the variability of the observed data defined by the model as a correlation coefficient.
3. Proposed Methods
In this study, as noted in Section 2.1, the 585-line maintenance data in two years from a Turkish aviation company were used. The dataset consists of nine input variables and an output variable (failure count). The input variables/factors are operational and environmental parameters which could influence failure occurrence and the length of operation before failures occur. Input variables include such parameters as flight hours, the number of removals of equipment, and the number of faults with planned/unplanned removals. These data were analysed and represented in a format suitable for modelling, and variables were characterised with the corresponding domain classification, shown in Table 1. The output variable is the number of equipment failures. A sample of the dataset is provided in Table 2.
Feature selection is carried out using these ten attributes. The number of equipment failures is the target of the analysis. For this purpose, feature selection ReliefF algorithm was used to find relations and weighting coefficient dependencies. According to the ranked values, four most effective attributes were selected (Table 3).
Noisy and inconsistent data in the prepared datasets often affect prediction negatively and reduce the performance of the system. Therefore, the modified K-means algorithm was used to eliminate the noisy and inconsistent data to increase the performance of the prediction. It was developed using the pseudocode given in Algorithm 1. In this model, set centers are initially allocated, and instances are properly distributed to the sets. A predetermined number (N = 5) of records furthest to the center in each set were eliminated. The distance criterion was the Euclidean distance measurement. The eliminated instances are shown in Figure 2.
Seventy-five records (approximately 13%) of the dataset were eliminated by the pseudocode of the proposed data preparation model. Five hundred and ten records were obtained from 585 records of the dataset. Our proposed hybrid data preparation model is comprised of two stages, as shown in Figure 3. In the first stage, nine input attributes were reduced to four attributes by feature selection ReliefF algorithm. In the second stage, the dataset was reduced to 510 records by the modified K-means algorithm. The obtained dataset with 510 records were provided as inputs to the MLP, SVM, and LR prediction algorithms.
4. Experimental Results
A program is developed to gather data for analysis through machine algorithms. Selected equipment’s maintenance and operational data were identified. Nine input variables and an output variable were determined. According to using pure 585 rows, nine inputs, and an output (585 × 10) data, MLP, LR, and SVR models were trained and tested. The parameters of the predictors used in the study are provided in Tables 4–6, respectively.
To illustrate the performance of the suggested two-phase hybrid system, the prediction results for the raw dataset that is composed of 585 records and 9 attributes are presented in Table 7. Table 7 shows that based on the CC performance criterion, the best results were provided by the SVR algorithm, while the LR algorithm provided the best results based on the MAE and RMSE performance criteria.
Then, the ReliefF algorithm was applied to the raw data to identify the features that prove the most effective in prediction. Feature selection ReliefF algorithm was applied to nine input parameters, and according to ranked values, the last five of them were eliminated. MLP, LR, and SVR models are built with selected 585 rows, four inputs, and an output. The dataset (585 × 5) was trained and tested. As seen in Table 8, all the results are better than those obtained without feature selection. The results provided by the MLP (CC = 0.9127, MAE = 0.7301, RMSE = 0.9853) is better than those of the other algorithms. The performance results for the error parameters in each prediction algorithm are provided in Table 8.
In the final phase, the modified K-means algorithm was applied to the dataset to eliminate noisy and inconsistent data (585 × 5). The best k value was found to be (k = 15). Five parameters that were farthest from the center of each cluster were eliminated. As a result, 75 rows were eliminated. So, a hybrid model approach was applied to the maintenance data, and the quality of the data was improved. The LR, MLP, and SVR models were built with the selected 510 rows, four inputs, and an output. The selected data (510 × 5) were trained and tested. The results indicated that the performance of the model was highly successful, compared to the other results obtained without feature selection and data reduction. The performance of the LR, MLP, and SVR algorithms are presented in Table 9.
As shown in Table 9, based on the CC, MAE, and RMSE performance criteria, the best results were provided by the LR algorithm in the suggested two-phase hybrid system. For the test data, CC = 0.9316, MAE = 7108, and RMSE = 0.835 were obtained.
Figures 4–6 provide the linear correlation between the predicted and target results for the test data of LR, SVR, and MLP, respectively. The results indicated that the regression line in the test and the predicted data of LR provide y1 = 0.9976x + 0.0155; those of SVR provide y2 = 0.9184x + 0.48, and those of MLP provide y3 = 0.9999x − 0.0744.
The target and predicted values in the test dataset of the suggested two-phase hybrid system were provided for the LR, SVR, and MLP in Figures 7–9, respectively. The test results provided in Figures 7–9 are the graphical representation of the test results of the 1-fold cross validation.
Table 9 presents the predicted and target values obtained through the two-phase hybrid system in Figures 7–9. As seen in Figures 7–9, the proposed hybrid data preparation model increased the performance of prediction models LR, SVR, and MLP. The suggested hybrid system helped attain higher accuracy in prediction as it enabled us to select the most effective features and eliminate noisy or redundant data that could lower the accuracy of predictions.
In aviation, the use of maintenance data is highly critical in the analysis of reliability and maintenance costs. This is because predictive maintenance scheduling can be planned in line with estimates. The main target of predictive maintenance is to predict equipment failures and planning strategies for spare parts of the system components to analyze the reliability and maintainability of a complex repairable system. In this study, a hybrid data preparation model was applied to the landing gear system maintenance dataset using feature selection ReliefF algorithm to select attributes and a modified K-means algorithm to eliminate noisy and inconsistent data. The proposed hybrid data preparation method was put into practice through LR, SVR, and MLP models. The results indicated that the LR model had better performance than MLP and SVR models in predicting the failure counts. The results indicate that the proposed hybrid data preparation model significantly improves the accurate prediction of failure counts. This study could function as a guide for using hybrid data preparation methods in machine learning algorithms and data mining.
The maintenance data used to support the findings of this study have not been made available because sharing the data might compromise data privacy. Moreover the authors are not allowed to share these data due to security concerns.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This study was supported by the Scientific Research Project of Havelsan and Presidency of Defence Industries project, grant no. HVL-SÖZ-18/033.
P. Bastos, I. Lopes, and L. Pires, “A maintenance prediction system using data mining,” in Proceedings of the Proceedings of the World Congress on Engineering, vol. 3, pp. 2–7, London, UK, July 2012.View at: Google Scholar
S. Lecturer and A. Pradesh, “Feature selection using ReliefF algorithm,” International Journal of Advanced Research in Computer and Communication Engineering, vol. 3, no. 10, pp. 8215–8218, 2014.View at: Google Scholar
S. F. Rosario, “RELIEF: feature selection approach,” International Journal of Innovative Research and Development, vol. 4, no. 11, pp. 218–224, 2015.View at: Google Scholar
S. K. A. Fahad and M. Alam, “A modified K-means algorithm for big data clustering,” International Journal of Computer Science Engineering and Technology, vol. 6, no. 4, pp. 129–132, 2016.View at: Google Scholar
P. Ramos, J. M. Oliveira, and P. Silva, “Predictive maintenance of production equipment based on neural network autoregression and ARIMA,” in Proceedings of the 21st International EurOMA Conference-Operations Management in An Innovation Economy, pp. 1–10, Helsinki, Finland, June 2014.View at: Google Scholar
A. A. Trani, F. C. Wing-Ho, G. Schilling, H. Baik, and A. Seshadri, “A neural network model to estimate aircraft fuel consumption,” in Proceedings of the AIAA 4th Aviation Technology, Integration, and Operations Forum, ATIO, vol. 2, pp. 669–692, Chicago, IL, USA, September 2004.View at: Publisher Site | Google Scholar
M. Chen, R. Zhou, R. Zhang, and X. Zhu, “Application of artificial neural network to failure diagnosis on process industry equipments,” in Proceedings 2010 6th International Conference on Natural Computation, ICNC 2010, vol. 3, pp. 1190–1193, Yantai, China, August 2010.View at: Publisher Site | Google Scholar
P. Kozik, “Aircraft engine overhaul demand forecasting using ANN,” Management and Production Engineering Review, vol. 3, no. 2, pp. 21–26, 2012.View at: Google Scholar
K. Kira and L. A. Rendell, “Feature selection problem: traditional methods and a new algorithm,” in Proceedings Tenth National Conference on Artificial Intelligence, 1992.View at: Google Scholar
I. Kononenko, “Estimating attributes: analysis and extensions of RELIEF,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 784, pp. 171–182, Springer, Berlin, Germany, 1994.View at: Publisher Site | Google Scholar
J. MacQueen, “Some methods for classification and analysis of multivariate observations,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA, 1967.View at: Google Scholar
N. Nadai, A. H. A. Melani, G. F. M. Souza, and S. I. Nabeta, “Equipment failure prediction based on neural network analysis incorporating maintainers inspection findings,” in Proceedings of the Annual Reliability and Maintainability Symposium, Orlando, FL, USA, January 2017.View at: Publisher Site | Google Scholar
T. S. Lan, P. C. Chen, M. Y. Wang, K. S. Hsu, and T. Y. Chen, “A study of using back-propagation network to predict aircraft component life span,” in 2016 International Conference on Applied System Innovation,” in Proceedings of the IEEE ICASI 2016, Okinawa, Japan, May 2016.View at: Publisher Site | Google Scholar
M. Mahmudul, A. Mia, S. K. Biswas, M. C. Urmi, and A. Siddique, “An algorithm for training multilayer perceptron MLP for image reconstruction using neural network without overfitting,” International Journal of Scientific & Technology Research, vol. 4, no. 2, pp. 271–275, 2015.View at: Google Scholar