Outlier Detection Algorithm Suite in MATLAB

Abstract

For many applications, data mining systems are required to detect anomalous (abnormal or unexpected) observations. This has so far proven to be a difficult challenge because anomalies are usually considered to be “non-normal” observations, where “normality” is typically defined by very complex concepts. Because of these and other reasons, there are no standard and principled approaches for outlier detection. Outlier detection is an important problem that has been researched within diverse research areas and application domains. Many outlier detection techniques have been specifically developed for certain application domains, while others are more generic. Although outliers are often considered as an error or noise, they may carry important information. Detected outliers are candidates for aberrant data that may adversely lead to model misspecification and incorrect results. The aim of this project is to implement various well known outlier detection algorithms for use by people dealing in data mining as well as for general purposes. Different approaches for outlier detection namely Statistical based approach, Distance based approach and Conditional Anomaly Detection are covered under this project. Various algorithms are implemented namely Univariate outlier detection using Boxplot, Multivariate outlier detection using Mahalanobis Distance Measure, Multivariate outlier detection using Euclidean Distance Measure, Multivariate outlier detection using Randomization and Pruning, Bayesian Network based outlier detection for discrete nodes using conditional probability tables and Bayesian Network based outlier detection for discrete and continuous nodes using parameter learning. The project also includes a graphical user interface for the suite in MATLAB. In addition to above, the recall for above stated algorithms has been calculated.