A Bayesian Approach to Feature Selection in Classification Problems

A Master of Science in Mathematics by Maher Emarly entitled, “A Bayesian Approach to Feature Selection in Classification Problems”, submitted in July 2024. Thesis advisor is Dr. Ayman Alzaatreh. Soft copy is available (Thesis, Completion Certificate, Approval Signatures, and AUS Archives Consent Form).

Abstract

The exponential growth of data, as well as the widespread use of machine learning in daily life, demonstrate the importance of feature selection. Feature selection, defined as the process of identifying and selecting a subset of relevant features from a larger set of available features, is a crucial step in machine learning. The performance and efficiency of machine learning models are improved by focusing on the most informative features and eliminating unnecessary or redundant ones. Furthermore, model interpretability is enhanced, resulting in clearer insights and an actionable understanding of the results. The resulting models are more robust, less prone to noise, and can be efficiently trained and deployed, ultimately contributing to more effective and efficient data-driven decision-making processes. We propose a Bayesian approach using the relative belief ratio (RBR) as a filter method in this paper. The proposed method showed an excellent performance in binary and multiclass classification problems. In addition, the proposed method generates a strength value that can be used as an importance score for each feature. The numerical value of the strength of the RBR is used to rank the features. This method aims to discern the relative importance of features concerning a target variable and test for their significance. The proposed method’s performance is evaluated using both synthetic and real-world datasets, and it is compared to various popular filter methods.