A Comparative Study of Machine Learning Models for Two-Tier Android Malware Classification with Dynamic Behavioral Analysis

The rapid proliferation of android malware has emerged as a critical threat to global cybersecurity. This study comparatively evaluates five supervised classification algorithms, including Random Forest (RF), Support Vector Machines (SVM) with RBF kernel, Artificial Neural Networks (ANNs), Naive Bayes and the novel TabNet model. The CCCS-CIC-AndMal-2020 dataset is used that comprises 200,000 malware samples categorized into 14 classes and 191 families, with features dynamically extracted during application execution in emulated environments. The predictive performance was assessed at two hierarchical classification approaches, distinguishing between broad malware categories and family-level attribution. To address class imbalance, oversampling techniques were considered. Precision, recall, and F1-score metrics, complemented by confusion matrices and ROC curves, were utilized for comprehensive evaluation. Statistical significance of differences among classifiers was determined using Friedman and Nemenyi post-hoc tests. Experimental results showed that RF, SVM, and ANNs consistently outperform other models across most metrics. This research provides a robust analytical framework for developing intelligent malware detection systems, contributing significantly to enhanced mobile cybersecurity.