A Multistage PCA–SMOTE Preprocessing Pipeline for Diabetes Prediction Using XGBoost and Hybrid Transformers

Abstract

In this study, the Pima Indians Diabetes dataset (PIMA), comprising 768 clinical records with eight metabolic and hereditary attributes, is used to develop a binary diabetes prediction pipeline based on Principal Component Analysis (PCA) and the Synthetic Minority Oversampling Technique (SMOTE). PCA is applied to reduce multicollinearity and obtain orthogonal features, while SMOTE corrects the strong class imbalance, yielding a more stable and informative representation for learning algorithms. Within this preprocessed space, a wide spectrum of models is optimized by systematic GridSearch (GS)  based Hyperparameter (HP) tuning, ranging from Logistic Regression (LR), Support Vector Machine (SVM), and tree ensembles to deep neural networks and Transformer based architectures. The results show that, although Extreme Gradient Boosting (XGBoost) remains a strong traditional baseline, a hybrid Transformer combined with Gradient Boosted Decision Trees (GBDT) achieves the highest Accuracy (ACC), F1 score, and Receiver Operating Characteristic (ROC) Area Under Curve (AUC) on the Pima Indians dataset, demonstrating that rigorous data conditioning together with architecture aware Hyperparameter Optimization (HPO) can substantially enhance the reliability of medical diagnostic models.