A Multistage PCA–SMOTE Preprocessing Pipeline for Diabetes Prediction Using XGBoost and Hybrid Transformers

Authors

  • Amir Mohammad Rahimi Department of Computer Engineering, University of Seyed Jamal E Asadabadi, Kermanshah, Iran.
  • Hedieh Noorian * Department of Computer Engineering, Hamedan University of Technology, Hamedan, Iran. https://orcid.org/0009-0002-7852-3787

https://doi.org/10.22105/ahse.v3i2.62

Abstract

In this study, the Pima Indians Diabetes dataset (PIMA), comprising 768 clinical records with eight metabolic and hereditary attributes, is used to develop a binary diabetes prediction pipeline based on Principal Component Analysis (PCA) and the Synthetic Minority Oversampling Technique (SMOTE). PCA is applied to reduce multicollinearity and obtain orthogonal features, while SMOTE corrects the strong class imbalance, yielding a more stable and informative representation for learning algorithms. Within this preprocessed space, a wide spectrum of models is optimized by systematic GridSearch (GS)  based Hyperparameter (HP) tuning, ranging from Logistic Regression (LR), Support Vector Machine (SVM), and tree ensembles to deep neural networks and Transformer based architectures. The results show that, although Extreme Gradient Boosting (XGBoost) remains a strong traditional baseline, a hybrid Transformer combined with Gradient Boosted Decision Trees (GBDT) achieves the highest Accuracy (ACC), F1 score, and Receiver Operating Characteristic (ROC) Area Under Curve (AUC) on the Pima Indians dataset, demonstrating that rigorous data conditioning together with architecture aware Hyperparameter Optimization (HPO) can substantially enhance the reliability of medical diagnostic models.

Keywords:

Diabetes prediction, Pima Indians dataset, Principal component analysis, Synthetic minority oversampling technique, Extreme gradient boosting, Transformer, Hybrid gradient boosted decision trees

References

  1. [1] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization.
  2. [2] Kelleher, J. D., Namee, B. M., & D’Arcy, A. (2020). Fundamentals of machine learning for predictive data analytics, second edition: Algorithms, worked examples, and case studies. MIT Press. https://books.google.com/books?id=1Iv-DwAAQBAJ
  3. [3] Chawla, N. V, Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953
  4. [4] Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
  5. [5] Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
  6. [6] Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106. https://doi.org/10.1007/BF00116251
  7. [7] Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM Sigkdd International Conference on Knowledge Discovery and Data Mining (pp. 785–794). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2939672.2939785
  8. [8] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
  9. [9] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647
  10. [10] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … ., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems (pp. 5998–6008). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  11. [11] Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems (pp. 18932–18943). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2021/file/9d86d83f925f2149e9edb0ac3b49229c-Paper.pdf
  12. [12] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems (pp. 6638–6648). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2018/file/14491b756b3a51daac41c24863285549-Paper.pdf
  13. [13] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., … ., & Liu, T. Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems (pp. 3146–3154). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf
  14. [14] Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), 1189–1232. http://www.jstor.org/stable/2699986
  15. [15] Zhou, Z. H. (2025). Ensemble methods: Foundations and algorithms. Chapman and Hall/CRC. https://doi.org/10.1201/9781003587774

Downloads

Published

2026-06-15

Issue

Vol. 3 No. 2 (2026): Annals of Healthcare Systems Engineering

Section

Articles

How to Cite

Rahimi, A. M., & Noorian, H. (2026). A Multistage PCA–SMOTE Preprocessing Pipeline for Diabetes Prediction Using XGBoost and Hybrid Transformers. Annals of Healthcare Systems Engineering, 3(2), 102-112. https://doi.org/10.22105/ahse.v3i2.62

Similar Articles

11-20 of 30

You may also start an advanced similarity search for this article.