In this study, the challenges of classifying imbalanced datasets—particularly in medical applications such as diabetes detection—are investigated. The research evaluates the impact of various resampling techniques, including both oversampling and undersampling methods, on the performance of classification models. By comprehensively combining four oversampling and four undersampling approaches within a Multi-Criteria Decision-Making (MCDM) framework for criterion weighting and ranking, the study proposes an integrated and practical framework for selecting optimal resampling strategies for diabetes detection using the large Behavioral Risk Factor Surveillance System (BRFSS) dataset. Machine learning algorithms such as XGBoost and the Support Vector Machine (SVM) were employed and their performance assessed under different resampling regimes. Results show that, on average, sensitivity improved across all resampling methods, with a mean increase of 87.32%, an improvement that was most pronounced for XGBoost. The F1-score likewise exhibited substantial gains across all methods, with SVM contributing a relatively larger share to the F1-score improvements. Although Area Under Curve (AUC) showed little change, the findings indicate a clear enhancement in the models’ ability to detect the minority class (individuals with diabetes). To identify the best resampling approaches, a MCDM procedure was applied, using the Analytic Hierarchy Process (AHP) for criterion weighting and Multi Attributive Ideal-Real Comparative Analysis (MAIRCA) for ranking and prioritizing the classification and resampling methods. In addition to the multi-criteria ranking, an unsupervised clustering analysis based on the K means algorithm was conducted on the resampling–classifier combinations to further explore similarities and differences in their overall performance profiles. The optimal number of clusters was determined using the silhouette coefficient, leading to a partition that revealed distinct groups of methods characterized by different trade offs among accuracy, precision, sensitivity, F1 score, AUC, and computational cost. The clustering results were consistent with the MAIRCA ranking, with high ranked alternatives forming well separated, high quality clusters, while poorly performing methods were grouped into low performance clusters.