belief networks, Hybrid approach, Hierarchical clustering.

The large intestine has sections responsible for absorbing water and salts and eliminating waste. The colon, where waste is collected before excretion, can develop diseases such as cancer, requiring a physician's examination to detect changes. Colonoscopy is a key diagnostic tool for ulcers, inflammation, polyps, and colon cancer. A colonoscope, a flexible instrument with a camera, allows a detailed examination of the colon's interior. However, diagnosing lesions from colonoscopy remains challenging due to diverse abnormalities and lack of precise measurements. This complexity challenges even specialized physicians. Image processing and AI can significantly improve diagnosis speed and accuracy. This study proposes a hybrid approach to improve lesion detection accuracy, combining techniques such as Bag of Words (BoW), Hierarchical Clustering, Deep Belief Networks (DBNs), and Hierarchical Autoencoders to address challenges like imprecise measurements and variability in colonoscopy images [1–4]. Key challenges include:

I. Unpredictable performance of the BoW system in large datasets.

II. Focus on specific local descriptors limiting generalizability.

III. Scalability and computational efficiency issues with complex, large datasets.

IV. Lack of resources for further research validation and expansion.

To overcome these, we used advanced computer science and medical imaging methods, including Hierarchical Autoencoders, Hierarchical Clustering, and DBN [4–7]. Hierarchical Autoencoders reduced data dimensionality and compressed complex features, improving BoVW performance with large datasets [8]. Hierarchical Clustering helped the model generalize across datasets, increasing versatility [9]. DBNs identified complex, non-linear patterns in extensive data, enhancing model accuracy and efficiency [10]. These methods, leveraging unsupervised learning and reducing model complexity, addressed scalability and efficiency challenges, allowing for training without labeled data. Ultimately, this research developed an automatic, accurate colonoscopy lesion detection system, improving diagnosis speed and effectiveness. The overall design of the method is illustrated in Fig. 1.

An overview of Computer-Aided Diagnosis (CAD) in colonoscopy from the perspective of endoscopists has been presented, specifically focusing on the evidence, limitations, and clinical applications [8]. Proposed a novel model for detecting and segmenting medical images based on spatio-temporal information, utilizing CAD methods to detect Pituitary Microadenomas (PM). This model aims to assist clinicians in the decision-making process, ultimately leading to improved lesion identification and diagnostic accuracy explored gastrointestinal endoscopy and the use of CAD in colonoscopy, addressing the inherent limitations of colonoscopies performed by humans. They utilized CAD techniques to enhance the quality of polyp detection and characterization, improving accuracy in identifying and diagnosing polyps [9]. The use of DBN and Capsule Networks (CapsNet) for the development of automated oral cancer detection systems has been proposed, as these models are capable of processing and identifying complex patterns in data [10]. have utilized deep autoencoders specifically designed for dimensionality reduction. Layer-wise pretraining of this model is carried out using an algorithm known as the DBN. Furthermore, we have concluded that deep autoencoders effectively reconstruct medical images with minimal quality loss. These findings suggest that these methods can be reliably employed in medical imaging, where maintaining image integrity is crucial for accurate diagnosis. Additionally, the study emphasizes the advantages of patch-based training, which reduces training time and model size, thereby improving efficiency and optimizing performance. They have concluded that deep autoencoders are effective tools for reconstructing medical images with minimal quality loss. These findings suggest that these methods can be reliably used in medical imaging, where maintaining image integrity is crucial for accurate diagnosis. Additionally, the study emphasizes the benefits of patch-based training, which reduces training time and decreases the model size, thereby improving efficiency and optimizing performance [11]. They have proposed a semi-supervised approach that utilizes a neural autoencoder to leverage unlabeled data. This autoencoder is trained on text documents to generate semantic features, subsequently used to improve document ranking and query refinement in later stages. Combining these features with BoW features significantly enhances performance in both tasks. In many recent studies, advanced machine learning and data analysis methods have been utilized to identify and predict diseases, particularly cancer and heart disease. These studies primarily focus on the application of various techniques such as Principal Component Analysis (PCA), Artificial Neural Networks (ANN), DBN, and Extreme Learning Machines (ELM). In a study conducted in 2022, PCA was used for feature generation, which was combined with ANN technology for cancer prediction. In this study, 28 attributes were provided to the models, and DBN was employed for unsupervised training and extracting relevant features. ELM and BP algorithms were used to optimize the DBN and perform supervised classification. Ultimately, cancer prediction models, including PCA-ANN and DBN-ELM-BP, were developed. This approach contributes to improved accuracy in cancer prediction [12]. In another study conducted in 2021, gene expression data were used for cancer classification. In this study, the data were first reduced in dimensionality using Probabilistic Principal Component Analysis (PPCA) and then classified using DBN after dimensionality reduction. This method improves the accuracy of identifying various types of cancer [13]. In a study investigating sepsis-induced inflammation in the large intestine, PCA and DBN were used for data analysis. The research demonstrated that combining in vivo and silico approaches can lead to more accurate results in simulating and diagnosing diseases. In 2025, DBN and CapsNet were proposed to develop automated oral cancer detection systems. These models can process and identify complex patterns in the data and enhance the accuracy of oral cancer detection [10]. Unsupervised and semi-supervised learning techniques are crucial in medical imaging due to the scarcity of labeled data. These methods allow for the simultaneous execution of supervised and unsupervised tasks [1]. Recent advancements focus on minimizing reconstruction errors and enhancing model invariance to data perturbations [14]. Additionally, Generative Adversarial Networks (GANs) have been widely used to generate high-quality synthetic data, such as image synthesis and augmenting real-world datasets. Using GANs in unsupervised learning can lead to the generation of fake data and privacy violations. This technology is associated with a lack of transparency in the data generation process, which can undermine trust in its results. Therefore, careful ethical and security management is essential to prevent negative consequences.

A hybrid unsupervised learning method has been employed for colon disease detection for the first time using unlabeled data. By leveraging multi-modal learning in natural language processing (BoW), pattern recognition with hierarchical clustering, deep learning with autoencoders and DBN, and computer vision, this approach reduces reliance on labeled data and significantly improves diagnostic accuracy.

The main steps of this study include data collection, preprocessing, feature extraction, dimensionality reduction, clustering, and classification. Additionally, all experiments are presented step by step to ensure methodological transparency. The methodology and code developed in this study are not specific to the current dataset and can be applied to any dataset with similar characteristics. This approach's flexibility ensures its adaptability to various domains and types of data. (For details on the methods, see Supplementary information.)

This study employed unsupervised clustering techniques for analyzing colonoscopy images and improving the diagnosis of gastrointestinal diseases. The results indicated that vocabulary size and distance metric significantly impacted clustering accuracy. The combination of a vocabulary size of 200 and 100 with hidden layers [100, 50] and [150, 75, 30] using the Chebyshev distance metric or a vocabulary size of 200 and hidden layers [100, 50] with the Cosine distance metric, yielded better performance. Additionally, using DBN significantly enhanced clustering accuracy. These findings demonstrate the potential of unsupervised learning in medical image processing, although there is a need for larger datasets and more precise hyperparameter tuning. Table 1 presents the performance metrics for various experimental configurations, including accuracy, precision, recall, specificity, and F1-score, based on different vocabulary sizes, clustering metrics, autoencoder layer settings, and DBN configurations.