Deep Unsupervised Learning for Colonoscopy Lesion Representation and Clustering

  • Vahid Bayrami RadDepartment of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran.
  • Mahdi MazinaniDepartment of Electrical and Electronic Engineering, ShQ.C., Islamic Azad University, Shahr-e Qods, Iran.
  • Mitra MirzarezaeeDepartment of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran.

Abstract

Detecting lesions in colonoscopy remains a significant challenge due to the complexity of images and the limitations of labeled datasets. This study proposes a combined unsupervised learning approach for clustering colonoscopy images, which utilizes the Bag of Words (BoW) model for extracting local features, hierarchical autoencoders for dimensionality reduction, hierarchical clustering for effective data grouping, and Deep Belief Networks (DBNs) for identifying nonlinear patterns. This method significantly enhances lesion detection, especially when labeled data is scarce. Experimental results show clustering accuracy ranging from 91.97% to 100%, with a strong silhouette score above 0.80. Performance improves with larger vocabulary sizes and distance metrics such as Euclidean, Chebyshev, and Cosine. Integration of hierarchical autoencoders and DBNs enhances scalability and computational efficiency under limited labeled data conditions. The proposed method improves clustering quality without requiring large labeled datasets. This unsupervised hybrid framework is applied for colon disease detection using unlabeled data and integrates techniques from pattern recognition, deep learning, computer vision, and natural language processing, reducing reliance on labeled data while improving diagnostic accuracy.

Keywords:Small datasetsMedical image classificationUnsupervised learningHierarchical autoencodersDeep

belief networks, Hybrid approach, Hierarchical clustering.

1. Introduction

The large intestine has sections responsible for absorbing water and salts and eliminating waste. The colon, where waste is collected before excretion, can develop diseases such as cancer, requiring a physician's examination to detect changes. Colonoscopy is a key diagnostic tool for ulcers, inflammation, polyps, and colon cancer. A colonoscope, a flexible instrument with a camera, allows a detailed examination of the colon's interior. However, diagnosing lesions from colonoscopy remains challenging due to diverse abnormalities and lack of precise measurements. This complexity challenges even specialized physicians. Image processing and AI can significantly improve diagnosis speed and accuracy. This study proposes a hybrid approach to improve lesion detection accuracy, combining techniques such as Bag of Words (BoW), Hierarchical Clustering, Deep Belief Networks (DBNs), and Hierarchical Autoencoders to address challenges like imprecise measurements and variability in colonoscopy images [1–4]. Key challenges include:

I. Unpredictable performance of the BoW system in large datasets.

II. Focus on specific local descriptors limiting generalizability.

III. Scalability and computational efficiency issues with complex, large datasets.

IV. Lack of resources for further research validation and expansion.

To overcome these, we used advanced computer science and medical imaging methods, including Hierarchical Autoencoders, Hierarchical Clustering, and DBN [4–7]. Hierarchical Autoencoders reduced data dimensionality and compressed complex features, improving BoVW performance with large datasets [8]. Hierarchical Clustering helped the model generalize across datasets, increasing versatility [9]. DBNs identified complex, non-linear patterns in extensive data, enhancing model accuracy and efficiency [10]. These methods, leveraging unsupervised learning and reducing model complexity, addressed scalability and efficiency challenges, allowing for training without labeled data. Ultimately, this research developed an automatic, accurate colonoscopy lesion detection system, improving diagnosis speed and effectiveness. The overall design of the method is illustrated in Fig. 1.

This figure is shown in the PDF version of the article.

Figure 1. The proposed framework.

2. Related Works

An overview of Computer-Aided Diagnosis (CAD) in colonoscopy from the perspective of endoscopists has been presented, specifically focusing on the evidence, limitations, and clinical applications [8]. Proposed a novel model for detecting and segmenting medical images based on spatio-temporal information, utilizing CAD methods to detect Pituitary Microadenomas (PM). This model aims to assist clinicians in the decision-making process, ultimately leading to improved lesion identification and diagnostic accuracy explored gastrointestinal endoscopy and the use of CAD in colonoscopy, addressing the inherent limitations of colonoscopies performed by humans. They utilized CAD techniques to enhance the quality of polyp detection and characterization, improving accuracy in identifying and diagnosing polyps [9]. The use of DBN and Capsule Networks (CapsNet) for the development of automated oral cancer detection systems has been proposed, as these models are capable of processing and identifying complex patterns in data [10]. have utilized deep autoencoders specifically designed for dimensionality reduction. Layer-wise pretraining of this model is carried out using an algorithm known as the DBN. Furthermore, we have concluded that deep autoencoders effectively reconstruct medical images with minimal quality loss. These findings suggest that these methods can be reliably employed in medical imaging, where maintaining image integrity is crucial for accurate diagnosis. Additionally, the study emphasizes the advantages of patch-based training, which reduces training time and model size, thereby improving efficiency and optimizing performance. They have concluded that deep autoencoders are effective tools for reconstructing medical images with minimal quality loss. These findings suggest that these methods can be reliably used in medical imaging, where maintaining image integrity is crucial for accurate diagnosis. Additionally, the study emphasizes the benefits of patch-based training, which reduces training time and decreases the model size, thereby improving efficiency and optimizing performance [11]. They have proposed a semi-supervised approach that utilizes a neural autoencoder to leverage unlabeled data. This autoencoder is trained on text documents to generate semantic features, subsequently used to improve document ranking and query refinement in later stages. Combining these features with BoW features significantly enhances performance in both tasks. In many recent studies, advanced machine learning and data analysis methods have been utilized to identify and predict diseases, particularly cancer and heart disease. These studies primarily focus on the application of various techniques such as Principal Component Analysis (PCA), Artificial Neural Networks (ANN), DBN, and Extreme Learning Machines (ELM). In a study conducted in 2022, PCA was used for feature generation, which was combined with ANN technology for cancer prediction. In this study, 28 attributes were provided to the models, and DBN was employed for unsupervised training and extracting relevant features. ELM and BP algorithms were used to optimize the DBN and perform supervised classification. Ultimately, cancer prediction models, including PCA-ANN and DBN-ELM-BP, were developed. This approach contributes to improved accuracy in cancer prediction [12]. In another study conducted in 2021, gene expression data were used for cancer classification. In this study, the data were first reduced in dimensionality using Probabilistic Principal Component Analysis (PPCA) and then classified using DBN after dimensionality reduction. This method improves the accuracy of identifying various types of cancer [13]. In a study investigating sepsis-induced inflammation in the large intestine, PCA and DBN were used for data analysis. The research demonstrated that combining in vivo and silico approaches can lead to more accurate results in simulating and diagnosing diseases. In 2025, DBN and CapsNet were proposed to develop automated oral cancer detection systems. These models can process and identify complex patterns in the data and enhance the accuracy of oral cancer detection [10]. Unsupervised and semi-supervised learning techniques are crucial in medical imaging due to the scarcity of labeled data. These methods allow for the simultaneous execution of supervised and unsupervised tasks [1]. Recent advancements focus on minimizing reconstruction errors and enhancing model invariance to data perturbations [14]. Additionally, Generative Adversarial Networks (GANs) have been widely used to generate high-quality synthetic data, such as image synthesis and augmenting real-world datasets. Using GANs in unsupervised learning can lead to the generation of fake data and privacy violations. This technology is associated with a lack of transparency in the data generation process, which can undermine trust in its results. Therefore, careful ethical and security management is essential to prevent negative consequences.

3. The Proposed General Approach

A hybrid unsupervised learning method has been employed for colon disease detection for the first time using unlabeled data. By leveraging multi-modal learning in natural language processing (BoW), pattern recognition with hierarchical clustering, deep learning with autoencoders and DBN, and computer vision, this approach reduces reliance on labeled data and significantly improves diagnostic accuracy.

3.1 Fusion of Visual Features with Bag of Words, Autoencoder, and Deep Belief Network

The dataset consisted of 480 images, with 384 images used for training and 96 for testing. Split The data was 80% for training and 20% for testing. For feature extraction, vocabularies of sizes 50, 100, and 200 were used, and for DBN modeling, hidden layer configurations [100, 50] and [150, 75, 30] were used. The number of images was reduced to 480 to manage the time and resources required for precise evaluation of hierarchical clustering metrics, as these metrics require more detailed processing. This reduction in the number of images enabled more efficient and accurate experiments to achieve higher accuracy using fewer images, which is one of the key challenges in deep learning.

3.2 Advanced Feature Extraction from Colonoscopy Images Using Bag of Words

This section discusses the BoW method for feature extraction from colonoscopy images. This method extracts features such as texture, shape, and pixel intensity, capturing high-level patterns (e.g., shapes and colors) and low-level patterns (e.g., textures and more complex structures) [15], [16]. After feature extraction, the data undergoes stages like dimensionality reduction to optimize its size while retaining essential information. The processed features are then fed into classification algorithms for disease detection. Due to its precise feature extraction, the BoW method is a practical approach for disease diagnosis in colonoscopy images. The BoW workflow is illustrated in Fig. 2.

This figure is shown in the PDF version of the article.

Figure 2. BOW feature workflow: Texture, shape, and pixel intensity.

3.3 Using Hierarchical Autoencoders for Extracting Complex Features

The auto-encoder deep ANNs create a precise map of the input data and minimize the display error. This results in a new feature space of data in the central layer of the auto-encoder network, which has smaller dimensions than the input data. The image dimensions decrease in the auto-encoder neural network that includes an encoder and a decoder and allows for mapping the data in the feature space and vice versa. Fig. 3.a displays the structure of the auto-encoder of the ANN [17].

As shown, an auto-encoder is composed of two neural networks, one for encoding the input to the lower dimension and the other for decoding it to the main dimension. So, we have

Encoder: h(t) = fθ(x(n)), (1) decoder: r(n) = gθ(h(t)), (2) where x represents an n-dimension input vector, h represents an encoded vector with the dimension of t calculated from the encoder neural network, and r is a decoded vector with the dimension of n calculated from the decoder neural network. By training the auto-encoder, we aimed to achieve r = x with the minimum error. So, h represents the input at a lower dimension [18]. We employed this idea to reduce the feature space in the proposed framework. To address the challenges of deep neural networks, each layer of the network is trained separately and then the layers are joined together. The training process of the network layers is as follows. We trained the neural network using three layers for deep representation learning. The first layer consists of 1000 neurons, the second layer has 500 neurons, and the third layer has 150 neurons, followed by a final layer with 80 neurons. In addition, the data is reconstructed in the 80- dimensional feature space (Fig. 3.b).

This figure is shown in the PDF version of the article.

Figure 3. a. Auto Encoder, and b. Autoencoder training.

3.4 Hierarchical Clustering for Extracting Features and Improving Classification

Using hierarchical clustering to organize features extracted by an autoencoder and passed into a DBN enhances the model's ability to comprehend complex data, thereby improving classification accuracy. This approach reduces the complexity of the data while preserving its relevant features, making the classification process more efficient. Furthermore, utilizing appropriate clustering metrics such as Euclidean, Cosine, Chebyshev, and Cityblock enables the model to identify relationships between features, further enhancing classification performance more accurately.

The Silhouette index was used to evaluate clustering quality. This index assesses how well the data is grouped within clusters and how well the clusters are separated from each other. Considering intra-cluster [19], [20] and inter-cluster distances aids in selecting the most suitable clustering metrics for optimizing the process (For details of the Silhouette method, please refer to the supplementary methods).

3.5 Deep Belief Networks for Feature Learning and Classification

DBNs consist of layers of Restricted Boltzmann Machines (RBMs) and employ unsupervised learning to extract complex features from data [21]. Training in DBNs occurs layer by layer, with a Softmax layer used for classification in the final stage. DBNs are widely applied in tasks such as classification and processing complex data, including images and audio [22], [23]. This study used two hidden layer configurations: One with two hidden layers containing 100 and 50 units and the other with three hidden layers containing 150, 75, and 30 units. Using Autoencoders instead of RBMs adds flexibility to the model, enabling it to learn more complex, non-linear features.

4. Materials and Methods

The main steps of this study include data collection, preprocessing, feature extraction, dimensionality reduction, clustering, and classification. Additionally, all experiments are presented step by step to ensure methodological transparency. The methodology and code developed in this study are not specific to the current dataset and can be applied to any dataset with similar characteristics. This approach's flexibility ensures its adaptability to various domains and types of data. (For details on the methods, see Supplementary information.)

4.1 Experimental Setup

In this study, the number of experiments conducted was determined by the combinations of three primary parameters: vocabulary sizes (with three different values: 50, 100, and 200), distance metrics for clustering (with four options: 'Euclidean,' 'Cosine,' 'Cityblock', and 'Chebychev'), and Autoencoder layer size settings (with two configurations). Thus, the total number of experiments was calculated as 24, derived from the product of the number of options for each parameter (3 × 4 × 2).

5. Conclusion

This study employed unsupervised clustering techniques for analyzing colonoscopy images and improving the diagnosis of gastrointestinal diseases. The results indicated that vocabulary size and distance metric significantly impacted clustering accuracy. The combination of a vocabulary size of 200 and 100 with hidden layers [100, 50] and [150, 75, 30] using the Chebyshev distance metric or a vocabulary size of 200 and hidden layers [100, 50] with the Cosine distance metric, yielded better performance. Additionally, using DBN significantly enhanced clustering accuracy. These findings demonstrate the potential of unsupervised learning in medical image processing, although there is a need for larger datasets and more precise hyperparameter tuning. Table 1 presents the performance metrics for various experimental configurations, including accuracy, precision, recall, specificity, and F1-score, based on different vocabulary sizes, clustering metrics, autoencoder layer settings, and DBN configurations.

Table 1. Presents the performance metrics for different experiment configurations.

This table is shown in the PDF version of the article.

References

  1. [1] Chapelle, O., Scholkopf, B., & Zien Eds., A. (2009). Semi-supervised learning [Book reviews]. IEEE Transactions on Neural Networks, 20(3), 542. https://doi.org/10.1109/TNN.2009.2015974
  2. [2] Celebi, M. E., Iyatomi, H., Schaefer, G., & Stoecker, W. V. (2009). Lesion border detection in dermoscopy images. Computerized Medical Imaging and Graphics, 33(2), 148–153. https://doi.org/10.1016/j.compmedimag.2008.11.002
  3. [3] Friedman, R. J., Rigel, D. S., & Kopf, A. W. (1985). Early detection of malignant melanoma: The role of physician examination and self-examination of the skin. CA: A Cancer Journal for Clinicians, 35(3), 130–151. https://doi.org/10.3322/canjclin.35.3.130
  4. [4] Barata, C., Ruela, M., Mendonça, T., & Marques, J. S. (2014). A bag-of-features approach for the classification of melanomas in dermoscopy images: The role of color and texture descriptors. In Computer Vision Techniques for the Diagnosis of Skin Cancer (pp. 49–69). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-39608-3_3
  5. [5] Stanley, R. J., Stoecker, W. V, & Moss, R. H. (2007). A relative color approach to color discrimination for malignant melanoma detection in dermoscopy images. Skin Research and Technology, 13(1), 62–72. https://doi.org/10.1111/j.1600-0846.2007.00192.x
  6. [6] Raju, A. S. N., Gatla, R. K., Sucharitha, G., Rajababu, M., Hussain, S. J., Lakshmi, C. B. N., … ., & Abass, K. S. (2026). ColoXAI-RecomNet: Explainable recommender framework for colorectal cancer classification using integrated CNN ensemble and LIME interpretability. Journal of Imaging Informatics in Medicine. https://doi.org/10.1007/s10278-026-02007-w
  7. [7] Dong, C., & Du, G. (2026). Polyp segmentation for colonoscopy images via Hierarchical Interworking Decoding. Biomedical Signal Processing and Control, 112, 108737. https://doi.org/10.1016/j.bspc.2025.108737
  8. [8] Mori, Y., Kudo, S., Berzin, T. M., Misawa, M., & Takeda, K. (2017). Computer-aided diagnosis for colonoscopy. Endoscopy, 49(08), 813–819. https://doi.org/10.1055/s-0043-109430
  9. [9] Nishida, N., Yamakawa, M., Shiina, T., & Kudo, M. (2019). Current status and perspectives for computer-aided ultrasonic diagnosis of liver lesions using deep learning technology. Hepatology International, 13(4), 416–421. https://doi.org/10.1007/s12072-019-09937-4
  10. [10] GV, H. M. (2025). A novel approach using CapsNet and deep belief network for detection and identification of oral leukopenia. https://doi.org/10.48550/arXiv.2501.00876
  11. [11] Juliet, S. (2020). Deep medical image reconstruction with autoencoders using deep boltzmann machine training. EAI Endorsed Transactions on Pervasive Health & Technology, 6(24). https://doi.org/10.4108/eai.24-9-2020.166360
  12. [12] Qi, H., Xie, S., Chen, Y., Wang, C., Wang, T., Sun, B., & Sun, M. (2022). Prediction methods of common cancers in China using PCA-ANN and DBN-ELM-BP. IEEE Access, 10, 113397–113409. https://ieeexplore.ieee.org/abstract/document/9924251/
  13. [13] Menaga, D., & Revathi, S. (2021). Probabilistic principal component analysis (PPCA) based dimensionality reduction and deep learning for cancer classification. Intelligent computing and applications (pp. 353–368). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-15-5566-4_31%0A%0A
  14. [14] Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647
  15. [15] Pratiwi, M., & Arsyah, U. I. (2025). Analysis of shape and texture identifying and detecting apple fruit. Innovative: Journal of Social Science Research, 5(1), 3826–3837. https://doi.org/10.31004/innovative.v5i1.17727
  16. [16] Guo, Z., Wang, Y., Shen, Y., Zhu, X., Nemoto, D., Takayanagi, D., … ., & Togashi, K. (2017). Automatic polyp recognition from colonoscopy images based on bag of visual words. 2017 IEEE 8th International Conference on Awareness Science and Technology (ICAST) (pp. 18–22). IEEE. https://doi.org/10.1109/ICAwST.2017.8256441
  17. [17] Li, H., Bai, L., Gao, W., Xie, J., & Huang, L. (2024). Many-objective coevolutionary learning algorithm with extreme learning machine auto-encoder for ensemble classifier of feedforward neural networks. Expert Systems with Applications, 246, 123186. https://doi.org/10.1016/j.eswa.2024.123186
  18. [18] Abdellatif, A., Mubarak, H., Abdellatef, H., Kanesan, J., Abdelltif, Y., Chow, C. O., … ., & Kendall, G. (2024). Computational detection and interpretation of heart disease based on conditional variational auto-encoder and stacked ensemble-learning framework. Biomedical Signal Processing and Control, 88, 105644. https://doi.org/10.1016/j.bspc.2023.105644
  19. [19] Gupta, R., Mathur, J., & Garg, V. (2025). A criteria-based climate classification approach considering clustering and building thermal performance: Case of India. Building and environment, 270, 112512. https://doi.org/10.1016/j.buildenv.2024.112512
  20. [20] Yao, Y., Peng, B., Qin, T., Gu, Y., Ling, N., & Lei, J. (2025). Hypergraph contrastive learning for large-scale hyperspectral image clustering. IEEE Transactions on Circuits and Systems for Video Technology. https://ieeexplore.ieee.org/abstract/document/10857448/
  21. [21] Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554. https://ieeexplore.ieee.org/abstract/document/6796673/
  22. [22] Jain, N., Kamalraj, R., & Agrawal, A. (2024). Classifying clinically important cancers using deep belief networks. 2024 International Conference on Optimization Computing and Wireless Communication (ICOCWC) (pp. 1–6). IEEE. https://ieeexplore.ieee.org/abstract/document/10470868/
  23. [23] Boopathi Raja, G. (2025). Deep learning techniques for gene selection and cancer classification: a detailed review. In Computational intelligence for genomics data (pp. 79–96). Academic Press. https://doi.org/10.1016/B978-0-443-30080-6.00015-8