AI, Robotics, and Markov Decision Processes: Enhancing Precision and Autonomy inHealthcare Systems

Abstract

Purpose: Robotic systems are increasingly integrated into healthcare to enhance precision, autonomy, and efficiency. This study
provides a systematic review of decision-making systems and control architectures for autonomous and social robots in hospitals,
with a specific focus on Markov Decision Processes (MDPs) and their variants.
Methods: A systematic search of ScienceDirect, SpringerLink, and IEEE databases was conducted covering the last three
decades. Inclusion criteria focused on studies describing action-selection or decision-making methods for autonomous or semi-
autonomous healthcare robots. The PRISMA methodology was applied to identify, screen, and analyze relevant publications.
Results: The review identifies major application areas of MDP-based decision-making in healthcare: (i) surgical robotics, pre-
dominantly using Completely Observable MDPs (COMDPs); (ii) rehabilitation, where Partially Observable MDPs (POMDPs)
combined with deep reinforcement learning are common; (iii) telemedicine, using COMDP frameworks with multi-agent coordi-
nation; (iv) elderly care, leveraging POMDPs with human feedback; and (v) emergency response, applying multi-robot COMDPs
enhanced with Bayesian updates. Emerging trends include hybrid COMDP–POMDP approaches and integration with machine
learning for real-world deployment.
Conclusion: MDP-based decision-making systems demonstrate strong potential to improve autonomy and adaptability in health-
care robotics. While COMDPs are effective in structured environments such as surgery, POMDPs are increasingly preferred for
human-centered and uncertain contexts. Key challenges remain, including lack of standardized benchmarks, limited clinical val-
idation, and computational complexity. Addressing these gaps will be essential for the safe, efficient, and ethical deployment of
robotic systems in healthcare.