Robotic systems are increasingly integrated into healthcare to enhance precision, autonomy, and efficiency. This study provides a systematic review of decision-making systems and control architectures for autonomous and social robots in hospitals, with a specific focus on Markov Decision Processes (MDPs) and their variants. A systematic search of ScienceDirect, SpringerLink, and IEEE databases was conducted covering the last three decades. Inclusion criteria focused on studies describing action-selection or decision-making methods for autonomous or semiautonomous healthcare robots. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology was applied to identify, screen, and analyze relevant publications. The review identifies major application areas of MDP-based decision-making in healthcare: 1) surgical robotics, predominantly using Completely Observable Markov Decision Processes (COMDPs), 2) rehabilitation, where Partially Observable Markov Decision Processes (POMDPs) combined with deep reinforcement learning are common, 3) telemedicine, using COMDP frameworks with multi-agent coordination, 4) elderly care, leveraging POMDPs with human feedback, and 5) emergency response, applying multi-robot COMDPs enhanced with Bayesian updates. Emerging trends include hybrid COMDP–POMDP approaches and integration with machine learning for real-world deployment. MDP-based decision-making systems demonstrate strong potential to improve autonomy and adaptability in healthcare robotics. While COMDPs are effective in structured environments such as surgery, POMDPs are increasingly preferred for human-centered and uncertain contexts. Key challenges remain, including a lack of standardized benchmarks, limited clinical validation, and computational complexity. Addressing these gaps will be essential for the safe, efficient, and ethical deployment of robotic systems in healthcare.