AI, Robotics, and Markov Decision Processes: Enhancing Precision and Autonomy in Healthcare

2.1 Factors Driving Robotic Integration in Healthcare

Several factors are driving the adoption of robotics and AI in healthcare environments:

I. Technological advancements: the increasing sophistication of AI and robotic systems has broadened their application to complex medical tasks, ranging from minimally invasive surgery to advanced imaging and diagnostic support [9].

II. Improved efficiency and value creation: healthcare institutions face growing pressure to enhance efficiency while reducing costs. Robotic systems can automate repetitive and time-intensive tasks, such as drug dispensing and logistics, thereby allowing healthcare staff to concentrate on patient-centered activities [12].

III. Personalized care delivery: intelligent robots enable more personalized healthcare by monitoring patientspecific parameters (e.g., vital signs) and adapting interventions or treatment plans to individual needs [2].

As shown in Fig. 2, research output in this field has increased sharply since 2016, underscoring the growing recognition of AI and robotics as transformative forces in healthcare.

4.1 Innovation Emerges: Artificial Intelligence as a Partner for Enhanced Safety

Given the inherent risks of surgery and the possibility of human error, exploring novel approaches to enhance safety has become imperative. AI is increasingly recognized as a critical enabler of surgical innovation, with applications extending far beyond robotic assistance. Instead, AI supports the entire continuum of surgical interventions, spanning the pre-operative, intra-operative, and post-operative phases. Table 1 summarizes selected applications of AI in each surgical stage. Pre-operatively, AI algorithms analyze medical scans, optimize imaging sequences, and assist in accurate diagnosis and risk stratification. During surgery, intra-operative tools based on AI can track surgical instruments, support malignant tissue identification, and predict complications such as excessive bleeding. In the post-operative stage, AI is leveraged to monitor patient recovery, predict recurrence, and support tailored therapeutic strategies. These integrated applications highlight the potential of AI to reduce complications and improve patient outcomes across the surgical pathway.

4.2 Recent Advancements

5.1 Enhanced Diagnostics and Treatment Planning

AI-powered image analysis: AI algorithms can analyze medical images with exceptional accuracy, assisting doctors in earlier detection of diseases such as cancer. This allows for more timely intervention and potentially improves treatment outcomes.

Personalized medicine: by analyzing vast amounts of patient data (medical history, genetics, etc.), AI can recommend personalized treatment plans. This approach tailors therapies to the individual’s specific needs and characteristics, potentially leading to more effective treatment.

5.2 Improved Surgical Precision and Efficiency

Robotic-assisted surgery has been widely adopted to enhance surgical precision and reduce invasiveness. Robotic arms provide surgeons with improved dexterity, tremor filtration, and high-definition visualization of the operative field. These features translate into more precise surgical maneuvers, smaller incisions, and reduced intraoperative blood loss. Clinical studies have shown that patients undergoing robotic-assisted procedures often experience shorter hospital stays and faster recovery compared to conventional surgery [1], [12].

5.3 Increased Efficiency and Reduced Costs

AI integration in healthcare contributes substantially to efficiency gains by automating administrative and repetitive tasks. For example, algorithms can manage appointment scheduling, update electronic health records, and generate clinical reports, thereby reducing the workload of healthcare staff and enabling them to devote more time to patient-centered care [8].

Another key advantage lies in the prevention of medication errors. AI-based clinical decision support systems are capable of verifying prescriptions, identifying potential drug interactions, and ensuring correct dosage administration. Such systems not only improve patient safety but also lower the economic burden of preventable adverse drug events, which represent a significant share of healthcare expenditures [9].

5.4 Improved Patient Care and Outcomes

Beyond cost and efficiency, AI also contributes to improved clinical outcomes by enabling earlier, more accurate, and more personalized care.

I. AI-enabled diagnostic systems support the early detection of diseases, thereby increasing the chances of successful treatment. Early intervention is particularly critical in conditions such as cancer and cardiovascular diseases, where prognosis strongly depends on timely diagnosis [7].

II. Remote patient monitoring has been enhanced through AI-powered wearables and sensors, which continuously track vital signs and provide real-time alerts in case of deterioration. This approach is particularly valuable for patients with chronic illnesses, allowing for timely interventions and more effective disease management [2].

III. AI facilitates the transition toward personalized and preventive medicine. By analyzing patient-specific risk factors and large-scale clinical data, AI systems can help identify high-risk individuals and recommend tailored interventions. This predictive capacity fosters a shift from reactive to proactive healthcare delivery, thereby improving long-term outcomes [9].

6.1 Search Strategy

To compile the dataset, we searched three major electronic databases widely used in engineering and healthcare research: ScienceDirect, Springer Link, and IEEE Xplore. These databases were selected due to their extensive coverage of peer-reviewed publications and global relevance.

Our search strategy relied on predefined keywords and Boolean queries related to AI, robotics, decision-making, and Human-Robot Interaction (HRI). Table 2 summarizes the queries applied across databases and the number of initial results retrieved.

6.2 Eligibility Criteria

The inclusion criteria were defined as follows: 1) description of the action-selection or decision-making method for autonomous behavior, 2) involvement of humans in the decision-making process, 3) specification of whether the method was implemented on real robots or only tested in simulation, and 4) description of the healthcare application domain.

Exclusion criteria consisted of: 1) papers not written in English, 2) studies focusing exclusively on fully teleoperated robots without autonomous features.

6.3 Screening and Selection Process

The initial search produced 355 records. After removing duplicates and non-English articles, 290 papers remained for screening. Of these, 42 were excluded based on the eligibility criteria. A total of 257 studies met the inclusion criteria and were retained for full analysis.

The overall process of identification, screening, eligibility assessment, and inclusion is illustrated in the PRISMA flow diagram (Fig. 6).

7.1 Patient Care

Decision-making systems have played a pivotal role in enhancing patient care within hospitals and clinical settings. Applications include automated medication management, patient monitoring, and personalized care delivery. Such systems leverage AI-driven models to optimize drug administration and reduce errors, thereby elevating the quality and safety of healthcare services [8], [9].

7.2 Surgical Robotics

Autonomous decision-making and control architectures have transformed the field of surgical robotics. Robots equipped with advanced perception and planning systems can assist surgeons in complex procedures, minimize invasiveness, and improve patient outcomes [1], [12]. These innovations demonstrate the potential of decision-making frameworks, including MDPs, to guide surgical actions under uncertainty.

7.3 Rehabilitation

Robotic systems in rehabilitation exploit decision-making algorithms to personalize therapy and accelerate recovery. Intelligent exoskeletons and assistive devices adapt to patients' specific needs, enabling tailored rehabilitation programs that improve mobility and functionality after injury or surgery [2].

7.4 Telemedicine

In the era of telemedicine, robots powered by decision-making systems facilitate remote consultations, diagnostics, and treatment delivery. Such systems enhance access to healthcare in underserved or remote regions, enabling physicians to extend expertise beyond traditional clinical boundaries [14].

7.5 Elderly Care

Autonomous robots are increasingly employed to assist older adults with daily activities, medication reminders, and social interaction. Decision-making systems enable these robots to adapt to user preferences and promote independent living, ultimately improving well-being and reducing caregiver burden [4].

7.6 Emergency Response

Robots equipped with advanced control and decision-making architectures play critical roles in disaster management, hazardous environment exploration, and search-and-rescue operations. These applications illustrate how robotic autonomy can mitigate risks, safeguard lives, and support emergency responders in time-sensitive scenarios [7].

By categorizing and synthesizing contributions across these six domains, our review provides a holistic perspective on decision-making and control systems in healthcare robotics. These systems not only augment patient care but also redefine the delivery of healthcare services, marking a new era of technological advancement.

7.7 The History of Robotics

A significant milestone in the realm of surgery unfolded approximately a quarter-century ago when the pioneering use of a robot marked its debut in the surgical theater. This groundbreaking event featured the PUMA 200, a creation of Westinghouse Electric based in Pittsburgh, PA. Its inaugural role was in the precise placement of needles during a CT-guided brain biopsy—a watershed moment that ignited the journey of robotic surgery.

Since that pivotal juncture, the trajectory of robotic surgery has been nothing short of remarkable, evolving by leaps and bounds. This transformation can be attributed to the manifold advantages that robotics bestows, virtues often absent in traditional surgical techniques. Notably, robots bring to the table attributes such as unwavering stability, pinpoint accuracy, seamless integration with cutting-edge imaging technologies, expanded ranges of motion, and the prospect of telesurgery, among a plethora of other benefits tailored to specific surgical domains.

In the wake of the advent of robotic surgery, a wave of advancements has surged through the field. To harness the full potential of surgical robotics, it is imperative to embark on a journey through the annals of history, allowing us to chart a course towards an innovative future. This retrospective expedition entails an exploration of the milestones and accomplishments within various surgical domains, encompassing otolaryngological, neurosurgical, gynecological, cardiothoracic, gastric, urologic, orthopedic, endoscopic, and miniature surgical specialties. By gazing into the past, we gain the insights needed to navigate the horizon of future developments that await our scrutiny.

In the pages that follow, we will embark on an illuminating journey through the annals of robotic history within the hospital environment. Our exploration will traverse various domains, each with its unique tale of robotic innovation and application. We will delve into the chronicles of assistant robots, surgical robots, and a myriad of other robotic companions that have found their purpose in healthcare settings. As we proceed, we will unveil the evolution of these remarkable machines and the profound impact they have had on the revolutionization of healthcare.

7.8 History of Robotics Timeline:

I. Unimate (1961): the Unimate, created by George Devol and Joseph Engelberger, is considered the first industrial robot. It was used for tasks like die casting and spot welding.

II. PUMA (1978): the Programmable Universal Machine for Assembly (PUMA) was one of the first robots designed for more versatile assembly tasks.

III. Robotic Arms in Space (1980s): robots like the Canadarm were used on space shuttles for tasks like satellite deployment and repair.

IV. Roomba (2002): the Roomba, a robotic vacuum cleaner developed by iRobot, became a commercial success, popularizing the use of robots in homes.

V. Boston Dynamics (2010s): Boston Dynamics introduced a series of advanced robots like Spot, Atlas, and Handle, showcasing agility and mobility.

VI. Emerging Applications (2020s): robotics continues to expand into various fields, including agriculture, logistics, and disaster response.

7.9 Healthcare

Over the past two decades, autonomous social robots have been increasingly integrated into healthcare, with notable applications targeting children, older adults, and therapeutic support for caregivers. A particularly active research domain has been the use of robots to assist children with autism. Early studies investigated the role of social robots in gaming and educational contexts [21], while subsequent work proposed robot architectures with predefined behaviors based on observed perceptions to engage autistic children in therapeutic sessions [22].

Beyond child-centered applications, efforts have also focused on supporting caregivers during therapy. Hiolle et al. [23] introduced a baby-like robot capable of adapting its emotional responses to evaluate caregiver sensitivity. In parallel, biologically inspired approaches have been proposed, such as a hormonal system for adaptive behavior that integrates external stimuli, valence assessment, and physiological functions [24].

Other researchers have explored adaptive frameworks for chronic disease management. For instance, Lewis and Canamero [25] designed the Robin (NAO) robot to help children learn how to manage diabetes, while subsequent work modeled stress responses in robotic systems to emulate physiological mechanisms and deficits [25]. In elderly care, robots such as Mario have demonstrated the ability to assist individuals with dementia by providing cognitive support and companionship [26]. Similarly, Mini has been applied in cognitive stimulation therapies, employing a fully autonomous decision-making system based on reinforcement learning [27].

Recent advancements have broadened the scope of healthcare applications. The Pepper robot has been used autonomously to collect patient information [28], while the iCub robot has demonstrated personalized adaptation of its behavior during HRI to maximize user pleasantness [29]. Social robots have also been employed to alleviate children's pain during medical procedures [30], promote healthy diets, and deliver personalized therapies for individuals with dementia. These examples illustrate the increasing specialization of robotic applications in healthcare, emphasizing the role of decision-making systems in tailoring social interaction and therapeutic interventions.

7.10 Humanoid Robots for Children

QTrobot, developed by LuxAI S.A., represents one of the most advanced humanoid social robots designed for education and therapy. With its expressive facial display and emotionally adaptive interaction, QTrobot has been widely used to support children with Autism Spectrum Disorder (ASD), promoting social skills, communication, and engagement [31]. Its design exemplifies the potential of decision-making systems in tailoring interaction to the specific needs of vulnerable groups.

7.11 Pepper: Redefining Human-Robot Interaction

Pepper, created by Softbank Robotics, has become a benchmark in HRI research. Deployed in diverse domains ranging from education and healthcare to customer service, Pepper demonstrates how social robots can adaptively engage users in emotionally meaningful ways. In healthcare, Pepper has been employed to autonomously collect patient information, thereby enhancing patient experiences and supporting medical staff [28]. Its decision-making systems rely on adaptive dialogue and context-sensitive behavior, which position Pepper as a valuable platform for personalized care.

7.12 Mario

Mario is a socially assistive robot developed for elderly care, particularly for individuals with dementia. Designed as part of the MARIO project, the robot integrates decision-making capabilities to provide companionship, cognitive stimulation, and support for independent living [26]. Mario demonstrates how decision-making systems can be adapted to long-term HRI, addressing the growing demand for assistive technologies in geriatric care.

7.13 Social Robots for Alleviating Children's Pain

In pediatric healthcare, social robots have been developed to reduce distress and pain during medical procedures. Moerman et al. [30] introduced a robot capable of autonomously selecting distracting behaviors, such as storytelling or playful interaction, to provide comfort for children undergoing treatment. This example highlights how decision-making systems can be used not only for functional tasks but also for emotional support in sensitive healthcare scenarios.

7.14 Comparative Analysis of Decision-Making Architectures

Our review reveals distinct trends in the application of MDPs and their variants across healthcare domains. In surgery, robotic systems such as the Da Vinci platform rely primarily on Completely Observable Markov Decision Processes (COMDPs) combined with heuristic planning [1]. Rehabilitation exoskeletons often adopt Partially Observable Markov Decision Processes (POMDPs) integrated with deep reinforcement learning, although most studies remain limited to simulation [11]. Telemedicine robots frequently use COMDP frameworks with multi-agent coordination [8], while social robots in elderly care (e.g., Pepper) increasingly adopt POMDPs that incorporate human feedback [30]. Emergency response robots leverage multi-robot COMDP frameworks enhanced with Bayesian updates, as described in foundational reinforcement learning studies Sutton and Barto [10].

8.1 Treatment Planning and Personalized Medicine

MDPs have been widely applied to optimize treatment planning, particularly for chronic and life-threatening diseases. By modeling alternative treatment options, patient responses, and uncertain outcomes, MDPs enable the development of personalized care strategies that adapt dynamically to changes in patient health states [32], [33].

8.2 Medication Adherence and Chronic Disease Management

Adherence to prescribed therapies is a critical determinant of treatment success. MDPs can represent patient adherence behavior by incorporating probabilities of missed doses, side effects, and preferences, thereby guiding interventions to improve compliance [34]. Similarly, in chronic disease management, MDPs help design optimal long-term care strategies by considering lifestyle modifications, disease progression, and treatment effectiveness [35].

8.3 Resource Allocation and Patient Flow Optimization

Healthcare systems face constant challenges in allocating limited resources. MDP-based models have been used to optimize staff scheduling, bed allocation, and medical equipment distribution, aiming to maximize patient outcomes while minimizing costs [36]. Applications also extend to patient flow management in hospitals and emergency departments, where MDPs inform strategies to reduce waiting times and improve service delivery [37].

8.4 Clinical Trial Design

Another promising application is in clinical trial design. MDPs allow adaptive trial protocols by modeling patient cohort selection, treatment arm allocation, and stopping rules, ultimately improving the efficiency and ethical conduct of clinical studies [38].

8.5 Telemedicine and Decision Support Systems

With the expansion of telemedicine, MDPs have been applied to remote monitoring and dynamic treatment adjustments based on real-time patient data. They also serve as the foundation for intelligent clinical decision support systems, enabling healthcare providers to make evidence-based and optimized decisions regarding diagnosis and care pathways [39].

8.6 Public Health and Policy Applications

At the population level, MDPs are employed to inform vaccination strategies, screening programs, and public health interventions by modeling disease spread and intervention outcomes [37]. These models guide policymakers in designing cost-effective health strategies that balance resource constraints with societal benefits.

9.1 Completely Observable Markov Decision Processes

In the classical setting, the environment state is assumed to be fully observable, and the agent can make decisions with complete knowledge of s ∈ S. COMDPs have been widely applied in structured environments such as robotic surgery and hospital logistics, where the system state can be reliably monitored [37].

9.2 Partially Observable Markov Decision Processes

In many real-world healthcare scenarios, the state of the system cannot be directly observed. POMDPs extend the MDP framework by incorporating a set of observations O and an observation function Z(o|s, a) that describes the likelihood of perceiving observation o given hidden state s and action a. Decision-making thus relies on maintaining a belief state, which is a probability distribution over possible states [40].

Solving POMDPs is significantly more challenging due to the high-dimensional belief space. Classical approaches include value iteration in belief space, point-based methods, and Monte Carlo Tree Search [32], [41]. These methods approximate optimal policies by iteratively refining decision boundaries in the belief space.

9.3 Partially Observable Markov Decision Processes Augmentation

Recent work has explored augmenting POMDP frameworks with additional computational tools to improve performance. For instance, deep neural networks have been integrated into belief-state estimation and policy learning, enabling scalable solutions for high-dimensional state spaces [42]. Transfer learning has also been employed to adapt POMDP-based decision-making models across related tasks, enhancing generalization in healthcare and robotic applications [43].

9.4 Applications

MDPs and their variants have found numerous applications in robotics and healthcare, including surgical planning, rehabilitation, elderly care, and resource optimization. In particular, POMDP-based models have been employed to handle uncertainty in HRI and patient monitoring, where hidden states (e.g., patient discomfort, fatigue) must be inferred from limited observations [37].

Overall, MDPs and their extensions provide a versatile and powerful set of tools for decision-making under uncertainty. Their integration with machine learning, particularly deep reinforcement learning, continues to expand their applicability in healthcare robotics and beyond.

14.1 Applications in Healthcare

In healthcare, HRI has been applied to a variety of contexts, including surgical assistance, rehabilitation, social companionship, and pediatric support [4]. Robots can help children with autism through socially assistive interactions, provide companionship to elderly individuals with dementia, and assist medical staff with repetitive tasks such as logistics and cleaning.

14.2 Human-Robot Interaction during the COVID-19 Pandemic

The COVID-19 pandemic accelerated the adoption of HRI in hospitals. Robots were deployed for contactless delivery of food and medication, cleaning and disinfection of rooms, and telepresence consultations, reducing infection risk for healthcare workers [7]. Other robots were used for remote vital sign monitoring, enabling continuous patient care without direct physical contact. These applications demonstrate how AI-powered HRI can improve safety and resilience in healthcare systems during crises.

14.3 Link with Human-Computer Interaction

Human-Computer Interaction (HCI) shares many principles with HRI, focusing on the design of intuitive, user-centered interfaces. In healthcare robotics, HCI contributes to developing ergonomic and accessible control systems that allow medical professionals to collaborate effectively with robots. By integrating HCI principles, HRI research ensures that robotic systems are not only functional but also acceptable and usable in real-world healthcare settings [47].

16.1 Data and Bias

AI and robotic systems depend heavily on large-scale, high-quality datasets. Limitations in data availability or representativeness can compromise performance. Moreover, privacy regulations may restrict data sharing, slowing innovation [9].

16.2 Technical Challenges

A persistent challenge is the lack of explainability in complex deep learning models, often considered “black boxes”. Without interpretability, it is difficult for clinicians to trust AI-driven recommendations. Interoperability with existing hospital infrastructures also remains a barrier to large-scale deployment [7].

16.3 Ethical and Social Concerns

Automation raises fears of job displacement, particularly in repetitive tasks. At the same time, unequal access to advanced robotic technologies may exacerbate healthcare inequities. Ensuring affordability, accessibility, and retraining programs for healthcare workers is essential for sustainable adoption [13].

16.4 Safety and Liability

Robotic systems deployed in surgical or diagnostic contexts must be rigorously validated to avoid malfunction risks. Furthermore, liability for errors involving autonomous systems remains a contested issue. Clear regulatory guidelines and safety certifications are required before widespread adoption [46].