Artificial intelligence for intraoperative surgical guidance in robotic

Robotic-assisted surgery (RAS) has improved dexterity, three-dimensional (3D) visualization, and precision. Artificial intelligence (AI) is increasingly being explored to further enhance intraoperative guidance, and the reliable translation of these methods into clinical practice requires both rigorous technical validation and, ultimately, clinical validation demonstrating patient benefit. A systematic review was conducted by searching PubMed and Scopus for publications on computer vision methods for intraoperative guidance in abdominal, thoracic, and pelvic robotic surgery, yielding 2,601 results, of which 95 were included after screening. This review characterizes current technical validation practices and examines the extent to which clinical validation is performed. Key shortcomings were identified, including limited clinical validation, inconsistent methodologies, poor metric transparency, and the use of evaluation metrics with unclear clinical relevance, collectively hindering interpretation, comparison, and clinical implementation. Future studies should adopt clinically relevant, standardized metrics, ensure transparent reporting, and integrate real-world validation to improve reproducibility and facilitate the safe clinical implementation of AI in robotic surgery.

Robotic assistance has transformed minimally invasive surgery by providing improved dexterity, three-dimensional (3D) visualization, and improved precision compared to conventional laparoscopy and thoracoscopy1,2. Building on these technical advances, artificial intelligence (AI) has emerged as a promising tool, and AI systems based on computer vision are increasingly explored for intraoperative guidance, with applications ranging from anatomical structure recognition and instrument tracking to surgical phase identification3.

Robotic platforms are particularly well-suited for these applications: their stabilized camera systems reduce the video variability seen in conventional video-assisted surgery, while stereoscopic vision and, in some systems, kinematic tracking further expand the possibilities for computer vision integration. For such computer vision systems to be safely and meaningfully integrated into operating room workflows, robust validation is essential.

In this context, a distinction can be made between technical and clinical validation. Technical validation refers to the assessment of algorithm performance on unseen data using established technical metrics4, and in the setting of intraoperative guidance, encompasses the measurement of accuracy in downstream tasks such as anatomy segmentation or surgical phase recognition. Clinical validation, by contrast, involves demonstrating with clinical data that a product is safe, effective, and fit for its intended clinical purpose within the target population and use environment. Crucially, this requires that AI outputs be associated with clinically meaningful outcomes, such as reductions in complications or improvements in intraoperative decision-making, rather than pixel- or frame-level accuracy alone, and that performance holds across representative users and clinical settings. Clinical validation, therefore, constitutes the evidence generation step that links AI deployment to patient benefit and acceptable risk. Although recent years have seen rapid scientific progress in intraoperative AI, validation practices have not evolved at the same pace.

Reported performance is often based on technical metrics whose clinical meaning is unclear, limiting their ability to inform clinical usefulness in daily practice5. Validation methods are often poorly aligned with the clinical tasks that these systems are intended to support. The usefulness of a performance metric depends on the specific surgical context in which the AI output will be used. However, commonly reported validation metrics, such as the Dice similarity coefficient (DSC), Hausdorff distance (HD), or sensitivity, may not capture clinically relevant aspects of algorithm performance when used without explicit reference to the underlying task6. As a result, algorithms can demonstrate strong performance in technical benchmarks while offering limited value during real-time intraoperative decision-making, contributing to a discrepancy between reported performance and clinical utility.

This gap between reported algorithmic performance and clinical utility highlights the need to systematically examine how intraoperative AI systems are currently validated. Rather than focusing solely on further technical development, greater attention is needed toward validation frameworks themselves, including how performance is measured, reported, and interpreted in relation to clinical use.

In recent years, multiple systematic reviews have been conducted on validation methods for robotic surgery. Kos et al. performed a systematic review of evaluation metrics for the use of AR in neurosurgical practice7. Younes et al. performed a review to find clinically relevant metrics for robotic surgery, without the use of any intraoperative AI methods8, whilst Kenig et al. conducted a systematic review of AI validation methods in surgery, not specifically for robotic surgery9.

Building on this previous work, the aim of this systematic review is to examine how computer vision-based AI methods for intraoperative guidance in robotic-assisted surgery (RAS) are currently validated across abdominal, thoracic, and pelvic procedures, with particular attention to validation approaches, the use and reporting of performance metrics, and the extent to which technical validation is complemented by clinical evaluation.

The systematic search resulted in a total of 2.601 results, of which 441 were duplicates. After the screening stage, a total of 1.941 records were excluded, leaving 219 records for full-text eligibility evaluation, including four manually added records. After the full-text evaluation, 95 studies were considered eligible and included in the systematic review, as shown in the PRISMA flow chart in Fig. 1.

T-A-P domain, thoracic, abdominal, and pelvic domain, CV computer vision, AI artificial intelligence.

The studies included in the systematic review were published between 2015 and 2025, and an increase in the yearly number of publications can be seen, as shown in Fig. 2. Eight different types of surgical procedures were included, namely, Urology, Upper gastrointestinal, Colorectal, Hepato-Pancreato-Biliary, Thoracic, General surgery, Gynecology, and unspecified. Unspecified includes all publications using surgical data that were confirmed to be abdominal, thoracic, or pelvic, but could not be assigned to a more specific category. Of these procedures, urology was the most prevalent surgical domain, with more than half of all publications, as shown in Fig. 3. More characteristics are shown per study in the Supplementary information.

Number of included robotic-assisted surgical AI publications per year, until the search date of July 28, 2025.

Upper GI Upper Gastrointestinal, HPB Hepato-Pancreato-Biliary.

An overview of the validation strategies identified in the included studies is presented in Fig. 4. Within this systematic review, the evaluation approaches reported were first assigned to the metric categories and subsequently assigned to the respective task types introduced in 4.3. This structure enables a task-specific comparison of how performance was assessed across the literature. For transparency, the frequency with which individual metrics and tasks appeared in the included studies is indicated in parentheses. Comprehensive definitions of all technical evaluation metrics, including their mathematical formulations where applicable, are provided separately in the Supplementary information.

Of the 95 included studies, 45 (47%) reported some form of clinical validation. Table 1 summarizes the number of studies that performed clinical or technical validation. With the exception of the 3D model overlay and registration task, clinical validation remains limited across all task categories, with fewer than 30% of studies incorporating a clinical evaluation.

Looking in more detail, the methodological nature of the studies differs greatly per task as well. For phase recognition and detection, all included studies are still in an algorithm development phase, using retrospective data collected from surgical video recordings. Almost all studies in the semantic segmentation task have the same methodological nature, only two studies reported a test during live surgery, on one and 10 live surgeries, respectively10,11. Similarly, the other tasks include a single study reporting a test during a single live surgery12. On the other hand, for the 3D model overlay and registration task, almost all models (88%) have already been developed and are deployed only during live surgery.

A closer examination of the types of clinical validation used further underscores this imbalance. As shown in Fig. 4, perioperative outcomes represent the most frequently reported form of clinical validation, appearing in 19 of the 34 clinically validated studies (55.8%). In these cases, the perioperative variables are typically compared between a cohort of patients undergoing surgery assisted by the proposed technology and a control group receiving standard care. In particular, this form of validation is exclusively observed in studies addressing the 3D model overlay and registration task. For all other tasks, survey-based assessments constitute the predominant approach to clinical validation.

Validation methods are categorized per metric class and reported per task. In brackets, the number of occurrences is shown. AUROC Area Under Curve - Receiver Operating Characteristics, FPR False Positive Rate, TPR True Positive Rate, FNR False Negative Rate, TNR True Negative Rate, ASSD Average Symmetric Surface Distance, AI Artificial Intelligence, RMSE Root Mean Squared Error, SSIM Structural Similarity Index Measure, PSNR Peak Signal-to-Noise Ratio, LPIPS Learned Perceptual Image Patch Similarity.

These surveys generally consist of questionnaires in which clinicians evaluate model performance, often using Likert-scale ratings13,14, or by comparing AI-generated annotations with human annotations15. In addition, Guan et al.12 used a survey to collect subjective assessments from surgeons on both the perceived clinical benefit of their scene reconstruction algorithm and its usability, including aspects such as convenience, speed, and precision. Similarly, Mezzina et al.16 used a survey-based evaluation to benchmark their surgical phase recognition algorithm against expert surgeons, thereby validating its performance.

Inspecting Fig. 4 reveals a notable lack of consensus among publications on the validation of AI methods for intraoperative use. This heterogeneity is particularly pronounced in detection tasks. Of the nine studies that perform detection tasks, only four employ mean average precision (mAP) as their primary evaluation metric. Even among these studies, there is variation in the way mAP is computed. Three studies report mAP at a fixed IoU threshold of 0.517,18,19, while Yoon et al. compute mAP by averaging IoU thresholds ranging from 0.5 to 0.95 in increments of 0.05, across all classes20. Similarly, in semantic segmentation tasks, in addition to DSC and IoU metrics, limited heterogeneity is present, where these are present in 24 and 22 of the included studies, respectively, the next highest used metrics are only used in eight and five studies.

Another important observation concerns the interpretation and reporting of evaluation procedures. In many cases, publications do not clearly specify how reported metrics are computed, limiting transparency and reproducibility. This ambiguity is particularly prevalent in surgical phase recognition tasks, where evaluation metrics may be calculated on a per-video basis and subsequently averaged across videos, or alternatively on a per-phase basis before aggregation. The work of Li et al. is among the few studies in this domain that explicitly describe the order of aggregation used to calculate the evaluation metrics21.

A similar lack of clarity is observed in the semantic segmentation literature. For example, Chen et al.22 report average values for mean IoU and DSC values, without specifying how these averages are computed or whether class-wise metrics are considered. Such ambiguities in metric reporting could complicate direct comparisons between methods and motivate the need for clearer evaluation protocols.

Furthermore, ambiguity is also present with respect to metric-specific averaging strategies. Evaluation metrics may be aggregated using micro, macro, or weighted averaging schemes; however, such details are often omitted. For example, Lou et al.23 report both the DSC and the F-measure for semantic segmentation, which should produce identical values under equivalent averaging strategies. The reported discrepancy between these metrics, combined with the absence of information on the averaging method, complicates reproducibility. In contrast, Nakawala et al.24 explicitly state that their evaluation metrics are computed using a prevalence-weighted macro average (PWMA), thus improving transparency.

An analysis of the evaluation metrics used to validate AI algorithms reveals that, for certain tasks, the diversity of metrics used is limited. In the semantic segmentation domain, Fig. 4 shows that the evaluation is predominantly based on DSC and IoU, both metrics based on overlap. Beyond these overlap measures, few alternative metric types have been consistently reported for semantic segmentation.

In total, only eight instances of distance-based metrics in seven publications are identified. The distance metric most commonly used is HD, or its 95th percentile variant, reported in five studies25,26,27,28,29. Two publications employ the Average Symmetric Surface Distance (ASSD)25,30, while only a single study reports the use of Mean Absolute Error (MAE)23.

This systematic review examined the validation methods used in 95 publications applying computer vision or AI to intraoperative guidance in robotic-assisted surgery of deformable organs. The findings reveal several recurring shortcomings that collectively pose substantial barriers to the clinical implementation of these systems, and are discussed below in relation to the broader literature.

Recent systematic reviews have examined closely related areas, including AI applications in robotic cancer surgery, broader overviews of AI in robotic surgery, surgical scene understanding in minimally invasive abdominal surgery, and AI-based technical skill assessment27,31,32,33. Although these reviews provide structured overviews of AI use cases and methodological characteristics across heterogeneous minimally invasive procedures, they do not examine in depth validation practices within the distinct context of robot-assisted surgery. Our review complements this literature by focusing exclusively on robot-assisted surgery in deformable organs and by systematically examining validation practices and translational readiness within this more narrowly defined setting, thereby highlighting challenges that are particularly relevant for clinical implementation in robot-assisted surgery.

A central finding of this review is the persistent imbalance between technical and clinical validation across the included literature. As outlined in the Introduction, technical validation assesses whether an algorithm performs its intended computational task with sufficient accuracy on unseen data, whereas clinical validation demonstrates that the system is safe, effective, and associated with meaningful patient benefit in real-world operative conditions. These are complementary validation methods: strong technical performance does not imply clinical utility, and the absence of clinical validation leaves a critical gap in the evidence base required for clinical adoption. Our findings illustrate this gap concretely: across all task categories except 3D model overlay and registration, fewer than 30% of studies incorporated any form of clinical validation. Where clinical validation was performed, it frequently relied on indirect proxies, such as operative time or blood loss, or subjective survey instruments, rather than outcomes directly attributable to AI-assisted intraoperative navigation. This pattern suggests that the field remains largely in an algorithm development phase, optimizing for technical benchmarks whose relationship to clinical outcomes has yet to be established.

This imbalance was consistent across surgical task categories, though its extent varied. With the exception of 3D model overlay and registration tasks, fewer than 25% of publications reported any form of clinical validation, underscoring the need for more clinically relevant metrics and validation efforts to ensure that these applications are ready for implementation in the operating room. Moreover, clinical validation is essential to generate the evidence required for evaluation and approval by regulatory bodies such as the Medical Device Regulation (MDR) and the Food and Drug Administration (FDA), and this must be taken into account during the development stages to prevent limitations towards clinical implementation.

The 3D model overlay and registration task represented a notable exception, with clinical validation performed in 23 of 34 studies. However, even here, validation was often based on perioperative outcomes such as operating time, blood loss, or complication rates34,35,36. Although informative, these metrics serve only as indirect proxies for model performance and are likely to be influenced by multiple confounders.

Among the studies that performed clinical validation on the other tasks, most relied on surveys or real-world implementation of algorithms. Such approaches help bridge the gap between technical development and clinical practice by providing feedback from clinicians during and after surgery, thereby facilitating algorithm refinement. While valuable as early-stage indicators, these methods fall short of the evidence standard required to establish patient benefit, and in isolation are insufficient to demonstrate that an AI system meets the threshold for clinical validation as defined by regulatory frameworks. In general, our findings emphasize the need for more robust clinical and technical validation to bridge the gap between algorithm development and real-world clinical implementation in robotic surgery.

The observed lack of standardization in the validation methods further complicates the assessment of progress in the field. Substantial variability exists not only in the choice of evaluation metrics but also in their definitions and computational procedures. Consequently, performance claims across studies are difficult to compare, even when similar tasks or datasets are addressed.

For instance, detection tasks frequently relied on mean average precision (mAP), yet their implementation varied considerably. Differences in IoU thresholds, ranging from 0.5 to 0.95, and the use of single versus multiple thresholds complicated mAP comparisons between studies. This exemplifies the broader methodological inconsistency and the absence of standardized benchmarking practices within the field.

Insufficient reporting of metric computation and aggregation strategies poses another obstacle to reproducibility. Ambiguities regarding whether metrics are calculated per frame, per phase, or per video, as well as how metric-specific results are averaged, limit the interpretability of reported performance. Such omissions hinder the ability to reproduce results and to assess whether observed improvements are statistically or clinically meaningful.

In phase recognition studies, technical validation was based primarily on standard classification metrics, including accuracy, precision, recall, and the F1 score. However, only a minority of studies specified the level at which metrics were computed, frame, phase, or video, even though performance estimates can vary substantially depending on averaging strategies and label hierarchies. This lack of transparency undermines reproducibility and complicates objective comparisons between publications.

Another finding of this review concerns the usefulness of commonly used validation metrics. In semantic segmentation studies, overlap-based metrics such as DSC and IoU were predominantly used, reflecting their established role in evaluating pixel-wise agreement. However, segmentation performance was rarely supplemented with distance-based metrics that better capture boundary errors, errors that are clinically meaningful in tasks such as instrument-tissue interaction or safety margin estimation. The limited use of distance-based metrics (11.8%) suggests that segmentation models may not be adequately evaluated for geometric precision, which is essential for intraoperative adoption.

Publications categorized under “other computer vision methods” frequently relied on similarity-based image reconstruction metrics, such as the SSIM, PSNR, and learned perceptual image patch similarity (LPIPS). While standard in image synthesis and depth estimation, these metrics primarily capture pixel-level fidelity; how closely the output resembles the ground truth, rather than clinical relevance. As with other task categories, clinical validation in this group was minimal, raising concerns about translation readiness, since no evidence demonstrates that model outputs are clinically meaningful or reliable in practice.

To address the lack of clinical relevance in technical validation metrics, surveys, such as the approach proposed by de Jong et al.37, can be used to obtain information on their clinical utility. In addition, Delphi studies could be used to establish expert consensus on clinically meaningful task-specific validation metrics in different surgical domains, complementing existing reporting frameworks such as DECIDE-AI, which emphasize early-stage clinical evaluation of AI-driven decision support systems38. This knowledge could then inform the development of new clinically relevant metrics tailored to specific surgical tasks, ultimately improving the readiness of computer vision models for real-world implementation.

To improve the clinical relevance, reproducibility, and comparability of computer vision methods in robotic surgery, future studies should adopt problem-aware metric selection, as proposed in the Metrics Reloaded framework39. The metrics should be chosen based on the characteristics of the task and the clinical objectives and complemented by measures that capture geometric or clinically significant errors, rather than relying solely on traditional metrics or standard perioperative outcome measures. Standardized reporting of metric computation and aggregation, combined with real-world clinical validation through surveys, expert feedback, or clinical deployment, will help bridge the gap between technical performance and clinical applicability. In addition, usability and workflow integration should be explicitly evaluated, for example, using validated instruments such as the System Usability Scale (SUS)40, as limited usability can reduce clinical adoption. The adoption of shared benchmarks and clear reporting standards will further enhance reproducibility and facilitate meaningful comparisons between studies.

This systematic review demonstrates that validation practices in surgical computer vision research remain highly variable, with inconsistent reporting and limited clinical relevance across studies. Although technical metrics such as accuracy, DSC, and mean average precision are widely used, their implementation lacks standardization, and their relationship to clinical outcomes is rarely established. More critically, clinical validation, in the sense of demonstrating patient-relevant benefit under real operative conditions, is performed in only a minority of publications, leaving a substantial gap between experimental development and clinical readiness. To advance the field, future research should prioritize transparent reporting of metric definitions and evaluation protocols, while integrating clinically meaningful validation frameworks early in the development process. Establishing consensus guidelines for validation and reporting will be essential to improve reproducibility, support regulatory approval, and ultimately enable safe and effective adoption of AI-driven tools in surgery.

A systematic search was conducted in the PubMed and Scopus databases on 28 July 2025. The results of both databases were combined to serve as a starting point for the review. The search process adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines41. The systematic review was not pre-registered in any register, and no protocol was prepared. Search strings were developed to include three principal concepts: robotic assisted surgery, computer vision for intraoperative use, and surgical procedures involving deformable organs. Consequently, only thoracic, abdominal, and pelvic surgeries were included. The search strings for each of the databases can be found in the Supplementary information. The inclusion criteria restricted the search to articles written in English and published within the last 10 years (i.e., from 2015 onward). To ensure completeness, relevant articles not retrieved by database searches were manually added.

After removal of duplicates, the following exclusion criteria were applied: (1) non-original research articles (e.g., reviews, perspectives); (2) no robot-assisted surgery; (3) incorrect domain of RAS: not thoracic, abdominal, pelvic; (4) articles using animal data; (5) articles using simulated or phantom data; (6) articles using ex vivo data; (7) articles that do not use computer vision; (8) articles not aimed at intraoperative use of computer vision.

The first author conducted an initial screening of all retrieved records based on title and abstract to assess relevance. This selection process was validated by two authors (AS and YK), and any disagreements were solved by coming to a consensus after discussing the individual articles. In the next phase, two authors (TK and GK) checked the remaining articles for full text eligibility. Any discrepancies in the exclusion decision were resolved again by consensus.

This review aimed primarily to characterize technical validation practices in the included literature, while additionally documenting the extent and nature of clinical validation performed.

For each of the included articles, the following data were extracted: year of publication, surgical procedure, computer vision method, and, if present, information on both technical and clinical validation methods. Validation is considered the process of evaluating the performance of a trained model or algorithm to ensure the quality of the model. Technical validation typically includes quantitative metrics that indicate algorithm performance, whereas clinical validation methods consist of patient outcomes, intraoperative and postoperative variables, as well as human evaluation of algorithms, based on surveys and/or quantitative assessment tools. Validation can be performed on retrospective data and prospective unseen data.

Intraoperative guidance methods were classified into one of five categories: (1) phase recognition; (2) semantic segmentation; (3) object or action detection; (4) 3D model overlay or registration; and (5) other computer vision methods, including but not limited to tasks like depth estimation and surgical scene reconstruction.

Surgical phase recognition involves automatically identifying which step or phase of a surgical procedure is being performed and supports real-time workflow analysis. Semantic segmentation assigns a label to every single pixel in an image, allowing for precise delineation of, for example, anatomical structures and surgical instruments. It is fundamental for tasks such as navigation and safety measures when avoiding critical structures. Object detection identifies and localizes anatomical structures or surgical instruments, differentiating from semantic segmentation by detecting the object using, for example, a bounding box instead of assigning a class to every pixel. Action detection extends this to detect what the surgeon is doing, enabling context-aware assistance, skill assessment, and automatic event logging.

Tasks like 3D model overlay or registration, aligning preoperative 3D models (e.g., CT-based), with the live surgical feed to create an overlay in AR. This improves spatial awareness, facilitates navigation, and can help in localizing deep or hidden structures. Depth estimation and surgical scene reconstruction provide a 3D understanding from the two-dimensional (2D) video stream, supporting AR overlays by estimating per-pixel distance or generating a 3D representation of the surgical scene, respectively.

During data extraction, five categories of validation methods were identified, of which four concern technical validation and one clinical validation: (1) classification metrics; (2) overlap-based metrics; (3) distance-based metrics; (4) other technical metrics; and (5) clinical validation methods.

Classification metrics (1) evaluate how well a model assigns data to discrete categories, with common examples being accuracy, precision, and recall. Overlap based metrics (2) are used to assess how closely a predicted region matches the ground truth, mainly in segmentation tasks, with metrics such as the DSC or Intersection-over-Union (IoU). They quantify the degree of overlap between the prediction and the ground truth. Distance based metrics (3) measure the spatial difference between the prediction and the ground truth. Metrics such as the HD or Root Mean Squared Error (RMSE) capture how far points or surfaces deviate from each other and are mostly used when geometric precision is important. The other technical metrics (4) contain all quantifiable metrics that do not fall into any of the other categories; examples include inference time or image similarity metrics such as the peak signal-to-noise ratio (PSNR) or the structural similarity index measure (SSIM).

For clinical validation methods (5), a distinction can be made between qualitative and quantitative approaches. Qualitative methods encompass surveys and real-world data tests. Surveys capture clinician feedback on the usability and performance of intraoperative surgical guidance algorithms, while real-world data tests involve algorithm deployment during live surgery, with surgeons rating performance against predefined criteria. Quantitative methods, by contrast, rely on measurable operative outcomes as clinical validation metrics.

The datasets generated and/or analyzed during the current study are not publicly available due to containing manually extracted information from published articles and are part of planned research, but are available from the corresponding author on reasonable request.

Rivero-Moreno, Y. et al. Robotic surgery: a comprehensive review of the literature and current trends. Cureus https://doi.org/10.7759/cureus.42370 (2023).

Reddy, K. et al. Advancements in robotic surgery: a comprehensive overview of current utilizations and upcoming frontiers. Cureus 15, e50415 (2023).

PubMed PubMed Central Google Scholar

Knudsen, J. E., Ghaffar, U., Ma, R. & Hung, A. J. Clinical applications of artificial intelligence in robotic surgery. J. Robot. Surg. 18, 102 (2024).

Park, S. H., Choi, J. & Byeon, J.-S. Key principles of clinical validation, device approval, and insurance coverage decisions of artificial intelligence. Korean J. Radiol. 22, 442 (2021).

Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 1–9 (2019).

Reinke, A. et al. Understanding metric-related pitfalls in image analysis validation. Nat. Methods 21, 182–194 (2024).

Kos, T. M., Colombo, E., Bartels, L. W., Robe, P. A. & van Doormaal, T. P. Evaluation metrics for augmented reality in neurosurgical preoperative planning, surgical navigation, and surgical treatment guidance: a systematic review. Oper. Neurosurg. 26, 491–501 (2024).

Younes, M. M. et al. What are clinically relevant performance metrics in robotic surgery? a systematic review of the literature. J. Robot. Surg. 17, 335–350 (2023).

Kenig, N., Monton Echeverria, J. & Muntaner Vives, A. Artificial intelligence in surgery: a systematic review of use and validation. J. Clin. Med. 13, 7108 (2024).

Hofman, J. et al. First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery. Healthc. Technol. Lett. 11, 33–39 (2024).

De Backer, P. et al. Improving augmented reality through deep learning: real-time instrument delineation in robotic renal surgery. Eur. Urol. 84, 86–91 (2023).

Guan, B. et al. Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction. Comput. Biol. Med. 165, 107472 (2023).

Amparore, D. et al. 3d anatomical digital twins: New generation virtual models to navigate robotic partial nephrectomy. BJUI Compass 6, e453 (2025).

Nakamura, T. et al. Precise highlighting of the pancreas by semantic segmentation during robot-assisted gastrectomy: visual assistance with artificial intelligence for surgeons. Gastric Cancer 27, 869–875 (2024).

Chen, W. et al. Pioneering AI-guided fluorescence-like navigation in urological surgery: real-time ureter segmentation during robot-assisted radical cystectomy using convolutional neural network. J. Robot. Surg. 19, 188 (2025).

Mezzina, M. et al. Surgeons versus computer vision: a comparative analysis on surgical phase recognition capabilities. Int. J. Comput. Assist. Radiol. Surg. 20, 1283–1291 (2025).

Ayobi, N. et al. Pixel-wise recognition for holistic surgical scene understanding. Med. Image Anal. 106, 103726 (2025).

Valderrama, N. et al. Towards holistic surgical scene understanding. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, (eds Wang, L., Dou, Q., Fletcher, P. T., Speidel, S. & Li, S.) 442–452 (Springer Nature Switzerland, 2022).

Frey, S. et al. Optimizing intraoperative AI: evaluation of yolov8 for real-time recognition of robotic and laparoscopic instruments. J. Robot. Surg. 19, 131 (2025).

Yoon, J., Lee, J., Park, S., Hyung, W. J. & Choi, M.-K. Semi-supervised learning for instrument detection with a class-imbalanced dataset. in Interpretable and Annotation-Efficient Learning for Medical Image Computing (eds Wang, L., Dou, Q., Fletcher, P. T., Speidel, S. & Li, S.) 266–276 (Springer International Publishing, Cham, 2020).

Li, Y. et al. Benchmarking and enhancing surgical phase recognition models for robot-assisted esophagectomy. in Medical Imaging 2025: Image-Guided Procedures, Robotic Interventions, and Modeling (eds Rettmann, M. E. & Siewerdsen, J. H.) 13408, 134080T. International Society for Optics and Photonics https://doi.org/10.1117/12.3040084 (SPIE, 2025).

Chen, J., Li, M., Han, H., Zhao, Z. & Chen, X. Surgnet: Self-supervised pretraining with semantic consistency for vessel and instrument segmentation in surgical images. IEEE Trans. Med. Imaging 43, 1513–1525 (2023).

Lou, A., Tawfik, K., Yao, X., Liu, Z. & Noble, J. Min-max similarity: a contrastive semi-supervised deep learning network for surgical tools segmentation. IEEE Trans. Med. Imaging 42, 2832–2841 (2023).

Nakawala, H. et al. "Deep-onto” network for surgical workflow and context recognition. Int. J. Comput. Assist. Radiol. Surg. 14, 685–696 (2019).

Xu, J. et al. Personalizing federated instrument segmentation with visual trait priors in robotic surgery. IEEE Transact. Biomed. Eng. 72, 1886–1896 (2025).

de Nijs, J. V. et al. Automated surgical urethral length estimation for robot-assisted radical prostatectomy. In Proc. IEEE International Symposium on Biomedical Imaging (ISBI) 1–5 https://doi.org/10.1109/ISBI56570.2024.10635376 (IEEE, 2024).

Den Boer, R. et al. Computer-aided anatomy recognition in intrathoracic and abdominal surgery: a systematic review. Surg. Endosc. 36, 8737–8752 (2022).

Bakker, A. F. et al. Estimating surgical urethral length on intraoperative robot-assisted prostatectomy images using artificial intelligence anatomy recognition. J. Endourol. 38, 690–696 (2024).

Matasyoh, N. M., Mathis-Ullrich, F. & Zeineldin, R. A. Samsurg: surgical instrument segmentation in robotic surgeries using vision foundation model. IEEE Access 99, 1–1 (2024).

de Jong, R. L. P. D. et al. Benchmarking pretrained attention-based models for real-time recognition in robot-assisted esophagectomy. in Medical Imaging 2025: Image-Guided Procedures, Robotic Interventions, and Modeling, (eds Rettmann, M. E. & Siewerdsen, J. H.). International Society for Optics and Photonics https://doi.org/10.1117/12.3045187 (SPIE, 2025).

Leszczyńska, A., Obuchowicz, R., Strzelecki, M. & Seweryn, M. The integration of artificial intelligence into robotic cancer surgery: a systematic review. J. Clin. Med. 14, 6181 (2025).

Iftikhar, M., Saqib, M., Zareen, M. & Mumtaz, H. Artificial intelligence: revolutionizing robotic surgery: review. Ann. Med. Surg. 86, 5401–5409 (2024).

Boal, M. W. E. et al. Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br. J. Surg. 111, https://doi.org/10.1093/bjs/znad331 (2023).

Zeng, S. et al. Holographic reconstruction technology used for intraoperative real-time navigation in robot-assisted partial nephrectomy in patients with renal tumors: a single center study. Transl. Androl. Urol. 10, 3386 (2021).

Porpiglia, F. et al. Augmented-reality robot-assisted radical prostatectomy using hyper-accuracy three-dimensional reconstruction (ha 3d™) technology: a radiological and pathological study. BJU Int. 123, 834–845 (2019).

Shi, X. et al. Artificial intelligence based augmented reality navigation in minimally invasive partial nephrectomy. Urology 199, 20–26 (2025).

de Jong, R. et al. Do segmentation metrics reflect clinical reality? A surgeon-centered evaluation in robot-assisted minimally invasive esophagectomy. Surg. Endosc. 20, 277–290 (2025).

Vasey, B. et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat. Med. 28, 924–933 (2022).

Article CAS PubMed Google Scholar

Maier-Hein, L. et al. Metrics reloaded: recommendations for image analysis validation. Nat. Methods 21, 195–212 (2024).

Lewis, J. R. The system usability scale: past, present, and future. Int. J. Hum. Comput. Interact. 34, 577–590 (2018).

Rethlefsen, M. L. et al. Prisma-s: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst. Rev. 10, 1–19 (2021).

This research was supported by an unrestricted grant from Stichting Hanarth Fonds, The Netherlands.

Department of Cardiothoracic Surgery, University Medical Center Utrecht, Utrecht, The Netherlands

Thomas Korsten, Massimo Mariani & Amir H. Sadeghi

Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands

Thomas Korsten, Ruisheng Su & Yasmina Al Khalil

Department of Surgery, University Medical Center Utrecht, Utrecht, The Netherlands

Gino M. Kuiper & Jelle P. Ruurda

Department of Cardiothoracic Surgery, Radboud University Medical Center, Nijmegen, The Netherlands

Search author on:PubMed Google Scholar

T.K.: Conceptualization, Methodology—Title abstract screening, full text screening, data extraction, Writing—original draft proposal, review and editing. G.K.: Conceptualization, Methodology—full text screening, Writing original draft proposal, review and editing. R.S.: Conceptualization, Writing—review and editing, Supervision. J.R.: Conceptualization, Writing—review and editing. A.V.: Conceptualization, Writing—review and editing. M.M.: Conceptualization, Writing review and editing. Y.K.: Conceptualization, Methodology—Title abstract screening, Writing—review and editing, Supervision. A.S.: Conceptualization, Methodology Title abstract screening, Writing—review and editing, Supervision.

Correspondence to Thomas Korsten.

J.P.R. is on the Editorial Board of NPJ Digital Surgery. J.P.R. was not involved in the journal’s review of, or decisions related to, this manuscript. The rest of the authors have no relevant financial or non-financial interests to disclose.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Korsten, T., Kuiper, G.M., Su, R. et al. Artificial intelligence for intraoperative surgical guidance in robotic-assisted ventral cavity surgery: a systematic review on the current state of validation methods. npj Digit. Surg. 1, 13 (2026). https://doi.org/10.1038/s44484-026-00013-7

Version of record: 06 July 2026

DOI: https://doi.org/10.1038/s44484-026-00013-7