Skip to main content
  • Oral presentation
  • Open access
  • Published:

Augmented reality: 3D image-guided surgery


Over the last three decades, surgical practice has undergone a significant change with a move towards minimally invasive surgery (MIS) as the standard of care [1]. Although this has brought with it significant benefits, problems have also been associated with the advent of MIS. Perhaps the most substantial limitation associated with MIS is the loss of haptic feedback; this deficit is at its most extreme in robot-assisted surgery, where at present such feedback is lost entirely [2].

The image-enhanced operating environment looks to mitigate for the loss of haptic feedback by providing the surgeon with visual cues to the subsurface anatomy. The use of intraoperative image guidance can be divided into that used for operative planning, to facilitate the rapid identification of critical anatomical structures, for example, and that used for task execution, an example of which is tumour resection [2]. These two steps have very different requirements, with the first needing a large amount of anatomical information to be displayed without the need to account for tissue deformation or accurate registration, while the second requires less information to be displayed, but with much greater spatial accuracy.


The solution proposed herein, the image-enhanced operating environment, utilises two different imaging modalities and plays on their respective strengths to meet the differing needs of the two outlined steps of planning and execution. The platform has been built around the index procedure of robot-assisted partial nephrectomy, although its potential application extends well beyond this scope.

The first step of operative planning utilises 3D reconstructions of preoperative cross-sectional imaging manipulated via a tablet-based interface [3]. This information was displayed to the surgeon both on the tablet and within the da Vinci console using the stereo TilePro™ function (Intuitive Surgical, Sunnyvale, CA).

The second step of execution utilises optically registered intraoperative ultrasound. Using a live imaging modality mitigates for the problems of deformation often faced when trying to use preoperative imaging for high precision guidance. The ultrasound data is used to create freehand 3D reconstructions which are overlaid onto the operative view [4].


To date, over 60 cases have been undertaken using the tablet-based planning component of the image enhanced operating environment. Over the course of this series, a subjective benefit has been demonstrated through the analysis of prospectively-collected questionnaire results. In addition, the platform has demonstrated objective safety, with no detrimental effects observed on outcome parameters. The use of registered ultrasound has been demonstrated in vivo [5], with results of an ex vivo study demonstrating potential efficacy awaited.


Replacing haptic feedback with visual cues to subsurface anatomy offers a number of potential direct and indirect benefits to the patient, including improved resection quality and a reduction in positive surgical margins. In addition to these direct benefits, the use of an image-enhanced operating environment could potentially influence case selection, where surgeons are prepared to take on cases with more challenging anatomy via a minimally invasive approach, because of the improved understanding they are given by the image guidance platform.


  1. Nicolau S, Soler L, Mutter D, Marescaux J: Augmented reality in laparoscopic surgical oncology. Surg Oncol. 2011, 20: 189-201. 10.1016/j.suronc.2011.07.002.

    Article  PubMed  Google Scholar 

  2. Hughes-Hallett A, Pratt P, Mayer E, Martin S, Darzi A, Vale J, Marcus H, Cundy T: Augmented Reality Partial Nephrectomy: Examining the Current Status and Future Perspectives. Urology. 2014, 83: 266-273. 10.1016/j.urology.2013.08.049.

    Article  PubMed  Google Scholar 

  3. Hughes-Hallett A, Pratt P, Mayer E, Martin S, Darzi A, Vale J: Image guidance for all - TileProTM display of three-dimensionally reconstructed images in robotic partial nephrectomy. Urology. 2014, 84: 237-42. 10.1016/j.urology.2014.02.051.

    Article  PubMed  Google Scholar 

  4. Pratt P, Hughes-Hallett A, Di Marco A, Cundy T, Mayer E, Vale J, Darzi A, Yang G-Z: Multimodal Reconstruction for Image-Guided Interventions. Proceedings of the Hamlyn Symposium. 2013, 59-60.

    Google Scholar 

  5. Hughes-Hallett A, Pratt P, Mayer E, Di Marco A, Yang G-Z, Vale J, Darzi A: Intraoperative Ultrasound Overlay in Robot-assisted Partial Nephrectomy: First Clinical Experience. Eur Urol. 2013, 65: 671-672.

    Article  PubMed  Google Scholar 

Download references


The authors are grateful for support from the NIHR Biomedical Research Centre funding scheme.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Erik Mayer.

Rights and permissions

Open Access  This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit

The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hughes-Hallett, A., Pratt, P., Dilley, J. et al. Augmented reality: 3D image-guided surgery. Cancer Imaging 15 (Suppl 1), O8 (2015).

Download citation

  • Published:

  • DOI: