Advancements of Image Processing and Vision in HealthcareView this Special Issue
Review Article | Open Access
Recent Development of Augmented Reality in Surgery: A Review
Introduction. The development augmented reality devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency, safety, and cost and to enhance surgical training. However, the awareness of possibilities of augmented reality is generally low. This review evaluates whether augmented reality can presently improve the results of surgical procedures. Methods. We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.” Results. The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review. Conclusions. The present literature suggest an increasing interest of surgeons regarding employing augmented reality into surgery leading to improved safety and efficacy of surgical procedures. Many studies showed that the performance of newly devised augmented reality systems is comparable to traditional techniques. However, several problems need to be addressed before augmented reality is implemented into the routine practice.
The first experiments with medical images date back to the year 1895, when W. C. Röntgen discovered the existence of X-ray. This marks the starting point of using medical images in the clinical practice. The development of ultrasound (USG), computed tomography (CT), magnetic resonance imaging (MRI), and other imaging techniques allows physicians to use two-dimensional (2D) medical images and three-dimensional (3D) reconstructions in diagnosis and treatment of various health problems. Further development of medical technology has given an opportunity to combine anatomical and functional (or physiological) imaging in advanced diagnostic procedures, that is, functional MRI (fMRI) or single photon emission computed tomography (SPECT/CT). These methods allowed physicians to better understand both the anatomical and the functional aspects of a target area.
The latest development in medical imaging technology focuses on the acquisition of real-time information and data visualization. Improved accessibility of real-time data is becoming increasingly important as their usage often makes the diagnosis and treatment faster and more reliable. This is especially true in surgery, where the real-time access to 2D or 3D reconstructed images during an ongoing surgery can prove to be crucial. This access is further enhanced by the introduction of augmented reality (AR)—a fusion of projected computer-generated (CG) images and real environment.
The ability to work in symbiosis with a computer broadens horizons of what is possible in surgery, as AR can alter the reality we experience in many ways. The wide range of possibilities it offers to surgeons challenges us to develop new techniques based on AR. In the future, AR may fully replace many items required to perform a successful surgery today, that is, navigation, displays, microscopes, and much more, all in a small wearable piece of equipment. However, the awareness of AR implementation and what it may offer is generally low, as at current state, it cannot fully replace most of long established surgical methods. The main aim of this work is to focus on the latest trends of the rapidly developing connection between augmented reality and surgery.
We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.”
The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review.
3. Results and Discussion
3.1. Basic Principles of Augmented Reality
An augmented reality system provides the surgeon with computer-processed imaging data in real-time via dedicated hardware and software. The projection of AR is made possible by using displays, projectors, cameras, trackers, or other specialized equipment. The main principle of a basic AR system is presented in Figure 1. The most basic method is to superimpose a CG image on a real-world imagery captured by a camera and displaying the combination of these on a computer, tablet PC, or a video projector [1–7]. In case it is impossible to mount a video projector in the operating room, a portable video projection device has been designed [3, 4]. The main advantage of AR is that the surgeon is not forced to look away from the surgical site as opposed to common visualization techniques.
Another possibility is to use a special head-mounted display (HMD, sometimes referred to as “smart glasses”) which resembles eyeglasses. They use special projectors, head tracking, and depth cameras to display CG images on the glass, effectively creating the illusion of augmented reality. Several AR systems with a HMD have already been developed with success [1, 8]. Using a HMD is beneficial as there is almost no obstruction in the surgeon’s view compared to a traditional display; it is not necessary to move the display, and the need of a proper line-of-sight alignment between the display and the surgeon is not as accented .
At present, the applications of AR are limited by the essential requisite of preoperative 3D reconstructions of medical images. It is possible to create these reconstructions by using commercial or self-made software from the Digital Imaging and Communications in Medicine (DICOM) format [7, 10–12]. The quality of a reconstruction depends on the quality of input data and the accuracy of the reconstruction system. Such reconstructions can be used for virtual exploration of target areas, planning an effective surgical approach in advance, and for better orientation and navigation in the operative field.
The type and amount of displayed data rely on the requirements of the procedure and personal preferences of the surgical team. AR is especially useful in visualizing critical structures such as major vessels, nerves, or other vital tissues. By projecting these structures directly onto the patient, AR increases safety and reduces the time required to complete the procedure. Another useful feature of AR is the ability to control the opacity of displayed objects [13, 14]. Most HMDs allow the wearer to turn off all displayed images, becoming fully opaque, thus removing any possible distractions in an emergency. Furthermore, it is possible to utilize voice recognition to create voice commands, enabling hands-free control of the device. This is especially important in surgery as it allows surgeons to control the device without the need of assistance or break aseptic protocols. Another interesting option is to use gesture recognition, allowing the team to interact with the hardware even on sterile surfaces or in the air through body movements .
3.2. Monitoring the Operative Field
Most surgeries target deformable structures, which change significantly during a procedure (i.e., a removal of tissue during a resection). This problem needs to be addressed by constantly monitoring the operative field and making real-time changes to the displayed 3D model. It is possible to use perioperative ultrasound [15, 16], CT, or MRI to update the surgical site model. However, the amount of time required to capture and reconstruct medical images is significant. Using CT  or its modifications (i.e., C-arm cone-beam CT ) expose the patient to radiation and therefore can only be performed in a limited number of times. An open MRI is a feasible option for perioperative imaging; however, a single surgery requires 40 ± 9.4 minutes of scanning in average and the use of specialized MRI-compatible instruments . Automatic medical reconstructions tend to include many different structures, which make the orientation difficult, especially in abdominal surgery. In some cases, this can be overcome by a method proposed by Sugimoto et al. . In this technique, carbon dioxide is introduced into the gastrointestinal tract and pancreatico-biliary duct in conjunction with an intravenous contrast agent, which allows better display of these individual structures . The quality of reconstructions may be improved in the future due to the development of imaging techniques, which allow making more detailed and refined images. Medical imaging may be replaced by depth-sensing cameras or video cameras in some cases; however, this method cannot detect structures beneath the surface that might be negated by developing a software to predict tissue behaviour by measuring forces applied to the target tissue. However, all currently proposed solutions need further improvement [7, 12, 17, 20, 21].
3.3. Augmented Reality for Education and Cooperation
Augmented reality proved to be an effective tool for training and skill assessment of surgery residents, other medical staff, or students [22–26]. Specialized training simulators can be created and used to improve surgeons’ skills in various scenarios, as well as to objectively measure their technical skills. This will be especially beneficial for trainee residents and students in developing intuition and proper decision-making abilities, which can otherwise be only gained through long clinical practice. It allows simulation of very unlikely scenarios, which can be solved with the assistance of an experienced surgeon. Compared to virtual reality (VR) simulators, where the whole simulation takes part in a CG environment, the main advantage of AR simulators is the ability to combine real life objects with CG images, resulting in satisfactory tactile feedback as opposed to VR. AR has also been reported to increase the enjoyment of basic surgical training .
AR enables experienced surgeons to remotely assist residents by using Internet connection and therefore opens the way of excellent distant teaching. Shenai et al.  created virtual interactive presence and augmented reality (VIPAR) system. By continually monitoring and transmitting the image of a surgical site between two distant stations, it enables remote virtual collaboration between two surgeons . This concept is sometimes referred to as “telepresence.” Davis et al.  used VIPAR system in an effort to allow communication between Vietnam and the USA. Thanks are due to the high resolution of the transmitted image, submillimetre achieved precision, and average latency of 237 milliseconds; the interaction between both surgeons was described as effective .
3.4. Methods of Image Alignment
The exact alignment of a real environment and CG images is an extremely important aspect to consider. The simplest method is to manually align both images . Such a method is slow and may be imprecise, and therefore the registration process (the alignment of preoperative images with the currently treated patient) needs to be continuous to compensate for the changes in organ layout, that is, during breathing. The accurate alignment of both images is achieved by a set of trackers, which are used to determine the exact position of the camera and patient’s body. These trackers usually track fiducial markers placed on the surface of specific structures, which remain still during the surgery (i.e., iliac crest, clavicles, etc.) and thus providing the system with points of reference.
There are several types of markers described in the literature. Many authors used a set of optical markers and a dedicated camera to detect them for navigation [15, 31–36]. The positions of the camera and the patient are determined by measuring distances between individual trackers and markers. Several studies reported using a set of infrared markers [4, 10, 13, 33, 37–40]. A newer approach suggested by Wild et al.  involved using fluorescent markers during laparoscopic procedures. The main advantage of such markers is the clear visibility even in difficult conditions and the ability to remain in the patient after the surgery. However, this requires an endoscopic system capable of detecting the emitted light . Konishi et al. proposed an electromagnetic tracking system paired with an infrared sensor for laparoscopic surgery . The main problem of magnetic tracking is the distortion caused by metal tools and equipment, which can however be minimized by using a special calibration technique . The passive coordinate measurement arm (PCMA) method promises very high precision and the possibility to navigate without a direct line of sight by using a robotic arm for distance measurement . Another method of registration is tracking of a laparoscopic camera and consequent automatic alignment of captured video with a 3D reconstruction . Inoue et al. proved that using simple and affordable equipment, for instance a set of regular web cameras and freeware 3D reconstruction software, can be used to create a registration system thus depicting the purchase of such systems as possibility for all medical institutions . The necessity of a direct line of sight may reduce the maximum possible organ deformation and rotation as well as the maximum range of tracked tools, depending on the exact configuration of the tracking system. It is possible to employ an increased number of tracker markers to minimize the chance of tracking failure due to a line of sight obstruction. Several commercially available tracking systems for medical AR are available, relying mostly on infrared or electromagnetic tracking .
With advancement in technology in the future, it may be possible to track organ position in real-time without the use of dedicated markers, using various methods for analysis of the operative field. Hostettler et al. used a real-time predictive simulation of abdominal organs , and Haouchine et al. designed a physics-based deformation model of registration . These approaches are based on the usage of computing power to predict and visualize organ movement and deformation. It is also possible to use an RGB (red-green-blue) or range camera to perform registration without the use of markers [49, 50]. On the other hand, Hayashi et al. described natural points of reference as tracking points in the patient’s body for progressive registration of cut vessels as markers . Kowalczuk et al. created a system for real-time 3D modelling of the surgical site, with accuracy within 1.5 mm by using a high definition stereoscopic camera and a live reconstruction of the captured image .
Another possibility is to use a laser surface scanning technique, aligning images by scanning a high number of surface points without the use of fiducial markers. While Krishnan et al. found the accuracy of laser scanning to be sufficient , another study by Schicho et al.  suggests lower accuracy compared to using fiducial markers, with the overall deviation of 3.0 mm and 1.4 mm, respectively. Authors suggest that the accuracy may be improved by using an increased number of tracked surface points . An additional marker-less registration method has been described by Lee et al., where authors combined a cone-beam CT image with a 3D RGB image with a sufficient accuracy of 2.58 mm; the absence of real-time image alignment is however still an issue . All camera-based techniques are severely limited by the necessity of a direct line of sight. A novel method designed by Nosrati et al.  estimated organ movement by combining preoperative data, intraoperative colour, and visual cues with a vascular pulsation detection (also described by Amir-Khalili et al. ), resulting in a robust system not affected by common obstructions (light reflection, light smoke). This method increased the accuracy by 45% compared to traditional techniques. The proposed method, however, is not capable of real-time computation, as every registration requires approximately 10 seconds to be completed .
It needs to be noted that the amount of time required to prepare and calibrate an AR system needs to be considered for routine implementation. Pessaux et al.  reported a delay of a few seconds for each marker registration. The total time for AR-related registrations was 8 (6–10) minutes [7, 16]. The registration process of a marker-less system used by Sugimoto et al.  was completed within 5 minutes. The latency of displaying movement is important as lower latency generally means a better experience.
The accuracy and complexity of 3D reconstructed imaging are crucial in providing the correct data to the surgical team. An exact comparison of accuracy between specific studies is impossible due to variable conditions and different approaches for measuring accuracy. Optical systems feature precision within 5 mm in several studies [8, 10, 15, 18, 35, 37, 44, 57–59], which is considered sufficient for clinical application. The required precision differs greatly among various procedures and should be determined individually. The PCMA method of registering relative positions represents the best precision of all mentioned techniques, with an overall precision of <1 mm . Yoshino et al. used a high-resolution MRI image for the reconstruction and an optical tracking system, with reported accuracy of 2.9 ± 1.9 mm while using an operative microscope during an experiment in a phantom model . Two studies reported using a video projector during phantom experiments or an actual surgery, with results comparable to using an electronic display, featuring accuracy of 0.8–1.86 mm [1, 5]. Gavaghan et al. proposed a portable projection device with accuracy of 1.3 mm . A few authors achieved a deviation of <2.7 mm while using a head-mounted display [8, 40, 60]. Although the difference in precision between a computer monitor and an HMD is not statistically significant . It has been noted that the accuracy is not dependent on the surgeon’s experience . The maximum achievable precision is further diminished due to the difficulties in giving the projected image 3D appearance [1, 31] or giving a correct depth perception [8, 32, 34]. One of the possible solutions is to display objects with different opacity  or increasingly dark colour. This could be further minimized by taking advantage of motion parallax, which can be created by tracking surgeons head and modifying the projection accordingly .
3.6. Uses in Clinical Practice
Augmented reality can be used effectively for preoperative planning and completion of the actual surgery in timely fashion. The preoperative 3D reconstructed images can be modified and prepared for display in AR systems. Commonly, AR is used for tailoring individually preferred incisions and cutting planes [1, 36, 58], optimal placement of trocars,  or to generally improve safety by displaying positions of major organ components . Another benefit of AR is the ability to aid surgeons in difficult terrain after a neoadjuvant chemotherapy or radiotherapy . AR may be used to envisage and optimize the surgical volume of resection . In many procedures, the AR-assisted surgery is comparable to other methods of assistance and the usage of such devices depends on surgeon’s preference .
Augmented reality systems are the most useful during a surgery of organs with little movement and deformation (i.e., skull, brain, and pancreas) as the least amount of tracking and processing power is required whereas mobile organs, like the bowel, are significantly more complicated track and display. Considering these facts, the most frequently targeted areas in AR research are the head and brain [1, 8, 9, 11, 22, 31, 34, 36, 59, 61, 65–69], orthopaedic surgery [4, 37, 60, 70–77], hepato-biliary system [2, 3, 7, 13, 18, 32], and pancreas [13, 14, 35]. On top of that, AR may compensate the lack of tactile feedback usually experienced during laparoscopic surgery by presenting the surgeon with visual clues, thus improving hand-eye coordination and orientation, even in robotic surgery [78, 79]. Many studies proposed using augmented reality in laparoscopic procedures with success [7, 33, 38, 51, 64, 69].
Neurosurgical procedures have employed AR systems successfully, because of the inherent limitation of head and brain movement. It has been reported that AR had a major impact in 16.7% of neurovascular surgeries , allowing a higher rate of precise localization of lesions and shorter operative time compared to a traditional 2D approach . Neurosurgeons benefit mainly from precise localization of individual gyri, blood vessels, important neuronal tracts, and the possibility to plan the operation corridor, for instance, in a removal of superficial tumours [10, 66], epilepsy surgery , or in neurovascular surgery .
AR has also proved to be useful during orthopaedic surgeries and reconstructions, especially because it allows to view reconstructions directly on top of the patient’s body, which reduces the number of distractions caused by looking at an external display . The list of procedures successfully utilizing AR ranges from minimally invasive percutaneous surgeries to trauma reconstructions , bone resections, osteotomies [73, 77, 81], arthroscopic surgery, Kirschner wire placement , joint replacement, or tumour removal [75, 76]. Percutaneous interventions only require a surface indicator of the insertion point, which can be displayed by AR . A fluoroscopic dual-laser-based system was used by Liang et al.  for insertion guidance with satisfactory accuracy. However, the inability to perceive the depth of insertion forces the use of additional techniques . In spite of that, the use of AR limits the radiation exposure during fluoroscopy and the amount of time required to perform the task [71, 72] while minimizing the risk of unnecessary haemorrhage. Rodas and Padoy used AR to create a user friendly visualization of scattered radiation during a minimally invasive surgery , enabling to measure and visualize the amount of radiation received. AR can change the workflow of creating orthopaedic implants, replacing 3D printing of a patient-specific model . It is also possible to employ AR in orthopaedic robot-assisted surgeries .
It is more difficult to use AR in abdominal surgery, as the amount of organ movement is significant; nevertheless, it is currently used during liver and pancreatic surgeries as it allows better projection of large vessels  or tumour sites owing to comparative static nature of these organs. By comparing the reconstructed data with intraoperative ultrasound, AR can be used for intraoperative guidance during liver resections . It can also assist the surgeon with the establishment of laparoscopic ports  or phrenotomy sites . Similarly, the kidneys appear to be suitable for the usage of AR, as demonstrated by Muller et al. during a nephrolithotomy, where the AR was used to establish the percutaneous renal access . An AR system has been proposed to accurately detect a sentinel node using a preoperative SPECT/CT scan of the surrounding lymph nodes. This allows to precisely navigate to the sentinel node and perform a resection even in difficult terrain . Using a freehand SPECT to scan radiation distribution, reconstruction of the data and its implementation into AR in real time can extend the concept . AR has also been successfully employed during splenectomies in children  and urological procedures . With the increasing precision of AR systems, they can be safely used even in endocrine surgeries [87–89], otorhinolaryngologic surgery , eye surgery , vascular surgery [92, 93], or dental implantology too ; however, their exact usefulness in such surgeries is not exactly quantified owing to the complicated structures . An AR system for transcatheter aortic valve implantation was designed and successfully used by Currie et al., showing comparable accuracy to traditional fluoroscopic guidance technique, hence avoiding impending complications of contrast agent administration .
The use of AR in robotic surgery is expanding rapidly, due to its ability to easily incorporate AR directly into the operators’ console, which allows the surgeon to navigate more quickly and better identify important structures [7, 94]. The CG projection can be displayed separately or as an overlay of real-time video as needed by the surgeon. A few authors successfully applied AR during a robotic surgery with satisfactory results, showing a possible role of AR as a future trend in robotic surgery [12, 79, 95].
3.7. Advanced Image in Fusion Using Augmented Reality
Augmented reality cannot only display CG images but can also display images which are not normally visible. A handful of studies successfully combined optical images with near-infrared fluorescence images [96–99]. This technique can be used to increase precision by providing additional data for a surgeon to consider. Using this technique, surgeons could be able to detect blood vessels under the organ surface or detect other tissue abnormalities. Koreeda et al.  proposed another interesting concept. Here, AR has been used to visualize areas obscured by surgical instruments in laparoscopic procedures, making the tools appear invisible. This system also helped in effectively reducing the needle exit point errors during suturing, while achieving average accuracy of 1.4 mm and acceptable latency of 62.4 milliseconds .
3.8. Problems of Augmented Reality
Augmented reality introduces many new possibilities and adds new dimensions to surgical science. Surgeons can use additional data for decision making and improving safety and efficacy. Advances in technology allow AR devices to display information with increasing accuracy and lower latency. Despite of constant improvement, there are several difficulties which need to be addressed. Currently, all reconstructed images need to be prepared in advance using complex algorithms requiring powerful computers. However, thanks to an expected advancement in technology, a real-time acquisition of high-resolution medical scans and 3D reconstructions may be possible. Such reconstructions of the operative field would significantly improve overall accuracy. Even though AR can speed up a surgical procedure, the necessity to prepare the whole system and make required registrations and measurements generally increases the time necessary to complete the surgery, with the amount of time depending on the type of the procedure and the complexity of the AR system. The introduction of fully automatic systems would eliminate this problem and reduce the total time required for completion.
The time required for completing a procedure has been reduced while using any form of AR; however, there are certain limitations. One of them is inattentional blindness (an event where the surgeon does not see an unexpected object which suddenly appears in his field of view), which is a concerning issue that needs to be addressed while using 3D overlays . The amount of information a surgeon has presented through AR during a surgery is increasing and may be distracting . Therefore, it is necessary to display only important data or provide a method to switch between different sets of information on demand. A method for the reduction of cognitive demands was proposed by Hansen et al. . By optimizing the spatial relationship of individual structures, reducing the occlusion caused by AR to minimum, and maximizing contrast, surgeons were able to reduce visual clutter in some cases . Reaching an adequate image contrast during a projection directly onto organs is also necessary . Difficulties in creating a correct 3D and depth perception also persist. The latency of the whole system is also of concern because excessive latency may lower precision and reduce comfort of the surgeon. Kang et al. measured the latency of their optical system for laparoscopic procedures to be 144 ± 19 milliseconds .
Currently used head-mounted displays usually weigh several hundred grams and produce plenty of heat; therefore, long-term wear comfort is an issue. These need to be addressed in future to better fit the ergonomics and allow continuous usage for a long period. It is known that virtual and augmented reality projections in HMDs produce simulator sickness, which is presented by nausea, headache, and vertigo or vomiting in the worst scenario. The exact causes behind the simulator sickness are unknown; however, a discrepancy between visual, proprioceptive, and vestibular inputs is probably the case.
Studies suggest that AR systems are becoming comparable to traditional navigation techniques, with precision and safety sufficient for routine clinical practice. Most problems faced presently will be solved by further medical and technological research. Augmented reality appears to be a powerful tool possibly capable of revolutionising the field of surgery through a rational use. In the future, AR will likely serve as an advanced human-computer interface, working in symbiosis with surgeons, allowing them to achieve even better results. Nevertheless, further advancement is much needed to achieve maximum potential and cost-effectiveness of augmented reality.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
- L. B. Tabrizi and M. Mahvash, “Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique,” Journal of Neurosurgery, vol. 123, pp. 206–211, 2015.
- M. Sugimoto, H. Yasuda, K. Koda et al., “Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery,” Journal Hepato-Biliary-Pancreatic Sciences, vol. 17, pp. 629–636, 2010.
- K. A. Gavaghan, M. Peterhans, T. Oliveira-Santos, and S. Weber, “A portable image overlay projection device for computer-aided open liver surgery,” IEEE Transactions on Biomedical Engineering, vol. 58, pp. 1855–1864, 2011.
- K. Gavaghan, T. Oliveira-Santos, M. Peterhans et al., “Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies,” International Journal of Computer Assisted Radiology and Surgery, vol. 7, pp. 547–556, 2012.
- R. Wen, C. K. Chui, S. H. Ong, K. B. Lim, and S. K. Y. Chang, “Projection-based visual guidance for robot-aided RF needle insertion,” International Journal of Computer Assisted Radiology and Surgery, vol. 8, pp. 1015–1025, 2013.
- B. Kocev, F. Ritter, and L. Linsen, “Projector-based surgeon-computer interaction on deformable surfaces,” International Journal of Computer Assisted Radiology and Surgery, vol. 9, pp. 301–312, 2014.
- P. Pessaux, M. Diana, L. Soler, T. Piardi, D. Mutter, and J. Marescaux, “Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy,” Langenbeck’s Archives of Surgery, vol. 400, pp. 381–385, 2015.
- G. Badiali, V. Ferrari, F. Cutolo et al., “Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning,” Journal of Cranio-Maxillofacial Surgery, vol. 42, pp. 1970–1976, 2014.
- E. Watanabe, M. Satoh, T. Konno, M. Hirai, and T. Yamaguchi, “The trans-visible navigator: a see-through neuronavigation system using augmented reality,” World Neurosurgery, vol. 87, pp. 399–405, 2016.
- D. Inoue, B. Cho, M. Mori et al., “Preliminary study on the clinical application of augmented reality neuronavigation,” Journal of Neurological Surgery Part A, vol. 74, pp. 71–76, 2013.
- G. Callovini, S. Sherkat, G. Callovini, and R. Gazzeri, “Frameless nonstereotactic image-guided surgery of supratentorial lesions: introduction to a safe and inexpensive technique,” Journal of Neurological Surgery Part A, vol. 75, pp. 365–370, 2014.
- P. Pessaux, M. Diana, L. Soler, T. Piardi, D. Mutter, and J. Marescaux, “Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance,” Surgical Endoscopy, vol. 28, pp. 2493–2498, 2014.
- T. Okamoto, S. Onda, K. Yanaga, N. Suzuki, and A. Hattori, “Clinical application of navigation surgery using augmented reality in the abdominal field,” Surgery Today, vol. 45, pp. 397–406, 2015.
- J. Marescaux and M. Diana, “Inventing the future of surgery,” World Journal of Surgery, vol. 39, pp. 615–622, 2015.
- X. Kang, M. Azizian, E. Wilson et al., “Stereoscopic augmented reality for laparoscopic surgery,” Surgical Endoscopy, vol. 28, pp. 2227–2235, 2014.
- S. J. Gonzalez, Y. H. Guo, and M. C. Lee, “Feasibility of augmented reality glasses for real-time, 3-dimensional (3D) intraoperative guidance,” Journal of the American College of Surgeons, vol. 219, pp. S64–S64, 2014.
- R. Shekhar, O. Dandekar, V. Bhat et al., “Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography,” Surgical Endoscopy, vol. 24, pp. 1976–1985, 2010.
- H. G. Kenngott, M. Wagner, M. Gondan et al., “Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging,” Surgical Endoscopy, vol. 28, pp. 933–940, 2014.
- N. Tsutsumi, M. Tomikawa, M. Uemura et al., “Image-guided laparoscopic surgery in an open MRI operating theater,” Surgical Endoscopy, vol. 27, pp. 2178–2184, 2013.
- M. Kranzfelder, M. Bauer, M. Magg et al., “Image guided surgery,” Endoskopie Heute, vol. 27, pp. 154–158, 2014.
- W. H. Nam, D. G. Kang, D. Lee, J. Y. Lee, and J. B. Ra, “Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching,” Physics in Medicine and Biology, vol. 57, pp. 69–91, 2012.
- S. F. Shakur, C. J. Luciano, P. Kania et al., “Usefulness of a virtual reality percutaneous trigeminal rhizotomy simulator in neurosurgical training,” Operative Neurosurgery, vol. 11, pp. 420–425, 2015.
- F. Leblanc, B. J. Champagne, K. M. Augestad et al., “Colorectal Surg Training Group: a comparison of human cadaver and augmented reality simulator models for straight laparoscopic colorectal skills acquisition training,” Journal of the American College of Surgeons, vol. 211, pp. 250–255, 2010.
- N. Lelos, P. Campos, K. Sugand, C. Bailey, and K. Mirza, “Augmented reality dynamic holography for neurology,” Journal of Neurology, Neurosurgery, and Psychiatry, vol. 85, p. 1, 2014.
- A. M. Vera, M. Russo, A. Mohsin, and S. Tsuda, “Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology,” Surgical Endoscopy, vol. 28, pp. 3467–3472, 2014.
- V. Lahanas, C. Loukas, N. Smailis, and E. Georgiou, “A novel augmented reality simulator for skills assessment in minimal invasive surgery,” Surgical Endoscopy, vol. 29, pp. 2224–2234, 2015.
- A. C. Profeta, C. Schilling, and M. McGurk, “Augmented reality visualization in head and neck surgery: an overview of recent findings in sentinel node biopsy and future perspectives,” The British Journal of Oral & Maxillofacial Surgery, vol. 54, pp. 694–696, 2016.
- M. B. Shenai, M. Dillavou, C. Shum et al., “Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance,” Neurosurgery, vol. 68, p. 8, 2011.
- M. C. Davis, D. D. Can, J. Pindrik, B. G. Rocque, and J. M. Johnston, “Virtual interactive presence in global surgical education: international collaboration through augmented reality,” World Neurosurgery, vol. 86, pp. 103–111, 2016.
- A. S. Vemuri, J. C. H. Wu, K. C. Liu, and H. S. Wu, “Deformable three-dimensional model architecture for interactive augmented reality in minimally invasive surgery,” Surgical Endoscopy, vol. 26, pp. 3655–3662, 2012.
- M. Yoshino, T. Saito, T. Kin et al., “A microscopic optically tracking navigation system that uses high-resolution 3D computer graphics,” Neurologia Medico-Chirurgica, vol. 55, pp. 674–679, 2015.
- T. Okamoto, S. Onda, M. Matsumoto et al., “Utility of augmented reality system in hepatobiliary surgery,” Journal Hepato-Biliary-Pancreatic Sciences, vol. 20, pp. 249–253, 2013.
- S. Ieiri, M. Uemura, K. Konishi et al., “Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device,” Pediatric Surgery International, vol. 28, pp. 341–346, 2012.
- M. Kersten-Oertel, I. Gerard, S. Drouin et al., “Augmented reality in neurovascular surgery: feasibility and first uses in the operating room,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 1823–1836, 2015.
- T. Okamoto, S. Onda, J. Yasuda, K. Yanaga, N. Suzuki, and A. Hattori, “Navigation surgery using an augmented reality for pancreatectomy,” Digestive Surgery, vol. 32, pp. 117–123, 2015.
- M. Qu, Y. K. Hou, Y. R. Xu et al., “Precise positioning of an intraoral distractor using augmented reality in patients with hemifacial microsomia,” Journal of Cranio-Maxillofacial Surgery, vol. 43, pp. 106–112, 2015.
- J. T. Liang, T. Doke, S. Onogi et al., “A fluorolaser navigation system to guide linear surgical tool insertion,” International Journal of Computer Assisted Radiology and Surgery, vol. 7, pp. 931–939, 2012.
- S. Onda, T. Okamoto, M. Kanehira et al., “Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery,” Journal Hepato-Biliary-Pancreatic Sciences, vol. 20, pp. 448–453, 2013.
- S. Onda, T. Okamoto, M. Kanehira et al., “Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality-based navigation system,” Journal Hepato-Biliary-Pancreatic Sciences, vol. 21, pp. 281–287, 2014.
- B. Vigh, S. Muller, O. Ristow et al., “The use of a head-mounted display in oral implantology: a feasibility study,” International Journal of Computer Assisted Radiology and Surgery, vol. 9, pp. 71–78, 2014.
- E. Wild, D. Teber, D. Schmid et al., “Robust augmented reality guidance with fluorescent markers in laparoscopic surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 899–907, 2016.
- K. Konishi, M. Hashizume, M. Nakamoto et al., “Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: application to 20 clinical cases,” in CARS 2005: Computer Assisted Radiology and Surgery, H. U. Lemke, K. Inamura, K. Doi, M. W. Vannier, and A. G. Farman, Eds., vol. 1281, pp. 537–542, Elsevier Science Bv, Amsterdam, 2005.
- M. Nakamoto, K. Nakada, Y. Sato, K. Konishi, M. Hashizume, and S. Tamura, “Intraoperative magnetic tracker calibration using a magneto-optic hybrid tracker for 3-D ultrasound-based navigation in laparoscopic surgery,” IEEE Transactions on Medical Imaging, vol. 27, pp. 255–270, 2008.
- R. J. Lapeer, S. J. Jeffrey, J. T. Dao et al., “Using a passive coordinate measurement arm for motion tracking of a rigid endoscope for augmented-reality image-guided surgery,” International Journal of Medical Robotics and Computer Assisted Surgery, vol. 10, pp. 65–77, 2014.
- C. Conrad, M. Fusaglia, M. Peterhans, H. X. Lu, S. Weber, and B. Gayet, “Augmented reality navigation surgery facilitates laparoscopic rescue of failed portal vein embolization,” Journal of the American College of Surgeons, vol. 223, pp. E31–E34, 2016.
- S. Nicolau, L. Soler, D. Mutter, and J. Marescaux, “Augmented reality in laparoscopic surgical oncology,” Surgical Oncology-Oxford, vol. 20, pp. 189–201, 2011.
- A. Hostettler, S. A. Nicolau, Y. Remond, J. Marescaux, and L. Soler, “A real-time predictive simulation of abdominal viscera positions during quiet free breathing,” Progress in Biophysics and Molecular Biology, vol. 103, pp. 169–184, 2010.
- N. Haouchine, J. Dequidt, M. O. Berger, and S. Cotin, “Deformation-based augmented reality for hepatic surgery,” Studies in Health Technology and Informatics, vol. 184, pp. 182–188, 2013.
- T. Kilgus, E. Heim, S. Haase et al., “Mobile markerless augmented reality and its application in forensic medicine,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 573–586, 2015.
- Y. Hayashi, K. Misawa, D. J. Hawkes, and K. Mori, “Progressive internal landmark registration for surgical navigation in laparoscopic gastrectomy for gastric cancer,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 837–845, 2016.
- J. Kowalczuk, A. Meyer, J. Carlson et al., “Real-time three-dimensional soft tissue reconstruction for laparoscopic surgery,” Surgical Endoscopy, vol. 26, pp. 3413–3417, 2012.
- R. Krishnan, A. Raabe, and V. Seifert, “Accuracy and applicability of laser surface scanning as new registration technique in image-guided neurosurgery,” in Cars 2004: Computer Assisted Radiology and Surgery, H. U. Lemke, K. Inamura, K. Doi, M. W. Vannier, A. G. Farman, and R. JHC, Eds., vol. 1268, pp. 678–683, Elsevier Science Bv, Amsterdam, 2004.
- K. Schicho, M. Figl, R. Seemann et al., “Comparison of laser surface scanning and fiducial marker-based registration in frameless stereotaxy - technical note,” Journal of Neurosurgery, vol. 106, pp. 704–709, 2007.
- S. C. Lee, B. Fuerst, J. Fotouhi, M. Fischer, G. Osgood, and N. Navab, “Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 967–975, 2016.
- M. S. Nosrati, A. Amir-Khalili, J. M. Peyrat et al., “Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 1409–1418, 2016.
- A. Amir-Khalili, G. Hamarneh, J. M. Peyrat et al., “Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video,” Medical Image Analysis, vol. 25, pp. 103–110, 2015.
- L. Li, J. Yang, Y. K. Chu et al., “A novel augmented reality navigation system for endoscopic sinus and skull base surgery: a feasibility study,” PLoS One, vol. 11, p. 17, 2016.
- M. Muller, M. C. Rassweiler, J. Klein et al., “Mobile augmented reality for computer-assisted percutaneous nephrolithotomy,” International Journal of Computer Assisted Radiology and Surgery, vol. 8, pp. 663–675, 2013.
- W. W. Deng, F. Li, M. N. Wang, and Z. J. Song, “Easy-to-use augmented reality neuronavigation using a wireless tablet PC,” Stereotactic and Functional Neurosurgery, vol. 92, pp. 17–24, 2014.
- D. Katic, P. Spengler, S. Bodenstedt et al., “A system for context-aware intraoperative augmented reality in dental implant surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 101–108, 2015.
- I. Cabrilo, P. Bijlenga, and K. Schaller, “Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations,” Acta Neurochirurgica, vol. 156, pp. 1769–1774, 2014.
- M. Riechmann, L. Kahrs, C. Ulmer, J. Raczkowsky, W. Lamadé, and H. Wörn, “Visualisierungskonzept für die projektorbasierte Erweiterte Realität in der Leberchirurgie,” Proceeding of BMT, vol. 209, pp. 1-2, 2006.
- F. Volonte, F. Pugin, P. Bucher, M. Sugimoto, O. Ratib, and P. Morel, “Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion,” Journal Hepato-Biliary-Pancreatic Sciences, vol. 18, pp. 506–509, 2011.
- R. Souzaki, S. Ieiri, M. Uemura et al., “An augmented reality navigation system for pediatric oncologic surgery based on preoperative CT and MRI images,” Journal of Pediatric Surgery, vol. 48, pp. 2479–2483, 2013.
- A. Wang, S. M. Mirsattari, A. G. Parrent, and T. M. Peters, “Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance,” Computer Aided Surgery, vol. 16, pp. 149–160, 2011.
- A. Mert, K. Buehler, G. R. Sutherland et al., “Brain tumor surgery with 3-dimensional surface navigation,” Neurosurgery, vol. 71, pp. 286–294, 2012.
- I. Cabrilo, P. Bijlenga, and K. Schaller, “Augmented reality in the surgery of cerebral aneurysms: a technical report,” Neurosurgery, vol. 10, pp. 252–260, 2014.
- I. Cabrilo, A. Sarrafzadeh, P. Bijlenga, B. N. Landis, and K. Schaller, “Augmented reality-assisted skull base surgery,” Neurochirurgie, vol. 60, pp. 304–306, 2014.
- A. Schoob, D. Kundrat, L. Kleingrothe, L. A. Kahrs, N. Andreff, and T. Ortmaier, “Tissue surface information for intraoperative incision planning and focus adjustment in laser surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 171–181, 2015.
- N. L. Rodas and N. Padoy, “Seeing is believing: increasing intraoperative awareness to scattered radiation in interventional procedures by combining augmented reality, Monte Carlo simulations and wireless dosimeters,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 1181–1191, 2015.
- R. Londei, M. Esposito, B. Diotte et al., “Intra-operative augmented reality in distal locking,” International Journal of Computer Assisted Radiology and Surgery, vol. 10, pp. 1395–1403, 2015.
- F. Y. Shen, B. L. Chen, Q. S. Guo, Y. Qi, and Y. Shen, “Augmented reality patient-specific reconstruction plate design for pelvic and acetabular fracture surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 8, pp. 169–179, 2013.
- M. Zhu, G. Chai, Y. Zhang, X. F. Ma, and J. L. Gan, “Registration strategy using occlusal splint based on augmented reality for mandibular angle oblique split osteotomy,” The Journal of Craniofacial Surgery, vol. 22, pp. 1806–1809, 2011.
- S. Weber, M. Klein, A. Hein, T. Krueger, T. C. Lueth, and J. Bier, “The navigated image viewer - evaluation in maxillofacial surgery,” in Medical Image Computing and Computer-Assisted Intervention - Miccai 2003, Pt 1, R. E. Ellis and T. M. Peters, Eds., vol. 2878, pp. 762–769, Springer-Verlag Berlin, Berlin, 2003.
- C. Nikou, A. M. Digioia, M. Blackwell, B. Jaramaz, and T. Kanade, “Augmented reality imaging technology for orthopaedic surgery,” Operative Techniques in Orthopaedics, vol. 10, pp. 82–86, 2000.
- M. Blackwell, F. Morgan, and A. M. DiGioia, “Augmented reality and its future in orthopaedics,” Clinical Orthopaedics and Related Research, vol. 354, pp. 111–122, 1998.
- A. Wagner, M. Rasse, W. Millesi, and R. Ewers, “Virtual reality for orthognathic surgery: the augmented reality environment concept,” Journal of Oral and Maxillofacial Surgery, vol. 55, pp. 456–462, 1997.
- A. Hughes-Hallett, E. K. Mayer, P. Pratt, A. Mottrie, A. Darzi, and J. Vale, “The current and future use of imaging in urological robotic surgery: a survey of the European Association of Robotic Urological Surgeons,” International Journal of Medical Robotics and Computer Assisted Surgery, vol. 11, pp. 8–14, 2015.
- F. Volonte, F. Pugin, N. C. Buchs et al., “Console-integrated stereoscopic OsiriX 3D volume-rendered images for da Vinci colorectal robotic surgery,” Surgical Innovation, vol. 20, pp. 158–163, 2013.
- I. Cabrilo, K. Schaller, and P. Bijlenga, “Augmented reality-assisted bypass surgery: embracing minimal invasiveness,” World Neurosurgery, vol. 83, pp. 596–602, 2015.
- L. Lin, Y. Y. Shi, A. Tan et al., “Mandibular angle split osteotomy based on a novel augmented reality navigation using specialized robot-assisted arms-a feasibility study,” Journal of Cranio-Maxillofacial Surgery, vol. 44, pp. 215–223, 2016.
- M. Fischer, B. Fuerst, S. C. Lee et al., “Preclinical usability study of multiple augmented reality concepts for K-wire placement,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 1007–1014, 2016.
- D. Ntourakis, R. Memeo, L. Soler, J. Marescaux, D. Mutter, and P. Pessaux, “Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience,” World Journal of Surgery, vol. 40, pp. 419–426, 2016.
- J. Hallet, L. Soler, M. Diana et al., “Trans-thoracic minimally invasive liver resection guided by augmented reality,” Journal of the American College of Surgeons, vol. 220, pp. E55–E60, 2015.
- O. R. Brouwer, N. Van den Berg, H. Matheron et al., “Feasibility of image guided sentinel node biopsy using augmented reality and SPECT/CT-based 3D navigation,” Annals of Surgical Oncology, vol. 20, pp. S103–S103, 2013.
- O. Ukimura and I. S. Gill, “Imaging-assisted endoscopic surgery: Cleveland clinic experience,” Journal of Endourology, vol. 22, pp. 803–809, 2008.
- J. D'Agostino, J. Wall, L. Soler, M. Vix, Q. Y. Duh, and J. Marescaux, “Virtual neck exploration for parathyroid adenomas a first step toward minimally invasive image-guided surgery,” JAMA Surgery, vol. 148, pp. 232–238, 2013.
- J. D'Agostino, M. Diana, M. Vix et al., “Three-dimensional metabolic and radiologic gathered evaluation using VR-RENDER fusion: a novel tool to enhance accuracy in the localization of parathyroid adenomas,” World Journal of Surgery, vol. 37, pp. 1618–1625, 2013.
- J. Marescaux, F. Rubino, M. Arenas, D. Mutter, and L. Soler, “Augmented-reality-assisted laparoscopic adrenalectomy,” JAMA: The Journal of the American Medical Association, vol. 292, pp. 2214-2215, 2004.
- C. Winne, M. Khan, F. Stopp, E. Jank, and E. Keeve, “Overlay visualization in endoscopic ENT surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 6, pp. 401–406, 2011.
- P. Mezzana, F. Scarinci, and N. Marabottini, “Augmented reality in oculoplastic surgery: first iPhone application,” Plastic and Reconstructive Surgery, vol. 127, pp. 57E–58E, 2011.
- Y. Mochizuki, A. Hosaka, H. Kamiuchi et al., “New simple image overlay system using a tablet PC for pinpoint identification of the appropriate site for anastomosis in peripheral arterial reconstruction,” Surgery Today, vol. 46, pp. 1387–1393, 2016.
- M. E. Currie, A. J. McLeod, J. T. Moore et al., “Augmented reality system for ultrasound guidance of transcatheter aortic valve implantation,” Innovations, vol. 11, pp. 31–39, 2016.
- A. Balaphas, N. C. Buchs, J. Meyer, M. E. Hagen, and P. Morel, “Partial splenectomy in the era of minimally invasive surgery: the current laparoscopic and robotic experiences,” Surgical Endoscopy, vol. 29, pp. 3618–3627, 2015.
- F. Volonte, N. C. Buchs, F. Pugin et al., “Augmented reality to the rescue of the minimally invasive surgeon. The usefulness of the interposition of stereoscopic images in the da Vinci (TM) robotic console,” International Journal of Medical Robotics and Computer Assisted Surgery, vol. 9, pp. E34–E38, 2013.
- J. R. Watson, N. Martirosyan, J. Skoch, G. M. Lemole, R. Anton, and M. Romanowski, “Augmented microscopy with near-infrared fluorescence detection,” in Molecular-Guided Surgery: Molecules, Devices, and Applications, B. W. Pogue and S. Gioux, Eds., vol. 9311, Spie-Int Soc Optical Engineering, Bellingham, 2015.
- M. Diana, P. Halvax, B. Dallemagne et al., “Real-time navigation by fluorescence-based enhanced reality for precise estimation of future anastomotic site in digestive surgery,” Surgical Endoscopy, vol. 28, pp. 3108–3118, 2014.
- M. Diana, E. Noll, P. Diemunsch et al., “Enhanced-reality video fluorescence a real-time assessment of intestinal viability,” Annals of Surgery, vol. 259, pp. 700–707, 2014.
- N. L. Martirosyan, J. Skoch, J. R. Watson, G. M. Lemole, and M. Romanowski, “Integration of indocyanine green videoangiography with operative microscope: augmented reality for interactive assessment of vascular structures and blood flow,” Operative Neurosurgery, vol. 11, pp. 252–257, 2015.
- Y. Koreeda, Y. Kobayashi, S. Ieiri et al., “Virtually transparent surgical instruments in endoscopic surgery with augmentation of obscured regions,” International Journal of Computer Assisted Radiology and Surgery, vol. 11, pp. 1927–1936, 2016.
- H. J. Marcus, P. Pratt, A. Hughes-Hallett et al., “Comparative effectiveness and safety of image guidance systems in neurosurgery: a preclinical randomized study,” Journal of Neurosurgery, vol. 123, pp. 307–313, 2015.
- C. Hansen, J. Wieferich, F. Ritter, C. Rieder, and H. O. Peitgen, “Illustrative visualization of 3D planning models for augmented reality in liver surgery,” International Journal of Computer Assisted Radiology and Surgery, vol. 5, pp. 133–141, 2010.
Copyright © 2017 P. Vávra et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.