Abstract
The quality of surgical operation is very important for the patients’ safety and the reduction of aftereffects after surgery. To do this, practicing surgical operations is also increasingly important. Due to the development of hardware and simulation techniques such as virtual reality (VR), VR simulators have been developed as a method to train surgeons and medical students majoring in surgery. In addition, controllers that support various kinds of haptic feedback have also been developed for a more realistic and immersive user experience. In this study, we introduce a morphable haptic controller that provides geometric and tactile feedback to users and propose a VR simulation system using this controller. This system is designed to perform a high tibial osteotomy in a virtual environment. The morphable haptic controller attached to the system through the input mapping daemon provides the senses of multiple surgical instruments used in the simulation. As a result, the haptic controller achieves an outstanding result through an actual interaction between surgical tools and the surgical sites. We verified that this was effective in surgery training through orthopedic surgeons.
1. Introduction
Medical students can practice professional surgery and be familiar with basic surgical techniques through surgical training. Surgeons can also improve their surgical skills through training. When surgical skills are improved, the patients’ safety is guaranteed and the recovery period after surgery can be shortened. Because of this, several studies have been conducted to support more realistic and sophisticated surgery training.
One of them was to use a sample model that can replace a patient such as mannequins or animal carcasses. Medical students can practice surgical training using these models in a similar environment with a real operating room. In addition, surgeons may instruct trainees in real time or demonstrate by themselves to provide feedback on their skills. However, models once used for surgical training can no longer be used since these are already dissected. This disadvantage results in a high cost for training as well as a lack of skills for trainees. It is also difficult for surgeons to participate in the surgical training of trainees because surgeons have their own patient treatments and surgeries. To conduct training in an operating room using a real sample model, an operating room is needed to be leased. In addition, it is difficult to produce or acquire a model that is suitable for surgery.
For this reason, as the need to practice surgery training repeatedly outside the operating room increases, VR simulations are proposed as an alternative. Those can be repeated many times as needed until trainees acquire sufficient proficiency in specific surgical methods because real samples are not used. It consequently reduces the risk of patients undergoing surgery. Since the surgical training using VR is relatively easy to access and can be remotely monitored by surgeons, surgeons can provide basic trainees with feedback about surgeries without spending too much time. It can also be used to train only insufficient skills repeatedly after evaluating the proficiency of medical students by accurately recording their surgical procedures.
To maximize the advantage of VR simulations, surgeries within the virtual space should be displayed as realistic as possible. To do this, virtual environments seen by users’ eyes should be displayed realistically. Users utilize special input and output (I/O) devices such as haptic devices to interact with objects they see in a virtual space. Thus, synchronization between objects shown by the haptic device in a virtual space is also important. At this point, it is important to provide users with tactile and kinematic senses through haptic feedback to feel realistic experiences through the haptic device.
In this study, a VR simulation is proposed to train trainees with high tibial osteotomy, which is one of the orthopedic surgeries. To give a feeling of reality as if users are actually holding various surgical tools used in a high tibial osteotomy, our new type of morphable haptic controller was used [1, 2]. It is possible to present various kinaesthetic and tactile senses only by changing the thickness, length, and the center of mass of the controller without using many devices. To connect the newly developed device to the fabricated VR simulator using general-purpose graphics libraries, an extendable I/O interface was designed and implemented [2]. The effectiveness of medical education was verified by using the fabricated VR simulator by orthopedic surgeons.
This study is organized as follows. In Section 2, we introduce the VR operation simulation and haptic device proposed in this study. In Section 3, the high tibial osteotomy surgery simulation system is explained for each module. In Section 4, simulation results are described, and the conclusions are given in Section 5.
2. Related Works
Since the 1990s, Achilles’ tendon repair [3], cholecystectomy [4], and wound debridement and suturing [5] surgery simulators have been developed as a VR simulation program that can replace one-off surgical training in a real environment. Simulators in the early stages made it difficult to feel immersive due to slow feedback and heavy device as a result of low computing performance. As computer graphic processing capabilities improved, more realistic and anatomically accurate VR simulators were developed. Generally, a dedicated device was needed for a specific surgical practice in the past. However, users can practice various surgeries with a single device more recently. The NeuroTouch VR neurosurgery simulator enables various surgical simulations such as microdissections, tumor aspiration, debulking, and hemostasis [6]. The surgical training using VR simulations employs the following two methods: low fidelity, which teaches only basic surgical procedures, and high fidelity, which realistically produces a surgery. As the high-fidelity method, NeuroTouch, LapSim, and Lap Mentor were developed and tested [7]. As the technology that creates a three-dimensional model from two-dimensional images such as CT and MRI images has been developed, VR simulators using a patient-specific model have also been developed. These simulators are used in pancreatectomies, hepatectomies [8], renal surgery [9], and hand surgery [10], enabling visual communication regarding surgical planning with patients and colleagues of the operation team before the surgery.
One of the important elements in VR surgical simulations is to let trainees use their hands and follow the surgery instruction in a virtual environment while seeing the surgery scenes in a video. Thus, trainees can improve their hand-eye coordination and fine motor skills. In the VR system, a head-mounted display is used to show high-definition videos to increase the immersive feeling as well as a controller is used to receive input from a user. This controller has a button, by which the user’s command is inputted, and a built-in tracker that detects the position in the space. Some of the specialized controllers include a haptic feedback function where a kinematic sense can produce a tactile sense. The importance of haptic technology has increased because not only accurate positions of surgical tools but also their weight feeling and frictional response from the affected area should be felt during the surgery.
The haptic system is divided into three types depending on the computing platform that it uses: desktop haptic, surface haptic, and wearable haptic. Desktop and wearable haptic are mainly used in surgical simulations. A desktop haptic is a method of operating virtual tools on the screen of a general monitor by grasping and moving desktop haptic devices. On the other hand, a wearable haptic is a method, in which a user wears a haptic device that can be attached to their fingertips or hands, and then, the device simulates movement of fingertips or hands directly, thereby interacting with the objects within the virtual environment. A wearable platform is more immersive and realistic than a desktop platform. Thus, active studies on wearable platforms have increased. Typically, wearable haptic devices such as haptic gloves [11, 12] were developed to provide degrees of freedom of various movements and force feedback.
To provide haptic feedback to users through haptic devices, a haptic rendering technique is needed, which is an algorithm that connects the interaction with haptic devices in the virtual environment. The first method is a heuristic force synthesis algorithm [13, 14]. It performs simple collision detection using a set of predefined gestures. Once the collision occurs, its response is determined by the most similar one of those gestures. Zachmann and Rettig [15] configured predefined gestures through the distribution of contacts between finger phalanges, thumb phalanges, and palms and distinguished push and grasp motions. The second method is based on collision detection and accurate collision response [16]. The accurate collision detection is performed by modeling hands with hinged links and representing palms and phalanges with interconnected rigid bodies, and then, a collision response is made through penalty and constraints. The third method is a fine force algorithm [17] considering the physiological and mechanical properties of fingers. It takes into consideration the physiological structure of the human hand including bones, nonlinear stress-strain characteristics of the muscle, and detailed physiological phenomena between fingers and virtual objects. Talvas et al. proposed new aggregate constraints that simulated precise and soft movements of fingers [18].
More recently, hand-held haptic devices are more widely used such as the HTC VIVE controller instead of using haptic gloves, which are difficult to be attached and detached. Hand-held haptic devices support large-scale body movements and attaching and detaching them are simpler. Currently, these devices provide a simple level of haptic feedback only such as vibrotactile feedback. Studies on haptic feedback such as spatial-temporal vibrotactile patterns, texture feedback, thermal feedback, skin stretch, softness, and contact that can be implemented in a small volume such as hand-held haptic devices have been conducted. For example, a method using an actuator, which was composed of a single transparent and soft dielectric layer for haptic feedback through changes in the volume of the haptic device, and two transparent and soft ionic conductive layers, was proposed [1].
Studies on multimodal haptic devices that represent various attributes of virtual and remote objects have also been actively conducted to not only improve the user experience but also provide convenience and simplicity. Multimodal haptic devices simultaneously provide haptic feedback of various senses that can be felt in an object. Types of representable haptic feedback are hardness, warmness, macroroughness, fine roughness, and friction [19]. The representable geometric attributes are shape and size [20]. A haptic device called “the digital clay” tracks the movement of the hand and prepositions various stimuli in space to provide it to the user [21]. In addition, methods of providing feedback about many properties using soft robotics such as particle jamming [22] or electrostatic effect [23] or methods of making multimodal haptic devices using sensors and actuators made of new materials such as shape memory polymer [24].
Recently, many studies have suggested methods to increase the user’s accessibility while increasing the immersion provided by VR simulators. Mohsen et al. used a method of attaching markers to actual surgical tools to implement surgical tools used in laparoscopic surgical training simulations [25]. Kim et al. used two haptic devices and pedals to manipulate surgical tools in a simulation using VR and a 3D printed support fixture for training in endoscopic sinus and skull-base surgery [26]. Ferro et al. extended the simulator to use a commercial haptic controller in addition to the existing special haptic controller to increase the accessibility of the simulation developed on the da Vinci Research Kit (dVRK) platform [27]. Knopp et al. proposed a method of using an industrial robot as a haptic device to provide strong force feedback to trainees in surgeries requiring high force such as hip replacement surgery [28]. Wang et al. introduced a hand-held haptic device that provides haptic feedback by converting imperceptible forces generated in microsurgery into human-perceptible tactile signals [29]. These haptic devices provide the same tactile and kinematic senses as actual surgical instruments. However, these simulators use a single haptic device to represent a single surgical instrument. To represent several surgical tools used in a certain type of surgery, the same number of haptic devices are used. It makes the user uncomfortable while using the simulators, as well as makes the extra cost per adding a new instrument. Our simulator shows how to represent multiple surgical instruments with a single device through a morphable haptic controller.
3. High Tibial Osteotomy Surgery Simulation System
3.1. System Overview
In this study, a surgical simulator is designed and implemented by applying the process of high tibial osteotomy, used surgical tools, and how to use those surgical tools. This system consists of a haptic device that is carried by the hand of a user and an application program that provides the interaction with virtual objects in the virtual environment.
The haptic device delivers input values such as the user’s movement, button click, and rotating dial through the attached sensors to the simulation software. Data delivered by the haptic device are divided into stream data, which are continuously generated, and event data, which are generated once when a condition is met. Stream data refer to continuously transmitted data to the simulator such as position information of the haptic device. Event data are generated when a button of the haptic device is clicked or the device is hit more than a certain level of impact on it. The application program generates reactions that respond to specific events. Figure 1 shows the entire data flow in the simulation system.
The application program visualizes a virtual surgical environment and displays it on the screen. It also receives stream and event data of the haptic device and calculates the motion of the surgical tools in the virtual patient model and applies the calculation result. This system provides plug-ins and input mapping daemon (IMD) for the interaction between the VR simulator of the application program and the haptic device. The IMD, which uses a message-based interface, virtualizes the haptic device for the application program. Thus, if device developers define messages, any VR system can access the device through messages. By doing this, developer has no need to modify corresponding I/O control modules every time the haptic device changes so that it guarantees high scalability. The application program receives the position and direction data from the haptic device and applies them to virtual surgeons and surgical tools. Event data such as button clicks or collision data are used to change a surgical tool or represent contact between surgical tools and the affected area of the patient.
3.2. Morphable Haptic Controller
The proposed surgical simulator utilized a morphable haptic controller to improve the user’s immersion. Figure 2 shows a prototype of the morphable haptic controller. This haptic controller is a hand-held haptic device, which can change its length, thickness, and the center of mass. To simulate many various surgical tools, existing surgical simulators need to be connected to controllers as many as the number of surgical tools. Consequently, the simulators become complicated and require a large space for handling controllers. This is a drawback considering the current trend that requires a lightweight, simple installation, and easy use of VR simulators. Because the morphable haptic controller used in this study can be transformed into various shapes according to an object it imitates, it can simulate multiple surgical tools using a single controller.
The change of the length of the morphable haptic controller can be made through the electroactive polymer-based actuator. When a voltage is applied to the actuator, the dielectric elastomer of the actuator is expanded in the plane direction due to electrostatic attraction, in which the elasticity of the elastic layer of the actuator becomes larger than that of the dielectric elastomer, thereby changing the length of the controller as the filler inside the controller moves in the vertical direction. The center of mass of the morphable haptic controller can be moved by changing the location of mass module inside the controller.
It is possible to simulate the shape and the center of mass of surgery tools by using morphable haptic controller’s function, which changes length and the center of mass. For example, for surgical drills, its length is similar to the default length of the controller, and its center of mass is located on the middle of the controller. In other words, when a user changes a surgical tool to a drill, the morphable haptic controller maintains its length while moving the center of mass. As a result, the morphable haptic controller makes the user feel the length and weight of the drill like real. As another example, the center of mass of the hammer is located at the head of the hammer. Thus, if a surgical tool is changed to a hammer, the morphable haptic controller is extended through the length change actuator then the mass is located in the tip of the controller so that the user feels the length and weight of the hammer. The movement of the center of mass in both cases is presented in Figure 3.
(a)
(b)
3.3. Simulation Software
The standard I/O device interacts with the application program using the standardized software development kit (SDK) and application programming interface (API). On the other hand, nonstandardized I/O devices such as haptic controllers should interact with users using the original API or SDK rather than using the standardized SDK. When its own API or SDK is used, commercial haptic controllers provide plug-ins to conveniently use API or SDK. However, haptic controllers under development or new haptic controllers do not provide such plug-ins. Our system was designed to use both commercial controllers with plug-ins and morphable haptic controllers without plug-ins.
When our system is run with the commercial haptic controller where plug-ins are supported, developers can interact with haptic devices through information transferred from plug-ins without knowing detailed information about SDK or API. As the proposed VR simulator was implemented in the Unity engine environment, commercial haptic controller can also exchange I/O signals by using Unity plug-ins which is able to interact with the simulator. The data flow of a haptic device with plug-in support is shown in Figure 4.
However, developers should provide an I/O management module by modifying the current SDK if a haptic controller with sensors and actuators does not support a plug-in. Designing to control haptic controller directly by application program causes a problem that management module should be modified every time device changes. Furthermore, in case the I/O management module is dependent on the application program, the application may incur latencies during handling device I/O due to processing other tasks.
The proposed surgical simulator allows signals to be exchanged between the morphable haptic controller and the application program with IMD [2]. Figure 5 shows the data flow of the morphable haptic controller using the IMD. In the IMD, data and events provided by the morphable haptic controller, and mapped event handlers to handle events are defined. When the morphable haptic controller sends the data generated by the sensors to the driver, the data are transferred to the IMD via the original SDK. The IMD analyzes the data and checks whether the predefined event occurred. Once the event is occurred, the event handler that is mapped for that event is executed, and the result data are transferred to the application program. The VR simulator then accesses the data through the IMD SDK and reflects this to the VR simulation. On the other hand, the VR simulator may send event data to the morphable haptic controller through the IMD SDK to perform specific operations such as changing the length or moving the center of mass. The IMD handles the event data through the event handler in the same manner as when it receives the event from the morphable controller. The output data of the event handler are transferred to the morphable haptic controller via the device driver, and the hardware function mapped to that event is performed. This VR simulator uses the functions of the haptic device, which does not support plug-ins, through the IMD, thereby providing haptic feedback to trainees.
4. Results
The experiment was performed on a desktop with an Intel Core i5-8500 3.00 GHz central processing unit and 32 GB of memory. For the graphics processing unit, an NVIDIA GeForce RTX 2070 SUPER with 8 GB of memory was used. The VR simulator was implemented using the Unity engine.
Figure 6 shows the simulation results of the developed VR surgical simulator. We represented various surgical tools needed in each step of the operation in the same way as a real operation while conducting a virtual operation using our morphable haptic device. Users can employ various tools in each step of the operation in the same way as a real operation. We also verified that the controller was extended or reduced to a similar length as the real tool. In addition, we verified the change in the center of mass by moving the mass of the morphable haptic controller to feel the weight of the real tool.
(a)
(b)
(c)
(d)
(e)
(f)
The morphable haptic device provided the user with the sense of surgical tools. To give the sense of a drill, the center of mass was moved to the handle, and at the same time, the shape of the haptic device was bent to resemble a drill. Subsequently, when using a bone saw, the bended haptic device was restored to its original state to imitate the shape of a straight bone saw. In the process of using a hammer during the surgical procedure, the length of the device was changed to provide the user with a sense of the shape of a hammer longer than a bone saw. In addition, the center of mass was moved from the handle to the tip of the device to represent the center of mass of the hammer. It was confirmed that the length, bending deformation, and move of the center of mass occur depending on the tool used in surgical simulation. As a result, several surgical tools could be simulated with one device. By expressing several surgical tools as one haptic device instead of using one haptic device for one surgical tool, it was possible to reduce the actual space occupied by the haptic devices and increase user convenience. Moreover, the haptic device can be morphed to imitate the shape, weight, and movement of each surgical tool and provide the sense of using surgical tools without changing devices.
Recently proposed haptic devices use a method of using a real surgical tool to provide specific haptic feedback [25] or a method of using a special equipment such as a robot to convey the feeling of the corresponding surgical tool [28]. These methods provide the haptic feedback required by the surgical tool with considerable accuracy. However, each haptic device in these methods represents only one type of surgical tool. For learning about the entire surgical process, whenever a surgical tool is added, a haptic device for it is added. Thus, Additional costs are incurred, and user convenience is decreased. On the other hand, our simulator using a morphable haptic controller delivers haptic feedback for multiple surgical tools used in the entire surgery through a single device. As long as the user knows how to use the device, the user can learn how to use a variety of similar surgical tools by the simulator. As a result, the cost is reduced, and user accessibility is increased.
To prove the educational effect of the system, we surveyed the experiments with orthopedic surgeons. The result of the survey is presented in Table 1. We verified whether users felt that the VR surgical simulation is matched with real high tibial osteotomy through the survey. We also verified that surgeon trainees can get the educational effect of real operation through the VR surgical simulation. As a result of requesting feedback from orthopedic surgeons, it was confirmed that the simulation clearly indicated the operation sequence of the high tibial osteotomy and helped to learn surgical techniques such as selection of surgical tools and basic usage of surgical tools. In addition, it was confirmed that this simulation clearly informs the surgical procedure to be performed next as well as the surgical goal and also provides appropriate feedback when the learner selects the wrong tool or performs surgery in the wrong way. Normally surgeons operate using both hands so if controllers were used to handle virtual surgical tools with both hands, it would be possible to acquire techniques more similar to the actual operation.
5. Conclusions
In this study, we proposed the VR simulation system that performs complex operation with various surgical tools by the morphable haptic controller. Providing various types of haptic feedback to users is an important element for users to acquire realistic experiences from the VR simulation. The proposed VR simulation system can provide many experiences to users through the method that supports various haptic controllers. We verified this support can provide educational effects on real surgeries. The VR simulation system proposed in this study can be applied to other educational simulations. By applying this system, a system that can use specialized haptic controllers, which provide suitable haptic feedback to VR educational simulations in various fields, can be developed. This kind of system will provide a more realistic experience and educational effects to users.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The authors would like to give our deepest gratitude to Prof/MD. Sangmin Park for his review of this VR surgical simulation. This work was supported by the INHA University grant.