Abstract
Coal mining, regarded as a high-risk industry, has a strong demand for virtual reality (VR) to fulfill safety and emergency rescue training. In the past ten years, VR technology has significantly improved miner training on both the hardware and software side. However, it still has some drawbacks, such as expensive and unsuitable hardware, lack of satisfactory user experience, without direct browser access, and lack of humanized and intelligent design. To solve these problems, a cloud-based VR system is designed for the training of coal miners in this paper. The system, with browser/client architecture, includes eight modules demonstrating the full procedure of an underground coal mine. The online cloud-rendered video streaming is adopted to provide enough computing and rendering power and hence a better browser-based user experience. Furthermore, game artificial intelligence (AI) is also introduced into the system to increase the emotional exchange between the system and users. Unlike traditional VR training software, this system designs two virtual miners to enhance the experience of trainees. The first virtual miner is a task-oriented non-player-character (NPC) which conveys general knowledge about the mine and guides the users in visiting the underground work sites. The second virtual miner is a disaster-oriented character which prepares the users for typical disasters. The system has been successfully implemented in a laboratory environment, and its performance has been validated. Yet, further practices are needed to stimulate more innovative applications of VR-based miner training and disaster drilling.
1. Introduction
Coal mines, with dangerous working environments and complex production systems, have a strong demand for virtual reality (VR). VR technology has its unique advantages in several fields related to coal mining, such as miner safety training and emergency rescue drilling, besides drifting-producing process simulation, disaster scenario simulation, and mechanical operation training [1]. The fifth-generation (5G) cellular, artificial intelligence (AI), data analytics, Internet of Things (IoT), and cloud computing will provide the information infrastructure required for the intelligent mine construction in the next three to five years. With these emerging technologies, the next-generation VR systems for underground coal mining will be achieved, which will improve the professional adaptation process and occupational health and safety situation, especially for the young miners [2].
The United States, Australia, and the UK have used VR as a training environment for mining simulation, accident reconstruction and investigation, education, and safety training in the past two decades. The UK, in particular, has a long history of developing and using VR technology in coal mine safety training. The VR products, such as SafeVR and Vroom, are very famous for open-pit truck operator training[3]. The researchers of the National Institute for Occupational Safety and Health (NIOSH) explored how the mining industry could effectively use “serious games” for fire-escape training. Spokane Research Laboratory (SRL) developed fire-escape training software for a mine safety training course. The software allowed four trainees to work together in a virtual environment through a computer network. The trainees responded favorably and showed a marked improvement in locating the proper evacuation routes on subsequent trials [4]. The University of New South Wales developed iCinema (an advanced visualization and interaction environment), which included a 360° immersive theater environment with 18 modules for improving teaching and learning activities in mining engineering [5]. The University of Queensland Experimental Mine (UQEM) laboratory has been used by students for experiments since 1956. This institution developed several VR applications, including VR-based teaching modules, drill rig, Instron UCS rock testing simulations, and VR-based ventilation simulations [6].
In China, universities, large mining companies, and local governments utilize the VR hardware and software for safety and emergency rescue training [7]. Those VR systems provide the desktop versions of character “Q and A” sessions. The VR hardware is composed of a projection-based panoramic display system, the infrared tracking stereoscopic glasses, the VR headset display, a pad, and the other devices. Furthermore, the 3D visualization system is very popular in coal mines with the functions of the 3D geological model, ventilation simulation, real-time data monitoring, and emergency response.
In recent ten years, many coal mine enterprises established underground industrial networks and tried to apply cloud computing, big data, Internet of Things, and even AI technology into mining, ventilation, safety, and education projects. There has been a shift from traditional mining to intelligent mining. A three-tier IT framework with automatic control system, production execution system, and enterprise management system is established. Based on cloud computing techniques, big data related to underground coal mining can be analyzed quickly, and critical data associated with user requirements can be extracted precisely. With the established platform, the danger-spotting and subsequent decision-making capabilities of users can be greatly enhanced to ensure underground mining safety [8].
Although VR is an important part of intelligent mine construction, there are still some problems with these applications that need to be tackled. Firstly, most of the existing VR-based training systems lack a good learning experience. Although these systems adopt teaching and learning functions via a single-player mode and an ask-answer pattern, there is no complex AI man-machine interaction. VR training systems always focus on professional computation or disaster 3D simulation or text and voice presentation. The learning process needs to be improved. Furthermore, the advantages of immersion, interaction, and imagination of VR technology have not been fully utilized yet [9]. Secondly, traditional VR-based training systems are standalone or client/server versions, without direct browser access. Cloud rendering will become the mainstream to support various light terminals, such as PCs, mobile phones, tablets, and headsets. These technologies will enable real-time transmission and enhance cyber-physical and social experiences in a fully immersive environment [10]. Hence, VR training software should adopt gamification and online learning, which have become a tendency in education.
2. Methodology
To address the abovementioned problems and train the coal miners, a VR-based system using the Unreal Engine is designed and developed in this paper. Contrary to the traditional standalone or client/server VR software, this system adopts browser/client architecture and uses cloud rendering and Game AI technologies. It aims to enhance users’ learning experience, strengthen their memory, and improve safety awareness. The system uses the Unreal Engine from Epic Games, which includes a rendering engine (renderer), Blueprint scripting language, animation, artificial intelligence (AI), networking, terrain, streaming, and scene graphs. Figure 1 illustrates the system interface, which has eight modules as follows:(i)Geological environments, such as procedure of coal formation, geology condition, coal seam, and geology structures(ii)Geographical environment and surface plants(iii)Underground roadway arrangement(iv)Mining area and working face design(v)Coal mine production system, which includes ventilation, transportation, belt, lifting, power supply, water supply, and drainage(vi)Mining processes, including coal mine blasting and longwall mining processes(vii)Coal mine disaster (gas leakages, roof falls, and fire events) simulation(viii)Underground safety monitoring system (methane, wind speed, temperature, and carbon monoxide), self-rescue, and emergency response
In this paper, we use three technologies, namely, cloud rendering architecture, 3D modeling, and AI behavior design to achieve multiuser online operation and improve the interaction and immersion of VR devices.
The browser/client architecture of cloud rendering means that each user connects to the server and accesses 3D streaming over the cloud. The cloud servers handle all events and the state of the virtual world and transfer the VR render streaming to the client. These servers can be virtual servers or server clusters. They take full advantage of virtually infinite computing power to create, capture, and compress high-resolution rendered streaming, which is sent to the client with low performance and limited computing power [11].
Compared with other popular Web3D technologies, such as HTML5 and WebGL, cloud rendering has incomparable advantages mainly because it has no special or expensive requirements on the client side. It allows users to choose different platforms, including PCs, laptops, tablets, and smartphones. Furthermore, since Wi-Fi 6 and 5G will provide sufficient network bandwidth, online cloud rendering will deliver adequate computing and rendering power. With the advancements in the Internet infrastructure, the users can interact with VR programs through the network on the client with high-quality videos and low-response delays [12].
In this B/S architecture of the cloud rendering system, the system is divided into three parts: portal server, render server, and client, as shown in Figure 2.
The portal server is responsible for user login, finding the appropriate server according to the user’s choice, and controlling the server to start the virtual machine, i.e., render server. The client and the render server establish a connection when the portal server notifies the client with the address of the virtual machine. The user can run and interact with the 3D program in the remote server through the network.
The render server is responsible for processing the input events of the client sourced by mouse or keyboard, as well as translating and sending them to the interaction module of the 3D program for further parsing. The 3D scene images are captured, encoded, and sent to the client as a video streaming. The render server program includes three modules, namely, basic function, Game AI, and advanced function. The VR basic function module provides 3D scene management, sound, particle effect, physical simulation, and other basic functions. The Game AI module uses a scripting language to design a behavior tree for virtual miners. The VR advanced module builds complex computation models and simulates disasters, such as fire numerical simulation and escaping route computation.
The client is responsible for displaying the received streaming of the 3D scene and sending user inputs, such as keyboard and mouse events, to the server.
We use 3ds Max to build 3D scenes of the underground and the surface. Then, we import the 3D models into the Unreal Engine editor. We adopt a series of material tools to apply 3D models, such as a normal map, diffusion map, metal map, and texture map. These materials are like the “paint” applied to the model. Then, we set the lights to illuminate our scenes. From the computer-aided design (CAD) or geographic information system (GIS) files, we automatically build 3D geological and underground laneway models. We use the Blueprint scripting language to program the characters’ animation, the AI behavior tree, and the user interface (UI).
AI in games or “Game AI” is a module of a 3D engine. Traditional AI typically demonstrates some of the following behaviors associated with human intelligence: planning, learning, reasoning, problem-solving, knowledge representation, perception, motion, and manipulation. The emerging Game AI is becoming a narrow branch differentiated from the traditional AI. Game AI refers to the virtual man, also called non-player characters (NPCs), controlled by the computer, which simulates the intelligent behaviors of human beings or other creatures, providing users with a reasonable challenge to overcome [13].
Figure 3 shows the behaviors of the two NPCs we designed. These NPCs can simulate the behaviors of miners and guide young miners and/or students to understand the whole mining production, experience the underground scene, and improve the relationship between users and computer programs during the training, conversation, or other tasks. Besides, these AI characters are designed to automatically find the local navigation grid path.
The first NPC is a task-oriented character. When the user meets it somewhere on the ground, he will provide general knowledge about the coal mines and guide the user to descend a well from a cage and visit the underground work sites, such as longwalls or heading faces. Once the user completes the corresponding task and provides feedback, the NPC releases the next task. Those tasks are performed before entering the underground mine, preparing the miners to look for the entrance, enter the cage, and visit the underground working face. The second NPC is a disaster-oriented character which guides the user to experience typical disasters, such as gas explosions, roof falls, and fire accidents.
To realize the NPC’s behavior, the blueprint is used to build a behavior tree graph, which adds and connects a series of nodes. A behavior tree, mainly composed of a leaf node and composite node, is essentially an acyclic graph [14, 15]. We use composite tasks and design nodes of selectors, sequences, parallels, and decorators. Figure 4 shows an example of the second NPC’s behavior tree, where selector, sequence, and parallel are used to trigger accidents. A selector is a branch task that runs the behavior of each of its children. It will immediately return a success status code when one of its children runs successfully. Once one of the child nodes is successful, it returns to the selector, and the rest of the nodes are not executed. A sequence executes the node task sequentially instead. A sequence node tries to execute each child in turn until all children have reported that they have executed successfully. Once a task fails, the sequence will return to the upper node. In this case, the remaining tasks are not executed. A parallel is a composite task that handles concurrent behaviors. It is a special branch task that runs all children during execution time. The behavior tree allows parallel node tasks, which might be state machines. Multiple state machines can be executed in parallel. For example, when the second NPC’s behavior tree is triggered, it first judges whether it is dangerous or not and then runs the user and sends alerts. Following these, it gives the on-site emergency solution by a series of animation actions. The sequence executes the disaster animation in turn and tests whether the user’s reaction is correct.
3. Results and Discussion
3.1. System Implementation
The cloud rendering cluster is composed of two virtual rendering servers with a configuration of 4-core 8G M60 CPU, 32 GB RAM, and 40 Mbps BGP bandwidth. The experiments have shown that the hardware can support up to seven players at the same time. A client PC should have at least Intel i7 CPU and 16 GB RAM. The system has been tested by undergraduate students in the VR Teaching and Experiment Laboratory, China University of Mining & Technology, Beijing. The laboratory also has a 120-degree 2-channel VR projector and an infrared laser tracking system as shown in Figures 5 and 6, respectively. The system can run either on the immersive VR hardware or the PC headset.
3.2. AI Guidance
The virtual roaming function of the NPC will help the user to walk around buildings, enter the mine through a cage, and visit the longwall mining face or the blasting place. It will also introduce the ventilation system, transportation system, water supply, drainage system, and power supply system. This function enables the new miners or students to have an overall understanding of the mine.
When a user is wandering in the roadway, the accident guidance function of the NPC will start a dialogue with them to detect their behavior and provide information on illegal operations. The system will then trigger disaster animation depending on the location, and the NPC will demonstrate the escape routes for self-rescue.
3.3. Disaster Simulation
In our system, mining accidents are limited to gas explosions, roof falls, and fire events. The disaster simulation module integrates various 3D animation, audio, particle effect, and physical-mechanical models of the 3D engine. The 3D model is to establish the models of mines, terrain, tunnels, and NPCs. The audio function is to create sounds as realistic as possible considering the dimension of the source, such as footsteps when walking on the grass and the roadway, wheezing when running, and the mechanical sound of an opening cage. By analyzing the disaster/accident reasons, features, and occurrence conditions, a disaster simulation knowledge library is constructed. This library contains a brief description of prevention and control measures, emergency plans, site disposal programs, mitigation, and postdisaster evaluation. It is combined with system guidance to show the potential consequences of disasters.
3.3.1. Gas and Coal Dust Explosion
This simulation is to show the boundary conditions of gas and coal dust explosion as illustrated in Figure 7. It helps users to perceive the disaster from the vision, auditory, and other aspects and understand the movement and destruction of the wind that is mixed with dust. Besides, some emergency measures, such as collaborative gas extraction, are fully displayed.
3.3.2. Roof Fall and Wall Collapse
Through this simulation, the user will understand the main factors of roof fall and wall collapse as shown in Figure 8. It can simulate various rock burst and ground pressure disasters, introducing the form of disasters, their causes, and the potential methods of disaster prevention and control.
3.3.3. Fire Event
The user will see the main forms of fires, such as the external fires that are mostly caused by open flames, blasting, and short circuits, as well as the internal ones caused by the combustion of coal or other flammable substances due to their oxidation or burn. The VR system can simulate the conditions of various fires, illustrate their severity, and therefore inform decision making for disaster prevention. The system also shows the ways of treatment, including the closed grouting and nitrogen injection methods.
4. Discussion
The VR system uses cloud rendering and AI behavior trees to enhance training performance. It can realistically simulate the complex underground coal mine environment and provide the function of login, roaming, virtual simulation, and operation evaluation. However, due to intellectual property rights, only a small part of the functions is made accessible to the public through the website. To date (August 1, 2020), the system has been visited 20, 853 times and the training system drew perfect user reviews (5.0/5.0) on the National Virtual Simulation Experiment Teaching Project Sharing Platform [16]. According to the experiment score data, 1355 people have finished the experiment and the excellent score rate was 46.0%. The students commented that the system could give them a more intuitive understanding of the underground environment and disasters without visiting the real underground coal mines. Despite satisfactory performance and remarkable feedback, the current cloud render hardware needs to be upgraded to a larger cloud rendering cluster to provide a quicker response and hence a better experience. Furthermore, a performance test is needed to estimate the amount of concurrency and runtime efficiency. The system functions also need more research, where a more detailed behavior tree design can improve the user experience. Combined with the numerical computation, the system can be turned into a testbed to support the ventilation design, emergency drilling, and other systems and fields. We expect more innovative applications of VR-based miner training and disaster drilling in the near future.
5. Conclusions
VR, as a part of intelligence mine, provides the new tool for work procedure management, occupational health and disaster simulation, and visualization. The VR system provides macroscopic and microscopic underground 3D scenes for coal mine and displays information such as personnel positioning, safety monitoring, and equipment management. Combined with cloud rendering, it can achieve more realistic and more detailed 3D visualization effects. Considering the advantages of cloud rendering and AI, this paper designs a cloud-based VR system for mine safety training. AI guidance is adopted to roam underground and simulate accidents, such as gas explosions, roof falls, and fire events. We believe that the results of this study will provide a useful tool for coal mine safety training, avoiding the potential hazards of the extreme underground environment.
Data Availability
No data were used to support the findings of this study. The system will be considered to be openly accessed afterward.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was financially supported by the National Key Research and Development Program of China (grant no. 2016YFC0803108). It was also supported by the Key Research and Development Program of Inner Mongolia titled “Technology and application on digital mine resource management and ecological environment monitoring, 2015–2019.”