A Cross-Platform Web3D Monitoring System of the Three-Machine Equipment in a Fully Mechanized Coalface Based on the Skeleton Model and Sensor Data

Lu, Zhengxiong; Guo, Wei; Zhao, Shuanfeng; Zhang, Chuanwei; Wang, Yuan; Jia, Yanping

doi:https://doi.org/10.1155/2020/3147352

Journal of Sensors

On this page

Abstract Introduction Implementation Conclusions Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2020 | Article ID 3147352 | https://doi.org/10.1155/2020/3147352

A Cross-Platform Web3D Monitoring System of the Three-Machine Equipment in a Fully Mechanized Coalface Based on the Skeleton Model and Sensor Data

Zhengxiong Lu,¹Wei Guo,¹Shuanfeng Zhao,¹Chuanwei Zhang,¹Yuan Wang,¹and Yanping Jia¹

Academic Editor: Abdellah Touhafi

Received26 Nov 2019

Revised30 Oct 2020

Accepted12 Nov 2020

Published01 Dec 2020

Abstract

A fully mechanized coalface is a rugged environment that has poor visibility. The traditional video monitoring system has problems such as a lack of realism, a blurry monitoring effect, and poor reliability. It is an important task to monitor the operations of the three-machine equipment (we will refer to the shearer, hydraulic support, and scraper conveyor as the three-machine equipment) intuitively, accurately, and timely and ensure that it is operating safely. This study proposed a cross-platform Web3D monitoring system for the three-machine equipment. First, the virtual mesh model and skeleton model that was embedded in the mesh model were established according to three-machine ontology and basic motion units. Second, the kinematic model of the three-machine skeleton was established via the inverse kinematic modeling of the hydraulic support and the coordinate calculation of the vertices on the three-machine skeleton. Third, the motion data, which were captured by sensors, were applied to drive the movement of the three-machine skeleton and mesh model. Finally, WebGL, which is the latest Internet graphics standard, was used to render the three-machine models, and the performance of this monitoring system is tested on different equipment in the laboratory. The results of the test show that the three-machine cross-platform monitoring system has splendid performance, and it realizes cross-platform 3D monitoring effectively in the laboratory. In the future, this system will be used as a supervisory tool and be integrated with the traditional monitoring system to monitor the three-machine equipment with the field staff.

1. Introduction

Mining coal is one of the most important fundamental industries, which plays a crucial role in the development of the national economy. However, due to the poor environment and abstract monitoring results of the mining process in the fully mechanized coalface, technicians not can quickly and intuitively get the details of fully mechanized mining equipment in the mining process. It is difficult to have a full observation and a further comprehension on the operating details of mining equipment in a working face, which leads to the possibility of the emergence of various equipment faults and threaten the lives of staff, which has a direct impact on the coal production efficiency. If the whole working process is displayed intuitively in a more realistic manner, then the equipment reliability, the working efficiency, and security will be greatly improved.

With the advances in computer technology nowadays, virtual reality (VR) technology has been rapidly developing and, if combined with sensing technology of the three-machine equipment, shows great potential for remotely visualizing the entire mining process for improving the efficiency of production and reducing the number of miners in the fully mechanized coalface. Currently, VR technology is used in the field of coal mining because of its intuitive, immersive, and interactive features and has led to many research advancements [1–3]. Tichon et al. [4–6] applied VR to miners’ safety training and effectively improved their safety level. Stothard [7, 8] designed a VR coal industry training simulator that allows miners to experience the consequences of decision-making errors and learn from their mistakes. Based on complete simulation data, Akkoyun [9, 10] established an interactive mining and engineering visualization environment for teaching that modeled an open pit of magnesite. Torano et al. [11] simulated the behavior of the roof in longwall mining and realized the 3D display of the longwall mining face using the VRML language. Wan and Xu [12, 13] established a simple GUI interactive interface to simulate the mining process of a fully mechanized mining face. Zhang [14] obtained dynamic images that were a fusion of real images and a virtual scene of the fully mechanized coal mining face. These dynamic images had better results that were closer to reality. However, the functions of them are still relatively monotonous, which is mainly used for teaching and demonstration. These technologies have not been used to studying collaborative work of the three-machine equipment. Foster et al. [15, 16] conducted a training operation simulation for underground miners in an interactive and immersive environment, where the simulated continuous mining machine and drilling machine could be operated to train remotely. Yan et al. [17] used dynamic analysis and fast solution technology to synchronize the working state of a hydraulic support. Zhang et al. [18, 19] established a series of virtual mining models that included a shearer, scraper conveyor, and hydraulic support and simulated the entire kinematic process and technique of a fully mechanized coalface. Xie et al. [20] studied the key technologies of the virtual cooperative operations of fully mechanized mining by establishing a panoramic scene of fully mechanized virtual reality and more realistically reproduced the operating status of the three fully mechanized mining machines. However, none of these systems satisfies the demand of the directly displaying operation condition of the three-machine equipment in real time.

So far, many systems based on VR have been developed. Liao et al. [21] presented a collaborative and interactive design framework of a MEMS design system used in dynamic web Java Server Pages and computer graphics virtual reality modelling language. Birr et al. [22] established a real-time interactive Web3D system to learn 3D models based on WebGL and obtained a good demonstration effect. Sun et al. [23] developed a virtual reality system (VRCast) for the casting production process using WebGL technology and performed a virtual display of the casting process in a network environment. Bian et al. [24] developed a Web3D exhibition building system to build Web3D exhibitions in web applications, which has good compatibility and runs depending on the GPU without any plug-in. Xia et al. [25] established a zero plug-in for the cross-platform EAST visualization system based on WebGL, which allows users to browse the EAST device model and visually monitor the EAST experimental data through a web browser. In recent years, the rapid development of Internet technology, particularly the smart mobile technology, has penetrated into every aspect of life. It is a great challenge to combine Internet technology with VR to get a more powerful system.

As mentioned above, the VR simulations in the mining field focused on aspects including safety training, operation process simulation, kinematics simulation, and visualization of the underground scene. It has been used mainly in the training and teaching fields and does not have as much prevalence in the industrial field. Therefore, the application level of VR technology in the mining industry is still relatively low, as there is no real industrial application. Web3D monitoring has not yet been applied in the mining industry. With the rapid development of Web3D applications, Web3D technologies such as OpenGL, VRML/X3D, Java3D, Cult3D, Flash3D, Unity3D, and WebGL are constantly appearing. Each technology has its own advantages in rendering speed and interactive performance [26–29]. As an open-source 3D scene rendering API, WebGL not only displays 3D virtual scenes without plug-ins and delays in the browser through the GPU but also has good interactive performance and wide applications [30]. Hence, in view of the advantages of virtual reality and Internet technology, this research has developed a lightweight Web3D system, namely, a cross-platform Web3D monitoring system of the three-machine equipment based on multisensor data. It is convenient to be use without platform limit. In order to solve the difficulties of 3D graphics rendering in web browsers and the lightweight requirement of no plug-ins, the latest Internet graphics standard WebGL was used in the proposed system.

A fully mechanized coalface is the most critical link in the coal industry. Therefore, based on the above analysis, this paper will establish a Web3D monitoring system that can solve the problem of cross-platform remotely monitoring the three-machine equipment as a foundation that complements the traditional monitoring system in the fully mechanized coalface. This advancement will help miners quickly make decision in the remote control center and increase production. The remainder of this paper is organized as follows: Section 2 presents the structure of the Web3D monitoring system. In Section 3, we present the key technologies of establishing the kinematic model of skeleton model of the three-machine. Section 4 amply describes the implementation process. In Section 5, the performance of Web3D monitoring system is tested. This paper is concluded in Section 6.

2. Architecture Overview

A cross-platform Web3D monitoring system of the three-machine equipment is established in the laboratory as a supervision tool to complement the direct surveillance by workers. The structure of the monitoring system of the three-machine equipment is shown in Figure 1.

The system adopts the B/S (Browser/Server) structure, which has good compatibility, excellent cross-platform capabilities, and a perfect interactional mechanism. The data on the 3D model, scene, and working information of the three-machine equipment will be displayed in this system. The positions of the virtual model of the three-machine equipment are determined by the coordinates in the virtual scene. When the user accesses this system, the web server sends the HTML file, the CSS file, and the JS file to the client browser through the HTTP protocol; then, the browser-based WebGL renders the working state of the three-machine equipment. To achieve real-time display, we apply a perceptive system with sensors to obtain the working information of the three-machine equipment, which will be conveyed to three-machine equipment’s S7-1200PLC. The working information is stored in the SQL Server database via the OPC technology, which is processed by a program that calculates the state model of the three-machine equipment to reveal its operating process.

In this study, MVC technology was used to build the system’s program framework, and the structure is shown in Figure 2. The view layer uses the WebGL graphics technical standard that invokes OpenGL ES and the Three.js graphics function library through the JavaScript language. The GPU acceleration technology is used to render the 3D model file of the three-machine equipment on the HTML5 canvas. The controller receives the user instruction, calls the model, and then calculates the three-machine equipment’s working status, which provides the condition for the driver of the three-machine model. The 3D model files and running position files of the three-machine equipment are encapsulated in the model layer.

3. Key Technologies

3.1. Kinematic Model of the Three-Machine Equipment

Effective monitoring results will provide information on the position of the three-machine equipment in real time. The Web3D monitoring system includes two key technologies. The movement of 3D models must conform to the kinematic characteristics of the three-machine equipment in order to obtain a realistic image of the three-machine equipment. To ensure the smoothness, the rendering speed of the three-machine equipment must be fast enough. The time delay in rendering the real three-machine model cannot be too high; otherwise, the server will not be able to calculate the situation of the three-machine equipment in a timely manner, which can result in incorrect judgments and production accidents.

To realize the real-time 3D monitoring of the three-machine equipment, we use the WebGL-based lightweight 3D development framework Three.js and the “skeletal skin animation” technology [31]. The experimental platform sets the MG300/700-AWD shearer, the ZF6800/17/32 hydraulic support, and the SGZ-764/630 scraper conveyor as the original objects, and it sets the shrank pro rata as 8 times to complete the construction of the model according to the matching relationship of the three-machine of the fully mechanized coalface. According to the design drawing of the experimental platform, the 3D model of the fully mechanized mining machine is established at a 1 : 1 ratio in the 3D modeling software SolidWorks. Then, the 3D model is exported to the 3DMAX software in the STL format to complete the repair and optimization of the model. In particular, the connection constraint components between the three machines must be accurately modeled as the actual constraint relationship to ensure that the components remain in the assembled relationship in SolidWorks, and the three-machine equipment’s operating relationship [32] remains correct after the STL file of the fully mechanized three-machine model is imported into 3DMAX. The three-machine physical experiment platform and the virtual 3D model of the fully mechanized mining are shown in Figures 3(a) and 3(b), respectively.

(a)

(b)

The mathematical description of the positional relationship between the moving parts of the 3D model of fully mechanized mining is the basis for establishing its kinematic model. In this study, a Cartesian coordinate system is established at the moving nodes of each component to describe its positional relationship. The pose transformation of each moving part of the three machines is studied using a homogeneous coordinate transformation matrix. The form is shown in formula (1). The homogeneous matrix can describe the motion transformation of each moving component in the three-machine linkage process and intuitively represent the rotation and movement of the moving component in the Cartesian coordinate space. The parts when they are continuously rotated or moved are described using the matrix’s multiplication form.

As a link mechanism, the hydraulic support sets its own local coordinate system at the lower hinge point of each link, and the sublinks are affected by the movement of the parent link. The relative motion between the links is described in the form of a parent-child relationship chain, and the last-level guard panel inherits the transformation of the multilevel coordinate system. Each section of the scraper conveyor, the corresponding shifting lever, and the shifting cylinder are regarded as independent moving units. A local coordinate system is set at the hinge of the middle slot and the shifting lever to describe the displacement of the central slot. A local coordinate system is set at the hinge point of the shifting cylinder and the hydraulic support to describe the displacement of the hydraulic bracket, and each moving unit combination completes the production operation of the scraper transporter. The shearer is abstracted as a linkage mechanism, and the drum will inherit the transfer transformation of the multilevel coordinate system.

According to the movement characteristics of each moving part of the three-machine equipment that conducts fully mechanized mining, the relative position and attitude between these coordinate systems are described by a homogeneous coordinate transformation. We define homogeneous coordinate transformation matrix to describe the relative translation and rotation between adjacent connecting rod coordinate systems. We set to represent the position of the first connecting rod relative to the basic coordinate system and to represent the position of the second connecting rod relative to the first connecting rod. Then, the position transformation matrix of the second connecting rod relative to the basic coordinate system is as follows: . These steps repeat. The posture transformation matrix of the third connecting rod relative to the basic coordinate system is , and the posture transformation matrix of the nth connecting rod relative to the basic coordinate system is .

3.2. Establishment of the Three-Machine Skeleton Model

The skeleton model involves translation, rotation, and scaling, which can skillfully realize the motion process of the moving parts of the three-machine equipment. Therefore, it is necessary that the skeletal model of the three-machine equipment is established in order to drive the 3D model of the three-machine equipment. The skeleton models of the shearer, hydraulic support, and scraper conveyor are shown in Figures 4–6, respectively.

The establishment process of the skeleton model of the three-machine equipment is as follows. First, the structure of the skeleton model is abstracted according to the connective position of the pin shaft in the shearer, hydraulic support, and scraper conveyor. Then, the skeleton is drawn according to the structure of the three-machine equipment using 3DMAX. Finally, the skeleton model of the three-machine equipment is established by the joints connecting all skeletons. In addition, the binding of the skeleton and the 3D virtual model is established. The “envelope” function is used to bind each bone and its mesh of the three-machine equipment. The weight between the bone and the skin vertex is adjusted according to the effect of the skin traction action of the bone to realize the fine control of the current frame vertex [33].

3.3. Establishment of the Kinematic Model of the Skeleton Model

The three-machine skeleton model provides the foundation for the data drive of the three-machine virtual model. To realize the synchronous movement of the virtual three-machine model and its ontology, it is necessary to realize the synchronous motion of the skeleton and the ontology of the three-machine equipment.

3.3.1. Calculation Method of the Vertex Coordinate in Moving Skeleton

Given the world coordinate system of the virtual 3D space, is the coordinate of the origin of the parent coordinate system in the world coordinate system . is the coordinate of the origin of the subcoordinate system in the world coordinate system . The coordinate that is a vertex of the mesh is in the subcoordinate system , as shown in Figure 7.

When the skeletons are stationary, the coordinate of vertex is translated from the subcoordinate system to the parent coordinate system using formula (2). where is the translational matrix, and is the transformed vertex coordinate from the subcoordinate system to the parent coordinate system.

The coordinate of vertex is translated from the parent coordinate system to the world coordinate system using formula (3). where is the translational matrix, and is the transformed vertex coordinate from the parent coordinate system to the world coordinate system.

Therefore, the translational process of vertex of the mesh from the subcoordinate system to the world coordinate system is as follows:

Moreover, the motion process of skeletons is analyzed. The motion process of the skeletons is resolved into the rotational and translational motion in its local coordinate system, which is described by a single four-dimensional homogeneous matrix using formula (1). The motion matrix in the local coordinate system of the child skeletons is . The motional matrix in the local coordinate system of the parent skeletons is . The motion transformation of the child skeleton in the local coordinate system is converted to the parent skeleton coordinate system as follows: where is the coordinate of vertex in the parent coordinate system .

When coordinate of vertex is transformed into the world coordinate system , a new coordinate of vertex is obtained in the world coordinate system after bone movement. The transformational process is as follows:

Although the chain hierarchy of coordinate systems is complex, we can obtain the position of the vertices that are bound to the movement bone in the local coordinate system and calculate the position of the vertices in the world coordinate system after motion. When the vertices of the three-machine mesh model are traversed and calculated using the same method, the positions of the three-machine model are updated.

The updated coordinates of the vertexes that are bound to the skeleton are calculated. The initial coordinates of the vertices of the mesh model in the world coordinate system are converted into the local coordinate system of the bone where the vertices are located via coordinate transformation. Take Figure 8 as an example for analysis. Set the coordinate of vertex at the initial position of the world coordinate system as , and the world coordinate of vertex that is the origin of the local coordinate system of vertex V is .

The coordinates of vertex in its local coordinate system are calculated by formula (7). Its new coordinates are :

Thus, the coordinates of the grid vertices in the world coordinate system in their local coordinate system can be expressed as the product of matrix and the homogeneous coordinate system transformation matrix . The comprehensive reasoning shows that the world coordinate system of mesh vertices when the skeleton is moving is

Formula (9) implements the grid vertex coordinate transformation of two adjacent levels of the skeleton, and the method for calculating the vertex of the multistage skeleton motion is the same.

The initial state of the vertex motion of the coordinate of is the position in the world coordinate system, and the coordinate can be read from the grid model file. All vertices can be compared using the above formula to calculate the new coordinates after skeleton movement to update the position of the entire mesh vertex and realize the movement and rotation of the three-machine virtual model. The transform matrices between multistage skeletal coordinates, such as , and the initial values of the world coordinates and of the origin of each skeletal coordinate system can be obtained through the grid file. However, a matrix of motion transformations that occur at all levels of the skeleton under their respective local coordinate systems, such as and , cannot be obtained from the grid file. Without these skeleton transformation matrices, it will not be possible to realize the mathematical calculation of the model’s motion transformation. In the virtual 3D space, the motion of the 3D model of the three-machine equipment is driven by the motion of the skeletal model embedded in the mesh skin. Therefore, in order to control the motion of the 3D model of three-machine equipment, it is necessary to study the motion transformation calculation method of the joints in the skeleton model. The movement of the parts of the three-machine equipment includes the rotation around the axis and the movement in a specific plane, and the rotation movement mode is shown in Figure 8. Movement is achieved by the stretching skeletons. In the figure, , , , and are pin shaft nodes, and “,” “,” and “” are skeletons. Suppose that skeletons “b” and “c” rotate around pin shaft node by angle . The coordinate of point is , the coordinate of point is , and the coordinate of point is . This transformation is calculated using the axial angle method [34].

Regarding vector and vector , this rotation can be expressed as a counterclockwise rotation of vector around point , that is, . The direction is vertical plane outward. The transformation matrix of skeleton “” rotating by angle at point is , and the new coordinate of point point after the rotation is ; the pseudocode programmed with the Three.js function is as follows:

vec3 BA = new THREE.Vector3(x_a-x_b,y_a-y_b,z_a-z_b)//Compute vector BA;

vec3 BC = new THREE.Vector3(x_c-x_b,y_c-y_b,z_c-z_b)//Compute vector BC;

vec3 BA×BC = new THREE.Vector3.multiplyVectors(BA, BC)//Calculate the fork product of vector BA and vector BC, and calculate the normal vector of plane ABC;

vec3 normal = normalize(vec3(BA×BC))//The normalized of normal vector of plane ABC;

M₁ = new THREE.Matrix4.makeRotationAxis(normal, φ)//A 4-order rotational transformation matrix M₁ is calculated to rotate the angle φ of normal rotation around normal;

vec3 BC = normal.transformDirection9(M₁)//Vector BC makes rotation transformation with matrix M₁.

The local coordinate system of the skeleton is located at connection point of skeletons “” and “”, and so, vector is obtained after skeleton “a” rotates by angle around the normal in the local coordinate system, namely, is the coordinate value of point in the local coordinate system. Similarly, and can be calculated by using the method of calculating .

3.3.2. Solution of the Reverse Kinematics of the Hydraulic Support

When calculating the rotation transformation matrix of a skeleton in its local coordinate system, the angle that each moving part turns around the pin shaft must be clearly given. As a typical four-link mechanism, the hydraulic support rotates around the shaft of the connecting rod by means of a hydraulic cylinder hinged on it. The sensor can only collect the elongations of the hydraulic cylinder and piston rod, and it cannot directly obtain the angle information. This requires the reverse solution of the rotation angle information of each hinge point according to the actual size of the hydraulic support connecting rod, the mounting position of the hydraulic cylinder, and the elongation information of the hydraulic cylinder piston rod collected. According to the actual structure of the hydraulic support of the three-machine collaborative experimental platform, we establish the coordinate system, as shown in Figure 9. Through the motion model of the linkage mechanism of the hydraulic support, according to the height of the hydraulic support, length of the hydraulic cylinder of the front beam, and length of the end guard plate hydraulic cylinder, the axes of the moving members around the local coordinate system at the corresponding hinge point are _, …, and , and the angles of the turns are ,..., .

According to the dimensional relation of the connecting rod mechanism of the hydraulic support, as shown in Figure 9, the movement of the connecting rod mechanism of the hydraulic support during the production process of fully mechanized coal mining meets the following equation: where is the length of rear link, is the horizontal inclination of the shield beam, is the distance between the hinged points on front and rear links, is the length of the cover beam, is the vertical inclination of the rear link, is the height of the upright column, and is the vertical distance between the hinged point and the base surface under the rear link. The formula to calculate angle is where is the length of the front link, is the distance between the upper hinge point of the front link and the shield beam, is the vertical inclination of the front link, and is the vertical distance between the hinged point and the base surface under the front link. The formula to calculate is

The geometric relation of Figure 10 is calculated to obtain : where is the vertical distance between the lower hinge point of the cover beam jack and the top beam, is the distance from the hinge point on the cover beam jack to the turning point , is the distance between the turning point and the base of the hinge point of the top beam jack, and is the length of the cover beam jack. where is the vertical distance between the lower hinge point of the face guard jack and the cover beam, is the distance from the hinge point on the face guard jack to the turning point , is the distance between the turning point and the base of the hinge point of the cover beam jack, and is the length of face guard jack.

During the operation of the hydraulic support, the angles _, …, are the dynamic changes with the height of the support column, the length of the cover beam jack _, and the length of the cover beam jack . According to the homogeneous transformation matrix of the skeleton motion of the three-machine equipment, the rotation angle of the links of the hydraulic support is substituted into the relation formula (9). The positions and postures of the virtual three-machine mesh model can be obtained during the motion process of the virtual three-machine mesh model.

4. Implementation and Demonstration

The implementation process of virtual three-machine rendering is shown in Figure 10. First, the 3D modeling of the three-machine equipment is established. Then, the real texture of the device is processed. Next, the model is placed in a virtual environment. Finally, the scene is rendered with JavaScript and the third party engine library of WebGL, Three.js. The software system analyzes the collected running state data of the three-machine equipment and generates the displacement or rotation angle information that drives the bone movement, realizes the real-time driving of the model, and enables users to easily observe the real-time production process.

4.1. Model Visualization

In this work, the model of the three machines is established using the SolidWorks modeling software. Finally, the model data are rendered and displayed on the web page. The process is shown in Figure 11. The visualization method includes two levels. On the macro level, first, the 3D modeling of the three-machine equipment was established; then, the model file was converted from a DWG format to a unified STL format, which was used as the external call file of the web page. Finally, the model was rendered and displayed in the client browser by calling the JavaScript-based WebGL execution file. In the program coding layer, the external call file in the STL format was named according to the rules, and the model data were read in and parsed by JavaScript. In addition, the size of the model enveloping box and the coordinates of the original position of the model were calculated. The model was named and numbered with an ID, and the model material was uniformly added. Then, the mouse event handler was added. Finally, the model display and interactive operations were realized through the combination of HTML and CSS.

4.1.1. Parsing Model Files

In model visualization, the model data are read, and the geometric information of the model is extracted. In this work, STL files are stored via ASCII format. The files recorded the triangular face data describing the 3D model, including the vertex coordinates and normal vectors of the triangles. The format of the file is very readable and easy to program, code, read, and parse. Figure 12 shows the 3D model of the slot rail seat in the middle of the scraper conveyor and its STL format file. The file starts with the keyword “solid” followed by the name of the software. Subsequently, that data block describes the normal vector, and the vertex coordinates of a triangular facet are described using the keyword “facet…Endfacet.” Finally, the file ends with the keyword “endsolid.” In the STL file, the vertex coordinates of the triangular faces are all positive, the normal vector of the triangular faces is outward, and the vertex data of the triangular faces are arranged counterclockwise. According to the characteristics of the STL format files, when parsing STL files, it is necessary to process faces. To obtain the normal vector and coordinate data of the surface quickly and effectively, the regular expression is used to match the keywords, which help to store the normal vectors as strings in arrays; then, the important floating point data of faces are obtained by the JavaScript function “parseFloat.”

4.1.2. Visualization of the Model

After obtaining the relevant data of the triangular faces of the three-machine models, the data need to be transmitted to the graphics card for calculation, and then, the model visualization is completed. The process is shown in Figure 13. The vertex data of the triangle faces that are parsed using JavaScript are passed to the graphics card through the buffer object. Subsequently, the vertex shader is called to complete the primitive assembly and rasterization. The pixel color is written into the color buffer by the fragment shader. Finally, the three-machine models are displayed in the browser.

Through the transformation formula of the vertex coordinates (17) [35], the vertex shader transforms the coordinates, transforms and unites the plane normal, transforms the texture coordinates, calculates the model view matrix and projection matrix, and calculates the light and shadow to realize the vertex-by-vertex description of the three-machine model. The graphic assembly module assembles the isolated vertices of the model into geometric figures. The assembled geometric models are converted into faces via rasterization and processed by the fragment shader for each face so that the pixels of the three-machine model can be displayed on the screen. where (, , ) are the coordinates of the pretransformed vertex; are the coordinates of the transformed vertex; is the model matrix, which realizes the rotation and scaling of the model; is the view matrix, which represents the scene observed by the observer; and is a perspective projection matrix that displays the model in space.

4.2. Visualization Demonstration

In this work, SolidWorks is used to model the three-machine equipment, and the model is exported as an STL file. That STL file is parsed, and the model visualization and mouse event handler are realized by JavaScript to run the three-machine monitoring system in the browser. The three-machine monitoring system is divided into the main view area and the status area, as shown in Figure 14. The main view area displays the three-machine model in real time, and the user can roam the scene in different perspectives. The interaction area is used to display the environment status of the working surface.

Mouse clicks and movements need to be registered to display the model in different perspectives. When the main view window is clicked on, all models are selected and lightened. When the model selection is completed, the program needs to capture the mouse movements and give feedback to render the model in the specified perspective.

5. Performance Test

To evaluate the performance of the system, Apache HTTP Server 2.2 is used as the web server, and gzip is used to compress the data files. The wireless router is configured to test the system’s performance on a mobile terminal. The operating systems tested include Windows and Android. The browsers include Google Chrome and Android browser. The details are shown in Table 1.

When the number of display frames per second is more than 25 frames, continuous animation will be generated. Therefore, we set the data acquisition frequency of each displacement and angle sensor of the fully mechanized three-machine body to 25 times per second to prevent the fast data acquisition frequency from causing lag when rendering the browser scene model. At this time, the browser will smoothly render the 3D animation of the motion process of the three-machine equipment. The experimental platform is shown in Figure 15. When the three-machine equipment operates, the sensors will collect and transmit the motion data of each moving part of the equipment to the PLC. According to the predetermined transmission protocol, the PC server will encode all the motion data collected by the PLC at the same time and then transmit the encoded data over the LAN. The monitoring computer analyzes the motion angle and displacement information of each moving part of the three-machine equipment after receiving the data. The motion data are then transmitted to the nodes in the skeleton model. The angle and displacement data of each motion node are inserted into the inverse kinematics solution model of the hydraulic support and the joint motion transformation formula of the skeleton model to calculate the local transformation of each joint, and then the calculation result is further inserted into the joint chain transformation matrix (9) to calculate the coordinates of each vertex motion transformation on the mesh model.

First, the shearer, hydraulic support, and scraper conveyor are loaded separately when the three-machine equipment is not started. The results (see Table 2) showed that the rendering time is approximately 17 ms, and the rendering rate is close to 60 fps. The animation of the model is smooth. The animation data are updated synchronously during dynamic loading. The fastest refresh frame rate of animation can reach 58 fps (more than 25 fps) in the virtual scene. The model can be rendered quickly and the animation is very smooth.

The frame rates of the desktop PC, laptop PC, and Huawei Nova 3 render different numbers of faces, as shown in Figure 16. The analysis results show each device that can maintain a very high rendering frame rate when the number of faces is less than a certain number. However, when the number of faces exceeds a certain number, the rendering frame rate will drop sharply. This is mainly related to the hardware performance of the device; that is, the rendering effect depends on the comprehensive performance of the CPU and GPU. The CPU is mainly responsible for calculating the position of the animation runtime model, and the GPU is responsible for rendering the model. In this case, the performances of the desktop PC and laptop PC are higher than that of mobile phone. When the frame rate of the animation is 25 fps or higher, the picture is considered to be relatively smooth. On the desktop PC, laptop PC, and mobile phone, the horizontal coordinate values are 5.43, 6.75, and 5.64, respectively, when the frame rate is 25 fps. The average value of these three is 5.94. We can calculate the maximum number of faces that can be rendered on the above equipment as approximately 870,000 (10^5.94). The numbers of faces on the 3D models of the shearer, hydraulic support, and scraper conveyor are far less than the maximum value calculated above. Therefore, the system can render the three-machine model smoothly, and it can also obtain an excellent user interaction experience in the laboratory.

6. Conclusions

A cross-platform Web3D monitoring system for three-machine equipment was developed in this work. As the proposed system is based on the B/S mode, it allows many users access to the system at the same time, and it is convenient to upgrade the system. In this system, the latest Internet graphics standard WebGL is used to render the models and the virtual panoramic scene. By using WebGL, the system achieves acceleration on hardware level in web browsers. It is characterized as cross-platform and portable, which could run in almost any platform and device. And it has a strong ability to rendering models with high performance and graphical quality on different devices. The proposed system displays the virtual working process which keeps up with the working process of the three-machine equipment, which makes operators directly monitor the equipment condition so as to adjust the working parameters. Although the proposed system has been initially established at present, it only completes the specific functions in the laboratory environment. Future work will focus on applying this system into underground coal mines. Future research goals include integrating the real-time environment information to this system and better controlling the running of the three-machine.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare no conflict of interest.

Acknowledgments

The research presented here was supported by the National Key R&D Program of China (2017YFC0804310), National Key R&D Program of China (2018YFC0808203), and National Natural Science Foundation of China (No. 51805428). The authors are grateful for these supports.

References

J. H. Wang and Y. H. Fu, “Crucial technology research and demonstration of digital mines,” Journal of China Coal Society, vol. 41, no. 6, pp. 1323–1331, 2016.
View at: Google Scholar
J. Brodny, M. Tutak, and M. Michalak, “A data warehouse as an indispensable tool to determine the effectiveness of the use of the longwall shearer,” in Beyond Databases, Architectures and Structures. Towards Efficient Solutions for Data Analysis and Knowledge Representation. BDAS 2017, S. Kozielski, D. Mrozek, P. Kasprowski, B. Małysiak-Mrozek, and D. Kostrzewa, Eds., vol. 716 of Communications in Computer and Information Science, pp. 453–465, Beyond Databases, Architectures & Structures, Cham, 2017.
View at: Publisher Site | Google Scholar
J. Brodny, M. Tutak, and M. Michalak, “The use of the TGŚP module as a database to identify breaks in the work of mining machinery,” in Beyond Databases, Architectures and Structures. Towards Efficient Solutions for Data Analysis and Knowledge Representation. BDAS 2017, S. Kozielski, D. Mrozek, P. Kasprowski, B. Małysiak-Mrozek, and D. Kostrzewa, Eds., vol. 716 of Communications in Computer and Information Science, pp. 441–452, Springer, Cham, 2017.
View at: Publisher Site | Google Scholar
J. Tichon and R. Burgess-limerick, “A review of virtual reality as a medium for safety related training in mining,” Journal of Health & Safety Research & Practice, vol. 3, no. 1, pp. 33–40, 2011.
View at: Google Scholar
P. Perez, S. Pedram, and B. Dowcet, “Impact of virtual training on safety and productivity in the mining industry,” in MODSIM2013, 20th International Congress on Modelling and Simulation, Adelaide, Australia, 2013.
View at: Publisher Site | Google Scholar
S. Pedram, P. Perez, and B. Dowsett, “Assessing the impact of virtual reality-based training on health and safety issues in the mining industry,” in Proceedings of the International Symposium for Next Generation Infrastructure, Wollongong, Australia, 2013.
View at: Publisher Site | Google Scholar
P. Stothard and D. Laurence, “Application of a large-screen immersive visualisation system to demonstrate sustainable mining practices principles,” Mining Technology, vol. 123, no. 4, pp. 199–206, 2014.
View at: Publisher Site | Google Scholar
P. Stothard, “The feasibility of applying virtual reality simulation to the coal mining operations,” Australasian Institute of Mining and Metallurgy Publication Series, vol. 5, no. 1, pp. 175–183, 2003.
View at: Google Scholar
O. Akkoyun and N. Careddu, “Mine simulation for educational purposes: a case study,” Computer Applications in Engineering Education, vol. 23, no. 2, pp. 286–293, 2015.
View at: Publisher Site | Google Scholar
M. Kijonka and O. Kodym, “Coal industry technologies simulation with virtual reality utilization,” in Proceedings of the 13th International Carpathian Control Conference (ICCC), pp. 278–283, High Tatras, 2012.
View at: Publisher Site | Google Scholar
J. Torano, I. Diego, and M. Menéndez, “A finite element method (FEM) – Fuzzy logic (soft computing) – virtual reality model approach in a coalface longwall mining simulation,” Automation in Construction, vol. 17, no. 4, pp. 413–424, 2008.
View at: Publisher Site | Google Scholar
L. Wan, L. Gao, and Z. Liu, “The application of virtual reality technology in mechanized mining face,” G. Yang, Ed., vol. 181 of Advances in Intelligent Systems and Computing, pp. 1055–1061, Springer.
View at: Publisher Site | Google Scholar
X. Xu, X. Meng, and Y. He, “Research on virtual simulation of full mechanized mining face production based on 3D visualization and virtual simulation,” Journal of Safety Science & Technology, vol. 1, no. 1, pp. 26–32, 2014.
View at: Google Scholar
S. X. Zhang, “Augmented reality on longwall face for unmanned mining,” Applied Mechanics and Materials, vol. 41, no. 6, pp. 388–391, 2010.
View at: Publisher Site | Google Scholar
P. Foster and A. Burton, “Virtual reality in improving mining ergonomics,” Journal- South African Institute of Mining and Metallurgy, vol. 104, no. 2, pp. 129–133, 2004.
View at: Google Scholar
P. J. Foster and A. Burton, “Modelling potential sightline improvements to underground mining vehicles using virtual reality,” Mining Technology, vol. 115, pp. 85–90, 2013.
View at: Publisher Site | Google Scholar
F. Hai, X. Feng, and H. Zhi, “A study on the remote monitoring system of hydraulic support based on 3DVR,” in 2010 International Conference on Audio, Language and Image Processing, Shanghai, China, 2010.
View at: Publisher Site | Google Scholar
Z. Xiaoqiang, W. An, and L. Jianzhong, “Design and application of virtual reality system in fully mechanized mining face,” Procedia Engineering, vol. 26, no. 4, pp. 2165–2172, 2011.
View at: Publisher Site | Google Scholar
Z. Xiaoqiang and Z. Zhiqiang, “Application of dynamic selection algorithm to virtual reality monitoring of pump room,” Coal Mine Machinery, vol. 30, no. 11, pp. 172–175, 2009.
View at: Google Scholar
J. C. Xie, Z. Yang, X. Wang, Q. Zeng, J. Li, and B. Li, “A virtual reality collaborative planning simulator and its method for three machines in a fully mechanized coal mining face,” Arabian Journal for Science and Engineering, vol. 43, no. 9, pp. 4835–4854, 2018.
View at: Publisher Site | Google Scholar
N. B. Li, P. Yang, and H. J. Yang, “VR collaborative and interactive design over the web: a MEMS case study,” International Journal of Materials & Product Technology, vol. 31, no. 2-4, pp. 259–268, 2008.
View at: Publisher Site | Google Scholar
S. Birr, J. Mönch, D. Sommerfeld, and B. Preim, “A novel real-time Web3D surgical teaching tool based on WebGL,” in Bildverarbeitung für die Medizin 2012, T. Tolxdorff, T. Deserno, H. Handels, and H. P. Meinzer, Eds., Informatik aktuell, Springer, Berlin, Heidelberg, 2012.
View at: Publisher Site | Google Scholar
S. Fei, C. Zhao, and M. Dun, “A lightweight and cross-platform Web3D system for casting process based on virtual reality technology using WebGL,” The International Journal of Advanced Manufacturing Technology, vol. 80, no. 4, pp. 801–816, 2015.
View at: Publisher Site | Google Scholar
M. J. Bian, H. H. Gao, J. Gao, and J. P. Xu, “Research and application of Web3D exhibition based on WebGL and Html5,” in Proceedings of the 2015 International Conference on Electrical, Automation and Mechanical Engineering, Shanghai, China, 2015.
View at: Publisher Site | Google Scholar
J. Xia, B. Xiao, and D. Li, “Interactive WebGL-based 3D visualizations for EAST experiment,” Fusion Engineering and Design, vol. 112, no. 15, pp. 946–951, 2015.
View at: Publisher Site | Google Scholar
J. Zhao, “Designing virtual museum using Web3D technology,” Physics Procedia, vol. 33, no. 2, pp. 1596–1602, 2012.
View at: Publisher Site | Google Scholar
L. Chittaro and R. Ranon, “Web3D technologies in learning, education and training: motivations, issues, opportunities,” Computers & Education, vol. 49, no. 10, pp. 3–18, 2007.
View at: Publisher Site | Google Scholar
M. Yang, W. Zheng, and Q. Zhi, “Web3D technologies and motion estimation in education,” IEEE Computer Society, vol. 13, no. 2, pp. 391–405, 2014.
View at: Google Scholar
D. Luebke, “The present & future of Web3D,” in Proceedings of the 13th international symposium on 3D web technology - Web3D '08, Banff, Canada, 2008.
View at: Publisher Site | Google Scholar
C. Leung and A. Salga, “Enabling WebGL,” in Proceedings of the 19th international conference on World wide web - WWW '10, 2010.
View at: Publisher Site | Google Scholar
M. Bächer, B. Bickel, D. L. James, and H. Pfister, “Fabricating articulated characters from skinned meshes,” ACM Transactions on Graphics, vol. 31, no. 4, pp. 1–9, 2012.
View at: Publisher Site | Google Scholar
J. C. Xie, Z. J. Yang, X. W. Wang, J. L. Li, and R. Fang, “Research on key technology of virtual collaboration of three machines in fully mechanized coal mining face,” Chinese Journal of Engineering Design, vol. 25, no. 4, pp. 85–93, 2018.
View at: Google Scholar
L. Ying and D. Zorin, “A simple manifold-based construction of surfaces of arbitrary smoothness,” ACM Transactions on Graphics, vol. 23, no. 3, pp. 271–275, 2004.
View at: Publisher Site | Google Scholar
J. Feuvre, “Towards declarative 3D in Web architecture,” in Proceedings of the 21st International Conference on WWW.New York: ACM, pp. 161–168, Lyon, France, 2012.
View at: Google Scholar
I. Can, “Use WebGL 3D canvas graphics,” https://caniuse.com/#feat=webgl.
View at: Google Scholar

Copyright

Copyright © 2020 Zhengxiong Lu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

547

Downloads

1114

Citations