Abstract
Side plate offset is one of the grate system faults. If it is not dealt with in time, some accidents will occur and economic losses will be made. Aiming at the problems like time-consuming, labour-wasting, and low intelligent by the side plate offset detection method manually, an autoside plate offset detection method is proposed, based on You Only Look Once version 4 (YOLOv4). Two cameras were fixed to collect the image information of the grate trolley’s side plate. With reference to the grate trolley’s operation, the offset judgment rules were set. YOLOv4 object detection algorithm was used to detect the side plate and trolley’s chassis frame in video frame images. A baseline was set according to the position information of the trolley’s chassis frame output by detection, and then, the position intervals between side plates and the baseline could be determined by calculation. According to the judgment rules, the scheme in this paper could detect the offset fault of the trolley’s side plate timely, and an alarm would be made automatically when faults are detected. Our video images of the trolley’s side plate were collected and sorted in Baogang Group sintering plant for testing. In this experiment, no error judgment was made, and the average detection and judgment time was 0.024 s. In this paper, rather than manually, the real-time automatic detection was realized to detect the offset fault of the trolley’s side plate so as to provide a new solution for offset detection of the grate trolley’s side plate.
1. Introduction
In recent years, with the rapid development of China’s iron and steel industry, iron ore has been mined in large quantities, and the natural lump ore with high iron grade is decreasing day by day. At present, pellets have become the main raw material for blast furnace ironmaking. Grate-kiln pelletizing process is a widely used pelletizing production method in Chinese iron and steel enterprises; its main equipment consists of grate, rotary kiln, and annular cooler. The operation function schematic diagram of grate-kiln process equipment is shown in Figure 1.
The grate machine is mainly used for drying, preheating, and oxidizing pellets, and the output and quality of pellets depend on its working condition directly. The trolley used for conveying pellets is the core component of the grate, and the side plates are installed on both sides of the trolley to prevent pellet leakage [1]. Once the side plate offset occurs, the hot ore may fall into the lower loop of the grate and the cable is burnt, or it may cause accidents such as chain link breaking. Meanwhile, maintenance difficulties will be added with the offset fault when it appears in the heating area. The machine has to stop and wait for the heating area to cool before maintenance. When the maintenance is over, it takes several hours of heating to resume normal production. In the process, a lot of production time is wasted with serious economic losses [2].
1.1. Present Situation
Aiming at the grate system fault caused by the side plate offset, an offset detection mechanism of the grate’s side plate was designed by Chongqing Iron and Steel Co., Ltd., with the rotary cylinder with detection parts and proximity switch, which were sensitive and reliable to detect the offset fault of side plate [3]. An automatic alarm device of grate was designed by Xinxing Ductile Iron Pipes Co., Ltd., with a signal generating mechanism and an alarm mechanism. The alarm mechanism consisted of a proximity switch and a sound-light alarm component, both connected in series in a closed circuit to give an alarm in advance when the grate’s side plate was broken or other faults occurred [4]. An accident switch of the grate was designed by Ansteel Group Mining Co., Ltd., with mechanical transmission collision component and electrode switch. With it, scraping could be avoided when the side plate was warped or when small shaft ran out [5]. A state inspection device of the grate’s side plate was designed by Ansteel Co., Ltd., with a detection piece of grate plate, proximity switch, and sound-light alarm, to detect the fault and send sound-light alarm in advance and remind the on-site staff of resetting the side plate in time [6]. A deformation alarm device of the grate’s side plate was designed by Xuansteel Co., Ltd., with an arc plate, limiting displacement bar, sensor board, proximity switch, and alarm. Especially at the final term of grate service when a side plate was deformed, an abnormal side plate could be located in time [7].
However, due to complexity of the installation and generality of the detection device, the above detection schemes have not been applied widely in industry. At present, the offset detection of the side plate is still mainly by manual observation method which is time-consuming, labour-wasting and low intelligent.
1.2. Main Works and Structure
Nowadays, with the rapid development of deep learning techniques, object detection algorithm based on deep convolutional neural networks (DCNNs) has been constantly improved in detection accuracy and speed. DCNNs play an increasingly important role in the field of object detection, such as face detection [8], pedestrian detection [9], vehicle detection [10], and medical image detection [11]. With these inspirations, the object detection technology based on DCNNs is applied to the side plate offset detection, using machine vision instead of manual observation to detect the fault. In this paper, a side plate offset detection method based on DCNNs is proposed. The overall implementation method is as follows: firstly, two cameras are fixed to collect the image information of the trolley’s side plate. And then, You Only Look Once version 4 (YOLOv4) [12] object detection algorithm is applied to detect the side plate and trolley’s chassis frame in video frame images. A baseline is determined by the position of the trolley’s chassis frame output by detection. According to the position intervals between side plates and the baseline, whether an offset fault occurs can be judged as well as the level of fault. The main contributions of our work are summarized as follows: (i)With artificial intelligence technology applied to the traditional industrial inspection field, a side plate offset detection scheme is proposed, based on machine vision(ii)With reference to the grate trolley’s operation, the offset judgment rules of the trolley’s side plate are set(iii)With the proposed detection scheme, the real-time automatic detection of the fault is realized to replace the manual observation method
The rest of the paper is listed as follows: Section 2 gives a brief overview about the application of object detection algorithm based on DCNNs in industrial inspection field and introduces the neural network chosen for object detection in the offset fault detection process of the trolley’s side plate. Section 3 explains specific implementation details about offset fault detection of the trolley’s side plate. And Section 4 describes the datasets and the training process, and the experimental results of trolley’s side plate offset detection are presented. Finally, a conclusion is offered in Section 5.
2. Related Work
2.1. Application of the Industrial Inspection Based on DCNNs
In recent years, the neural network-based methods have been widely applied in intelligent transportation systems [13–15], intelligent video surveillance [16, 17], automatic monitoring [18, 19], and industrial inspection [20, 21] fields. By improving the Faster R-CNN network model, Sun et al. realized the detection of scratch, oil pollution, block, and grinning four kinds of wheel hub defects quickly and accurately [22]. By using the Faster R-CNN network model, Urbonas et al. realized the automated analysis of branch, scratch, stain, and core four kinds of wood panel surface defects, providing a new automatic solution for the lumber and wood processing industry [23]. By using the Faster R-CNN network model, Wang et al. realized the detection of scratch defects on turbine blades of automobile turbine engine, which proved the effectiveness of the application of DCNNs in the industrial automated surface inspection field [24]. Based on Mask R-CNN, Wu et al. proposed an approach which can classify, position, and segment the solder joint defect at the same time, and the method was expected to be used in the automatic inspection industry for analysis of defects with detail position and type information [25]. By combining a glance network with a multiple-channel Mask R-CNN, Shi et al. realized the accurate and fast detection of dead knots, live knots, and cracks three kinds of wood veneer defects, which provided a solution for wood veneer online production defect detection [26]. Based on an SSD network model, Yang et al. realized the real-time automatic detection of 0.8 cm darning needle defects (such as crooked shapes, length and endpoint size errors, and wringing), which provided new ideas for the automated detection of tiny part defects [27]. By improving the YOLOv3 network model and combining image filtering and smoothing techniques, Yu et al. realized the detection of gear defects (such as gear tooth surface tear, tooth fracture, and gear surface scratches) under a complex background during industrial gear production [28]. By improving the YOLOv3 network model, Wu et al. realized the detection of solder point, ground wire, and inner and outer ring three types of defects of a circular electrical connector, which basically met the requirements of industrial field for the accuracy and real-time detection. The method was expected to be used for reference in the identification of small target defects in industrial field [29].
2.2. You Only Look Once Version 4 (YOLOv4)
You Only Look Once (YOLO) is a well-known, real-time object detection algorithm. The algorithm can generate the class probabilities and location coordinates of an object in a stage directly. In 2020, Bochkovskiy et al. proposed the YOLOv4 [12] object detection algorithm by improving the classical real-time object detection algorithm YOLOv3 [30]. The YOLOv4 network model takes CSPDarknet53 [31] neural network as the backbone for a detector, since CSPDarknet53 contains a larger receptive field size ( receptive field) with a larger number of convolutional layers (29 convolutional layers ) and a larger number of parameters (27.6M). YOLOv4 adds the Spatial Pyramid Pooling (SPP [32]) block over the CSPDarknet53, since it significantly increases the receptive field, separates out the most significant context features, and causes almost no reduction of the network operation speed. Meanwhile, YOLOv4 uses Path Aggregation Network (PAN [33]) as the method of parameter aggregation from different backbone levels for different detector levels, instead of the Feature Pyramid Network (FPN [34]) used in YOLOv3. Finally, the architecture of YOLOv4 consists of CSPDarknet53 backbone, SPP additional module, PANet path-aggregation neck, and YOLOv3 (anchor based) head. The schematic diagram of YOLOv4’s network structure is shown in Figure 2. YOLOv4 combines a series of tuning tricks, and it also makes some additional improvements. The main improvements are as follows: (1) introduce a new method of data augmentation Mosaic, and Self-Adversarial Training (SAT). Mosaic mixes 4 training images. This allows detection of objects outside their normal context. In addition, batch normalization calculates activation statistics from 4 different images on each layer. This significantly reduces the need for a large minibatch size. SAT operates in 2 forward-backward stages. In the 1st stage, the neural network alters the original image instead of the network weights. In this way, the neural network executes an adversarial attack on itself, altering the original image to create the deception that there is no desired object on the image. In the 2nd stage, the neural network is trained to detect an object on this modified image in the normal way; (2) select optimal hyperparameters while applying genetic algorithms; (3) modify Spatial Attention Module (SAM [35]), Path Aggregation Network (PAN), and Cross-Iteration Batch Normalization (CBN [36]) to make the network suitable for efficient training and detection.
YOLOv4 improves YOLOv3’s AP and FPS by 10% and 12%, respectively, and YOLOv4 has more potential in practical application. In this paper, it was considered that both the detection accuracy and detection speed are important in the application scenario, and YOLOv4 achieves a better trade-off between detection accuracy and detection speed. Therefore, it is applied for object detection in the offset fault detection process of the trolley’s side plate.
3. Methodology
3.1. Technical System for Offset Detection of Trolley’s Side Plate
In the process of sintering production, the worksite of the grate trolley is shown in Figure 3. In the raw pellet drying section of the grate, the technical system for offset detection of the trolley’s side plate is set up. A support structure is put up beside the trolley’s rail. An offset detection camera is installed on the support structure above the trolley’s chassis frame, detecting whether the side plate offset occurs. A license plate recording camera is installed on the support structure to obtain the unique number information set for the trolley’s side plate. If possible, a light compensation device is used to obtain better video images. Two cameras are connected to the video image processing server through a switch, and the video images are transmitted to the server for algorithm processing. When the side plate offset fault is detected, the detection picture, the side plate number, and the corresponding alarm can be viewed in the display terminal, and the on-site staff can check and deal with it in time so that accidents can be avoided. The schematic diagram of the technical system for trolley’s side plate offset detection is shown in Figure 4.
3.2. Fault Classification Rules
In an actual production process, the offset fault of the trolley’s side plate can be classified into low-level offset fault and high-level offset fault. When a low-level offset fault occurs, the production accident may be caused by the subsequent offset aggravation of the side plate, and the fault should be dealt with regularly. When a high-level offset fault occurs, it is necessary to be dealt with immediately. Serious production accidents will occur otherwise.
The top view of the schematic diagram of the grate trolley’s side plate is shown in Figure 5. The possible position of the trolley’s side plate is between A and B boundary lines. Based on the definition of the offset fault in actual production process, a demarcation line is set on the inner side of the trolley’s chassis frame (left side in the figure) to judge whether offset fault occurs. The demarcation line divides into nonoffset area and offset fault area. The trolley’s chassis frame is defined as the baseline, which is used to classify the offset fault level. The baseline divides the offset fault area into a low-level offset fault part and high-level offset fault part. It should be noted that the trolley’s chassis frame is not always on the same line in actual scene. In this paper, to minimize the error, the line corresponding to the median value of each trolley’s chassis frame’s position in the same image is defined as the baseline.
When a side plate is on the left side of the demarcation line, as side plate 1 showed in Figure 5, no offset fault occurs to the side plate. When a side plate is on the right side of the demarcation line (or on the demarcation line) and on the left side of the baseline, as side plate 2 showed in Figure 5, a low-level offset fault occurs. When a side plate is on the right side of the baseline (or on the baseline), as side plate 3 showed in Figure 5, a high-level offset fault occurs.
3.3. Implementation Details
The flow chart of trolley’s side plate offset detection is shown in Figure 6. The detection process is mainly divided into five stages: model training, object detection, data processing, fault judgment, and fault warning.
In the model training stage, the collected side plate datasets are sent to YOLOv4 pretraining network for training. After training, a weight file can be obtained, which is used for identifying the side plate and trolley’s chassis frame in the following object detection stage.
In the object detection stage, under trolley’s normal operation, the offset detection camera collects the video data of the trolley’s side plate in real time, and the video frame image is intercepted at a fixed time interval. The YOLOv4 network model with the obtained weight file is applied to video frame images for object detection. The side plate and trolley’s chassis frame are detected, and the centre point coordinates of the side plate and trolley’s chassis frame are obtained and output.
In the data processing stage, according to the position data output in the object detection stage, for each image, (1) calculate the value, which is the median of trolley’s chassis frames’ centre point coordinate value in the -axis direction (the rectangular coordinate system is shown in Figure 5) and set the line as the baseline; (2) save every side plate’s centre point coordinate value in the -axis direction; and (3) calculate the difference between the value and the centre point coordinate value of each side plate in the -axis direction.
In the fault judgment stage, according to the difference between the value and the centre point coordinate value of each side plate in the -axis direction, whether an offset fault occurs can be judged as well as the level of fault. The specific judgment rules are as follows: (1) if the -coordinate value of the side plate’s centre point is more than or equal to the value, it is judged that a high-level offset fault occurs to the side plate; (2) if the difference between the value and the -coordinate value of the side plate’s centre point is less than or equal to a set threshold value, it is judged that a low-level offset fault occurs; (3) it is judged that no offset fault occurs in the other cases. The detection program can detect the offset fault of the trolley’s side plate real time according to the judgment rules.
In the fault warning stage, when the offset fault is detected, the alarm prompt, the side plate number, and the detection picture generated by redrawing the fault side plate’s bounding box are automatically sent to the storage device and display terminal, reminding the on-site engineer for confirmation. If it is judged that a high-level offset fault occurs to the side plate, the engineer should deal with it in time. If it is judged that a low-level offset fault occurs, the above fault warning data is stored, and the fault will be dealt with in the later process of regular equipment maintenance.
4. Experimental Results and Analysis
The experiment is implemented on Ubuntu 16.04 with CPU Intel (R) Core (TM) i9-9900K and a single GPU NVIDIA GeForce RTX 2080Ti, video memory is 11 GB, and memory is 64 GB. In this experiment, when the difference between the value and the -coordinate value of the side plate’s centre point is less than or equal to 60 pixels, it is judged that a low-level offset fault occurs to the side plate.
4.1. The Datasets
In the Baogang Group sintering plant, video data of the side plate on both sides were collected when the grate trolley was running. By intercepting the video frame image, a total of 145 images (48 images using light compensation device included) were sorted out as the experimental datasets. In the datasets, there were 91 images of no side plate offset fault, 53 of low-level side plate offset fault, and 1 of high-level side plate offset fault. The image resolution was all pixels.
4.2. Training
LabelImg tool was used to label the experimental datasets. The side plate, trolley’s chassis frame, and trolley’s wheel were labelled in each image. The labelled datasets were divided into a training set and test set, accounting for 90% and 10%, respectively. During the training process, the image set pixels was input, with a learning rate of 0.001, weight decay of 0.0005, momentum of 0.949, and max batches of 6000. The loss curve during the training is shown in Figure 7. And the AP and mAP for each point are shown in Figure 8. After training, there were side plate’s AP of 99.02%, trolley’s chassis frame’s AP of 97.96%, trolley’s wheel’s AP of 100%, [email protected] of 98.99%, and average IoU of 82.05%. So the side plate, trolley’s chassis frame, and trolley’s wheel were accurately identified in the dataset images. The object detection was shown in Figure 9. In the condition of dark light, such as at night or the light is not strong. Although it would cause uneven illumination phenomenon shown as Figure 9(b) by using light compensation device, the detection was not affected.
(a)
(b)
4.3. Experimental Results in Trolley’s Side Plate Offset Detection
In this experiment, based on the position intervals between side plates and the baseline, different results in side plate offset detection are shown in Figure 10. The detection program redrew the fault side plate’s bounding box in red to differ from the default bounding box. In Figure 10(a), all side plates’ bounding box had not changed in colour, and it was judged that no offset fault occurred to the side plate. In Figure 10(b), there was a side plate inside the trolley’s chassis frame (left side in the figure) with a red bounding box, and it was judged that a low-level offset fault occurred. In Figure 10(c), there were two side plates with red bounding boxes, one inside the trolley’s chassis frame (left side in the figure) and the other on the trolley’s chassis frame, and it was judged that both low-level and high-level offset faults occurred. And then, the engineer was informed to make an on-site inspection, and the judgment results given by the engineer were consistent with those by the detection program.
(a)
(b)
(c)
In the offset detection of the side plate, 145 images collected from the Baogang Group sintering plant were analysed. The results are shown in Table 1. There were 53 images with a low-level side plate offset fault and 1 image with both low-level and high-level side plate offset faults. The fault warning was sent to the on-site engineer for confirmation. After the on-site inspection, the results inspected by the engineer were consistent with those by the detection program. All the results by the detection program proved correct in this experiment.
4.4. Analysis of Trolley’s Side Plate Offset Detection Results
In this paper, the speed of grate trolley was 3-5 cm/s, and the video frame image was intercepted every 5 seconds. The average detection and judgment time was 0.024 s, which could meet the real-time requirements. Meanwhile, there were high degree of automation and low mistake rate in the whole detection process, with less manual involvement. The experimental results show that with the scheme in this paper, the trolley’s side plate offset fault can be detected effectively rather than in the manual observation method. So the working intensity of on-site workers can be reduced, and the efficiency of fault detection can be increased.
5. Conclusions
In the sintering production process, side plate offset is one of the grate system faults. If the fault cannot be dealt with in time, serious accidents will occur with economic losses. At present, the offset detection of the side plate is done mainly in the original manual observation method, which is time-consuming, labour-wasting and low intelligent. In this paper, a side plate offset detection method based on DCNNs is proposed. Two cameras are used to collect the images of the trolley’s side plate, and the YOLOv4 network model is used for object detection in the offset fault detection process of the trolley’s side plate. With reference to the grate trolley’s operation, the offset judgment rules are set. By calculating the position intervals between side plates and the baseline, whether offset fault occurs can be judged as well as the level of fault.
With the research scheme in this paper, a new solution for offset detection of the grate trolley’s side plate is suggested to replace the manual observation method to detect effectively the trolley’s side plate offset fault. So the real-time automatic detection of the fault can be realized. The scheme can be used to effectively avoid the accidents with grate in iron and steel enterprise sintering plants, reduce the consumption of manpower, improve the production efficiency, and enhance the management of enterprises. It plays an important role in promoting to develop and realize intelligent iron and steel plants and has potential value in application.
Data Availability
The datasets, codes, weight file, and test video used to support the findings of this study are available from the corresponding author upon request. And the YOLOv4 source code is available online at https://github.com/AlexeyAB/darknet.
Conflicts of Interest
The authors declare that there is no conflict of interest regarding the publication of this paper.
Acknowledgments
The authors gratefully acknowledge the refreshment done by Professor Huanbang Wang. This research was funded by the National Natural Science Foundation of China (grant number 61962046); the Natural Science Foundation of Inner Mongolia (grant number 2020MS06008); and the Inner Mongolia Science and Technology Plan Project.