Abstract
Aiming at the problem of multi-modal fault detection of different equipment in ultrahigh voltage (UHV) substations, a method for based on robot inspection and deep learning is proposed. First, the inspection robot is used to collect the image data of different devices in the station and the source data is preprocessed by standard image augmentation and image aliasing augmentation. Then, the HSV color space model based on saliency area detection is used to extract equipment defect areas, which improves the accuracy of defect image classification. Finally, the traditional YOLOv3 network is improved by combining the residual network and the K-means clustering algorithm, and the detailed flow of the corresponding detection method is proposed. The proposed detection method and the other three methods were compared and analyzed under the same conditions through simulation experiments. The results show that the detection accuracy and recall rate of the method proposed in this study are the largest, which are 95.9% and 91.3%, respectively. The average detection accuracy under multiple intersection ratio thresholds is also the highest, and the performance is better than the other three comparison algorithms.
1. Introduction
At present, with the increasing number of UHV substations, their importance in the power system is gradually getting promoted [1]. The operation state of the UHV substation has a great impact on the power system. The effective safety detection of the UHV substation is an effective means to ensure its safe and stable operation, which is of great significance [2–4]. The safety detection of the UHV substation is mainly to check whether the equipment fails. At present, the main detection means include fixed monitoring, manual inspection, and robot inspection [5, 6]. Using robots to perform inspection tasks instead of manual workers has broad application scenarios and huge development space, and it is a research hotspot for intelligent development of UHV substations at home and abroad [7, 8].
2. Related Work
In view of the mentioned problems related to fault detection of the UHV substations, the most prospective and research-based detection method is robot inspection that can automatically obtain infrared images of power equipment by installing inspection robots in UHV substations and judge the status of power equipment on this basis [9]. Finally, the online analysis and decision on the operation status of the UHV substation are realized. Aiming at the fault detection problem of GIS (GAS Insulated SWITCHGEAR) combined appliances, [10] designs a GIS equipment maintenance robot with a flexible structure and convenient controls. The whole GIS equipment can be checked without blind spots. However, this method is only used to detect the internal equipment of GIS integrated appliances and has some limitations. With the goal of practical inspection robot, [11] developed a set of intelligent inspection technologies that can realize automatic charging of robots and task planning. It improves the inspection efficiency of UHV substations based on inspection robots to a certain extent. However, this method has some disadvantages, such as low accuracy and a long time. Aiming at the problem that substation indoor inspection robot could not accurately identify the inspection range under some complicated circumstances, [12] proposes a flexible cable-driven inspection robot system scheme based on the design of hardware equipment and system software of the robot. Ref. [13] designs a trackless robot with a robotic arm that can navigate autonomously and inspect equipment for the narrow and complex indoor environment of the UHV substations. Autonomous movement and detection of the robot based on simultaneous positioning and mapping are realized. The above two methods only study the indoor inspection technology based on the inspection robot but do not analyze the outdoor situation, so their application scope is limited. Ref. [14] studies the key technologies of the UHV substation inspection robots in image acquisition, image recognition, intelligent inspection, and comprehensive diagnosis. By integrating the multisource data of the whole station and combining the corresponding sensors and acquisition equipment, it puts forward a kind of joint inspection system based on artificial intelligence analysis technology combined with automatic inspection. However, this method does not analyze the control complexity and motion precision of the robot itself. The goal of [15] is to improve the detection accuracy of the equipment and reduce the detection time in UHV substations. Aiming at the infrared image library collected by conventional UHV substation inspection robot equipped with infrared thermal imager, an infrared image detection method of UHV substation equipment based on improved Gaussian convolution kernel is proposed by combining deep learning. However, the image acquisition effect of the inspection robot in this method is poor in resolution, and the information is not intuitive and requires secondary processing. By designing the hardware structure and software system and integrating the existing technologies, [16] proposes a fully autonomous robotic system LongSword. Corresponding technical solutions are proposed for the process of equipment inspection using optical zoom cameras, thermal imaging cameras, and partial discharge detectors. However, the method does not consider the acquisition of sound information, and the detection accuracy needs to be improved.
Aiming at the core problems of small application range and low accuracy of current fault detection methods in UHV substations, this study proposes a multimodal fault detection method for UHV substations based on robot inspection and deep learning based on the above analysis. The basic idea is as follows: (1) The equipment image is obtained by inspecting the robot, and the image is augmented by standard image and image aliasing. (2) By introducing a HSV color space model to extract equipment defect areas, the accuracy of image classification can be improved. (3) The feature extraction capability and detection accuracy are improved by improving the traditional YOLOv3 network. Compared with traditional detection methods, the innovations of the proposed method are listed:(1)Both standard image augmentation and image aliasing augmentation increase the amount of data and improve the generalization ability of the model.(2)A defect region segmentation algorithm based on HSV is introduced to separate the defect region from the background, which improves the accuracy of the subsequent image classification algorithm.(3)The traditional YOLOv3 network is improved by using the residual network and K-means clustering algorithm, which improves the detection accuracy.
3. Fault Multimodal Detection Method
3.1. The Architecture of Multi-Modal Detection of Faults in UHV Substations
Due to the complexity and diversity of the UHV substation inspection environment, the design of the robot inspection system must ensure real-time performance and robustness, as well as consider the needs of system optimization and equipment replacement. It is necessary to design extensible functional modules for easy maintenance and upgrade. The overall scheme of the UHV substation robot inspection system is shown in Figure 1. The distributed architecture and modular design strategy are adopted to design different functional modules to realize the final fault detection.
As shown in Figure 1, the work content of the UHV substation robot inspection system mainly includes three aspects:(1)The inspection robot is used to collect image data for the operation status of various equipment in the UHV substation, and the collected image data is preprocessed. Then the processed equipment image data is sent to the remote monitoring system.(2)Based on the image data sent by the inspection robot, the remote monitoring system classifies the data and carries out network training and fault detection in combination with its own deep learning algorithm.(3)It outputs fault detection results.
3.2. Data Acquisition and Preprocessing
In the process of inspection in the UHV substation, the inspection robot mainly detects multisource data, such as meter reading, split and close state, temperature distribution, and foreign body recognition of different equipments in the station. Therefore, it can be seen that the inspection content includes the external conditions of the equipment and the temperature distribution of the equipment, and the comprehensive inspection in the UHV substation cannot be realized only by relying on single-source visual inspection. In addition, the structure of different devices is different, so it is difficult to achieve universal detection of infrared and visible images for all devices. The specific inspection contents and approaches of the inspection robot are as follows:(1)Meter reading detection: It is mainly for transformer and other oil-filled equipment oil level meter detection. Visible light is used for detection.(2)Status detection: It mainly detects the status of circuit breakers, disconnecting switches and signal lights. Visible light is used for detection.(3)Temperature distribution detection: The temperature changes in the body and the joint of the primary equipment in the UHV substation are mainly detected. Infrared light is used for detection.(4)Foreign body recognition and detection: It mainly detects the appearance of different equipment in the UHV substation and whether there are foreign bodies. Visible light is used for detection.
In the automatic detection of power equipment, it is necessary to identify the classification information of power equipment while locating the target area in the image so as to facilitate the later data processing and abnormal output. Therefore, it is necessary to construct a multisource heterogeneous data set for visible and infrared images of all the equipments in the UHV substation and clearly mark the equipment area and specific categories of power equipment before conducting a safety inspection for the equipment in the whole station.
The LabelImage tool is adopted here which can support multicategory simultaneous labeling and can select VOC, YOLO, and other data sets to save labels. Based on visible light and infrared detection data, equipment area and specific categories of power equipment in the obtained multisource heterogeneous images of the UHV substation are marked. The VOC2007 standard dataset is used for annotation. After annotation, corresponding label files containing image file name, size, position of annotation box, and annotation target category are generated in the Annotations folder. And labels, images, and related settings files of the dataset are generated in the VOC2007 folder to complete the establishment of the dataset. The VOC2007 storage format is shown in Figure 2.
In the process of actual data acquisition, it is difficult to obtain enough image data sets of different equipment in the UHV substation due to environmental factors and actual conditions. In this case, data augmentation is generally considered to solve the problem of insufficient sample data. The principle of data augmentation is to generate new data based on existing sample data through transformation, so as to increase the number of data and improve the ability of model generalization. Standard image augmentation and image aliasing augmentation are used for data augmentation.Standard image augmentation is realized through geometric transformation or color space transformation such as translation and rotation, flip and zoom, color channel transformation, and artificial addition of voice. The specific process is shown in .where in (1), represents the original image coordinates and represents the image coordinates after transformation. represents the geometric transformation relation matrix.
Image translation transformation is to move the original image pixels in different directions, and the movement process can be represented by matrix . The specific process is shown in .
In (2), is the offset in the horizontal direction and is the offset in the vertical direction.
Image rotation needs to consider the content and rotation angle of the original image to ensure the effectiveness of the transformed image. Generally, the central point of the image is the rotation center for rotation. The transformation matrix is . The specific process is shown in (3).
In (3), is the angle of rotation.
Image flipping transformation includes horizontal flipping and vertical flipping. Their transformation matrices are and, respectively. The specific process is shown in (4) and (5).
Image scaling transformation needs to consider the scaling multiple to meet the detection requirements and ensure its effectiveness while scaling the image at any scale. The scaling transformation matrix is as shown in (6).
In (6), and represent the scaling coefficients on the corresponding horizontal and vertical axes, respectively.
The image before and after transformation is shown in Figure 3.
(a)
(b)
(c)
(d)
(e)
(f)
The label corresponding to the image can be calculated, and the usual format is , where and are the coordinates of the upper left corner of the original image label box, and are the coordinates of the lower right corner of the original image label box, and indicates the category of marked objects in the label box. The new label box is derived from the geometric transformation expression (1).
Image aliasing data augmentation is achieved by stitching and aliasing different images. Mosaic data enhancement method is adopted here for stitching aliasing. It can realize the stitching of four images and each image has a target box. The image processed by Mosaic contains the images and labels of the four images, which can greatly enrich the background of the detected object and improve the performance of the model. The implementation process of the Mosaic data enhancement method is as follows:(1)Four photos are randomly read each time from the “JPEGImage” image file of the VOC2007 dataset as the original images of Mosaic.(2)For the above four original images, geometric transformation, color change, and other operations are performed. After the operation is completed, the offset coordinates will be randomly generated and the original pictures will be placed in the order of top left, bottom left, bottom right, top right, and bottom right.(3)The combination of images and boxes. The fixed area of four pictures is intercepted and spliced into a new picture by means of a matrix. The new picture contains a series of contents such as label box.(4)Finally, the label box is processed. If the clipping sample contains part of the label frame, it will be discarded. The label frame that is still intact after clipping will be retained.
3.3. Multimode Fault Detection of UHV Substation Based on Improved Deep Learning
The data collected and preprocessed by robot inspection is transmitted to the remote monitoring system, and the improved deep learning module is used for feature training and fault detection.
3.3.1. Equipment Defect Extraction
Before the defect classification of the infrared image of power equipment, it is necessary to preprocess the image to judge whether the power equipment has defects according to the characteristics of the infrared image itself. If the power equipment is judged to have defects, the defect area of the power equipment is marked and the defect area is separated from other background areas. The part of the image that only contains the defect area is extracted, which can improve the accuracy of subsequent defect image classification.
For human vision, saliency region detection helps to extract important regions in a specific visual scene or an image. The color space models commonly used for significance region detection are HSV and RGB color space. RGB color space is also known as the three primary color mode, which is composed of red, green, and blue. All colors in RGB color space are generated through different degrees of fusion in the color space. In HSV color space, H represents chroma, S represents saturation, and V represents brightness. Infrared images mainly describe temperature, and the part with higher temperature is usually brightly reflected in the image. Compared with RGB color space, HSV color space can better perceive the connection between colors, and different H values can represent different colors.
For the defective areas of power equipment in the infrared image, we are more concerned with the areas that had yellow and white colors. To improve the accuracy of the subsequent image classification algorithm, the defect region segmentation algorithm based on HSV is adopted to separate the defect region from the background in the infrared image of faulty power equipment. Since it is impossible to determine the defect type by simply analyzing the fault area, it is necessary to segment it based on mathematical morphology according to the location of the defect area. It can reduce the interference caused by the background area in the infrared image to the subsequent classification of the defect types.
The steps of the defect region extraction algorithm based on HSV are as follows:(1)The center point of image clustering is initialized so that each cluster center point is evenly distributed in the image with the distance , and the pixel point with the smallest gradient is selected as the center point of the cluster in each neighborhood.(2)The processed image was normalized and the value of H was calculated.(3)The threshold value was set, and the rough equipment defect area in the infrared image was separated by the threshold value. OTSU threshold segmentation algorithm is used to segment the original image to obtain the binary image , and each isolated defect point is connected according to the mathematical morphology closed operation, and finally, the defect area is determined.(4)Each connected region in is searched for If (7) is true in the defective region, this region is the defective power equipment region to be extracted.
3.3.2. Equipment Defect Training and Fault Detection
(1) Improved Deep Learning Algorithm. In this study, the target detection algorithm YOLOv3 with fast operation speed is mainly used as the main frame of detection. The main network of YOLOv3 is improved by combining residual network and deformable convolution and the convolutional neural network is redesigned so that the multiscale detection of YOLOv3 has a stronger ability of small target detection.
In the process of deep learning, problems such as gradient dispersion and gradient explosion make the model training difficult to converge. The residual network is shown in Figure 4, and the function represents the output of the residual block. The problem of gradient disappearance can be solved by transferring the main input and using residual to represent the part that needs to be optimized. The residual block retains the main information well while amplifying the sensitivity to small changes.
Based on the algorithm YOLOv2, YOLOv3 uses multiscale fusion to achieve target positioning and detection by adding an anchor mechanism to the model. The convolutional network related to YOLOv3 is the Darknet535 network. Darknet53 network does not contain the full convolutional network of the layer. YOLOv3 extracts different feature graphs of convolution for feature fusion, and respectively predicts three outputs. Finally, the detection results are obtained by the feature fusion of the three scales. Due to the different sizes of feature images, the unfused feature images cannot be directly calculated, and an interpolation algorithm is needed to fill pixels.
To improve the poor detection effect of the traditional convolution algorithm, the offset is added to the original convolution for improvement so as to extract features more effectively. The mathematical expression of the deformable convolution is shown in (8).
In (8), represents the specified point in the image and represents each element of the convolution kernel, and represents the translation distance of the convolution kernel.
Since the substation equipment environment may present very small targets, there will be missed detection in the detection process. Therefore, the YOLOv3 algorithm can be optimized and improved by increasing the number of feature fusion 4 times to further improve the model’s ability to detect small targets. The basic structure of the improved YOLOv3 is shown in Figure 5.
The K-means clustering algorithm aims to complete the division which has the advantages of fast classification speed and high accuracy. Therefore, it is considered to generate an anchor frame based on K-means clustering algorithm, and its objective function can be expressed as a data matrix. According to the objective function of K-means clustering, (9) is established.
In (9), represents the vector and represents the matrix, and represents the matrix of class center point , and represents the matrix of binary indicator variable. It is shown in (10).
In (10), represents that the data set is divided into classes.
To enable the improved YOLOv3 algorithm to converge quickly and improve the accuracy of detection during training, the size of the most appropriate anchor point prediction box is found by clustering and calculating the real coordinate box in the test. Then the classification is completed by calculating the distance between the samples and iterating continuously. K-means calculation is carried out for the labeled Ground True Box. The improved YOLOv3 has four detection scales, and 12 prediction boxes are need to be generated in total, so the clustering number is 12. In the target detection task, the intersection ratio between the anchor point prediction frame and the real sample frame is used to define the calculation method of clustering distance. It is shown in (11).
where in (11), is the intersection ratio of anchor point prediction box and the real sample box.
(2) Network Training and Fault Detection. The training samples come from the infrared images taken in the UHV substation. According to the source images, the target region is firstly extracted and the minimum rectangle of the target in the region is obtained, and then the boundary is appropriately expanded. Then the size of the image is adjusted to 32 × 32 to obtain the network input image. To enlarge the training samples and improve the recognition ability and generalization ability of the network, the sample images obtained by the above methods are augmented by standard image augmentation and image aliasing augmentation.
Based on the overall idea of image classification, image processing, and fault detection, a new infrared detection method is proposed combined with inspection robot and an improved deep learning algorithm. It is the multimode detection method of UHV substation fault based on robot inspection and deep learning. The specific process of this method is shown in Figure 6.
As shown in Figure 6, the specific process of multimode detection method for UHV substation faults based on robot inspection and deep learning includes the following steps:(1)The inspection robot is used to collect image data of various equipment in the UHV substation.(2)The target region is extracted from the source data and the minimum rectangle is obtained.(3)Standard image augmentation and image aliasing augmentation are performed on the image data.(4)Extract the defect area of the image and use the K-means clustering algorithm to generate the anchor box, and classify the image on this basis.(5)The improved YOLOv3 algorithm is used for target feature extraction and fault detection.(6)It outputs fault detection results and faulty equipment information.
4. Experiment and Analysis
4.1. Experimental Environment
The UHV substation equipment defect data set used in the experiment is mainly from a UHV substation of a provincial power company, and some images contain multiple detection objects. The data set involves a total of 14,000 images of insulators, lightning arresters, circuit breakers, isolation switches, current transformers, voltage transformers, and transformer high voltage bushing, with 2,000 images of each object category. The number of object types, labels, images, and instances detected in the dataset is shown in Table 1.
In the whole data set, 20% (2,800 pieces) are randomly selected and divided into test sets. Then 60% (8,400 pieces) are randomly selected and divided into training sets. The remaining 2,800 pieces are verification sets.
4.2. Comparison of Detection Accuracy
According to the detection results, the detection accuracy, recall rate, and average accuracy (AP) at multiple intersection ratio (IoU) thresholds are used as evaluation indexes to analyze the performance.
AP is generally calculated by interpolation, and the calculation process is shown in (12).
In (12), and respectively represent the accuracy rate and recall rate of a certain point on the original P-R curve, and represents the interpolation point.
The multimodal detection method of UHV substation fault based on robot inspection and deep learning proposed in this study is compared with the algorithms in [13, 14, 16].
Firstly, the recall rate and detection accuracy of detection results of different algorithms are calculated, and the results are shown in Table 2.
It can be seen that the recall rate and accuracy of the proposed method are the highest according to the calculation results of the proposed method and other comparison methods in Table 2. In terms of recall rate, the improvement of the proposed method is relatively small compared with other methods. The maximum is 3.7% and the minimum is 2.2%. The recall rate of the four algorithms is relatively close. However, the accuracy of the proposed method reaches 95.9%, which is greatly improved compared with the other three methods. The maximum is 12.6% and the minimum is10.3%. The significant improvement in detection accuracy also makes the target detection score of the proposed method reach 5,825 points, which is greatly improved compared with the other methods. The maximum increase is 2,249 points and the minimum increase is 2,130 points. In conclusion, the proposed detection method has a better target detection performance compared with other methods.
The average accuracy of detection by different algorithms at multiple intersection ratio thresholds is compared, and the results are shown in Figure 7.
In Figure 7, and represent AP when the value of the official evaluation index of Microsoft Common Objects in Context (MS COCO) data set is 50 and 75, respectively. , , and represent the Aps of small size target (area <322), medium-size target (area <962), and large-size target (area >962), respectively.
It can be seen from Figure 7 that the proposed method has effectively enhanced the target detection effect of UHV substation equipment defects compared with other comparison algorithms, especially in multiscale detection. The APs of small-sized, medium-sized, and large-sized targets are all significantly improved. The number of feature fusions of the improved YOLOv3 network structure increases, and the low-dimensional feature map keeps the same scale size while changing the number of feature map channels. So the high-dimensional sampling feature map and low-dimensional feature map can be better superimposed and merged to improve the detection effect.
5. Conclusion
Aiming at the problem of automatic fault detection of equipment in UHV substation, a multimode fault detection method of UHV substation based on robot inspection and deep learning is proposed. The comparison results show that source augmentation can improve the generalization ability of the detection method to a certain extent. Using the HSV color space model to extract defect areas in images can effectively improve the accuracy of subsequent image classification. By introducing residual network and K-means clustering algorithm into traditional YOLOv3, the accuracy of the detection method improves effectively. Future work will conduct further research on how to judge the cause of the fault based on the image analysis technology on the basis of using the infrared image of the equipment to detect the fault of the power equipment.
Data Availability
The authors declare that all data sources are original.
Conflicts of Interest
The authors declare no conflicts of interest.
Acknowledgments
The authors are thankful to the science and technology project funding from State Grid Corporation of China (Project number: kj2021-059).