Abstract
With the rapid development of networks and the emergence of various devices, images have become the main form of information transmission in real life. Image restoration, as an important branch of image processing, can be applied to real-life situations such as pixel loss in image transmission or network prone to packet loss. However, existing image restoration algorithms have disadvantages such as fuzzy restoration effect and slow speed; to solve such problems, this paper adopts a dual discriminator model based on generative adversarial networks, which effectively improves the restoration accuracy by adding local discriminators to track the information of local missing regions of images. However, the model is not optimistic in generating reasonable semantic information, and for this reason, a partial differential equation-based image restoration model is proposed. A classifier and a feature extraction network are added to the dual discriminator model to provide category, style, and content loss constraints to the generative network, respectively. To address the training instability problem of discriminator design, spectral normalization is introduced to the discriminator design. Extensive experiments are conducted on a data dataset of partial differential equations, and the results show that the partial differential equation-based image restoration model provides significant improvements in image restoration over previous methods and that image restoration techniques are exceptionally important in the application of environmental art design.
1. Introduction
Human beings have entered the age of information technology, and all kinds of data information such as text, images, video, and audio are disseminated through the Internet. Relatively speaking, images are more acceptable and understandable to human beings than text and audio, and based on their intuitive and efficient characteristics, most of them are used as an important medium and tool for information transmission. High-quality images bring people richer information and content and even give people a sense of beauty. When processing images, images that are easy to process are free of masking and missing, but in the process of image acquisition and transmission, due to the poor precision of the transmission equipment used to acquire images or unstable networks, some of the images are missing or pixels are lost, which not only affects the visual effect of the images but also destroys the integrity of the information conveyed by the images and reduces the quality of image processing. Therefore, to improve the image quality and ensure the effectiveness of information transmission, the repair work of mutilated images is very necessary, especially in environmental art design [1].
The purpose of image restoration is to remove the occlusion in the image and reconstruct the missing information in the image based on the background information; if the image restoration technique is used efficiently in environmental art design, then the efficiency of the design will be more efficient as well as the presented work will be more perfect. The process of filling the specified area of the visual input with real data and the process of restoration of the specified area of the real data filling the visual input are highly subjective. Image restoration is most widely used in digital image restoration, image encoding, and transmission. Image restoration techniques are widely used in the field of film and television, to restore the deteriorated film and also to remove unwanted dates from pictures and to eliminate mosaics, among other things. As early as the Renaissance [2], image restoration techniques emerged as an important branch of image processing, where professional restorers restored damaged oil painting articles by hand painting, but there were many shortcomings in using manual restoration, experts specializing in the restoration of artifacts were very rare, and manual restoration was time-consuming and risky.
The traditional image restoration is used to fill the missing parts with pixels in other parts of the image. The rough method is used to directly stretch the edge pixels, and the fine method needs to calculate the similarity of small blocks and gradually fill in the diffusion. The methods can be roughly divided into two categories: image restoration based on partial differential equations and image restoration based on texture synthesis.
The way computers perform automatic image restoration at the beginning of the 21st century means that image restoration has entered a new era. In recent years, convolutional neural networks are used in computer vision fields such as image segmentation, image restoration, and target detection. Deep learning-based image restoration can repair realistic and semantically coherent images, which cannot be achieved by traditional restoration methods. In dealing with mutilated images, with the development of artificial intelligence, more and more professional scholars focus on and study deep learning image restoration techniques, and image restoration has important academic research value and practical significance.
2. Related Work
With the development of science and technology, target detection and image restoration have grown well with the help of computer-related technology. In this paper, we mainly discussed the overview of the field of target detection and image restoration. Among them, target detection can be divided into two-stage and single-stage target detection, and image restoration can be divided into the traditional approach and deep learning approach. The R-CNN (Region-Based Convolutional Neural Network) algorithm was proposed in the literature, which can learn autonomously and extract the image features of the sample with high accuracy, but the candidate frame. When performing feature extraction, it consumes a lot of learning time and storage space, and the speed of detection is slow; the Fast R-CNN algorithm proposed in 2014 uses a selective search algorithm to scale the candidate regions in the mapped feature map [3], the normalization of features is performed using Region Of Interest (ROI), and Softmax is used for classification, but the method has more redundant information, which limits the speed up to some extent.
The Faster R-CNN algorithm was proposed in the literature, which uses Region Proposal Network (RPN) for direct prediction; the accuracy of this method is greatly improved, but the detection speed is still slow. In addition, although representative algorithms based on candidate regions such as R-FCN-FPN improve the detection accuracy, the detection process is mostly a step from candidate regions to convolutional neural networks for feature detection, and the detection speed is still greatly limited. The Yolo (You Only Look Once) algorithm was proposed in the literature, which segments the input image and predicts the targets whose centroids fall within the grid, and has a simplified process, improved speed, and good real-time performance; the SSD (Single Shot MultiBox Detector) algorithm proposed in the literature introduces the anchor mechanism and uses a multilayer feature map to predict the target class and position offset, but the SSD algorithm has difficulty in improving the detection of small targets; the Yolov4 algorithm proposed in the literature, which greatly surpasses the two-stage target detection algorithm in terms of speed, achieves good results. With the advent of digital cameras and other smart devices, a large number of tasks that should be done by humans are transferred to machines for automation development, and with the advancement of deep learning, image restoration based on deep learning is gradually growing. The combination of sparse coding and deep neural networks with self-encoders was proposed in the literature, which is more effective compared to traditional image restoration algorithms; then, two different multilayer perceptron-based image restoration algorithms were proposed, but both of them are generally effective in restoring larger blocks of occluded areas; contextual content-based image restoration algorithms were proposed in the literature, where high-dimensional features of the image are learned using a self-coding network and detail content repair uses adversarial network, but the method is difficult to train; DCGAN deep network-based image semantic repair algorithm generating model is proposed in the literature to repair the mutilated regions by iteratively training and optimizing the network model; the disadvantages of this algorithm include long training time, easy gradient explosion, and cannot be applied in real-time [4].
The Yolov5 algorithm, proposed in the literature, has also achieved better results, being very fast but not officially certified. Image restoration focuses on the structure and texture of an image, referring to the geometric structure and the known sample texture in the image, respectively. Image restoration can be divided into traditional image restoration methods and deep learning image restoration methods. Among the traditional image restoration methods, the BSCB (Bertalmio Sapiro Caselles Bellester) algorithm is the first image restoration algorithm proposed in the literature based on partial differential equation (PDE), but the algorithm has many drawbacks, such as complex structure and very slow [5]; the literature proposes Total Variation (TV) model, which uses Lagrange equation to restore the image; the image restoration method based on sparse decomposition is proposed in the literature, which uses CDD (Curvature-Driven Diffusions) model to restore the structure of the image and Criminisi method to repair the texture part of the image. However, traditional image restoration methods have many problems, such as slow speed and poor results, which cannot meet people’s needs. From the techniques used in the image restoration applied so far, among them, the image restoration technique based on the micropartial equation is the most useful and the restoration is the best.
3. Partial Differential Equations
A partial differential equation is the equation of the unknown function and its partial derivatives, describing the relationship between the independent variable, the unknown function, and its derivatives. The so-called solving partial differential equation is used to find a function so that it satisfies the equation and the corresponding initial marginal value conditions. Partial differential equation is one of the most important mathematical tools to describe the laws of the objective image repair world. The partial microequation mainly deepens the image restoration technology and pays more attention to detail restoration during image restoration, and the restoration efficiency is greatly improved.
3.1. Partial Differential Equations Based on Image Restoration
Many partial differential equations occupy an important place in their respective fields, such as the Allen-Cahn equation for phase transition problems and the Monge-Ampère equation for geometry. Because of their wide practical value, partial differential equations have long attracted a large number [5] of researchers to work on them. -order partial differential equations are defined formally as follows:
The first formula in the article explains the main core content of image restoration technology and explains what image restoration is mainly repaired, what method is used to repair, and what is the accuracy of repair.
Exact solutions of partial differential equations are difficult to obtain, so obtaining approximate solutions of the equation is generally considered. Numerical methods are the classical approximate solution methods for partial differential equations. The classical numerical methods mainly include finite difference methods, finite element methods, and finite volume methods. These methods obtain the approximate solution of the equation by performing a complex mesh discretization over a given definition domain and then approximating the result of the discretization using simple functions. Although these methods have a good theoretical basis and experimental results, the merit of their solutions depends heavily on the grid dissection, and a good grid dissection can entail huge computational and storage costs. Besides, the process of the same numerical method varies widely between different equations. For example, finite difference methods have to use different difference formats for different partial differential equations. These disadvantages limit the application of numerical methods. In recent years, research related to machine learning has been revolutionized by the explosive growth of data and computational resources. Several researchers have considered using machine learning models to solve practical image restoration problems. A subset of researchers considers purely data-driven approaches to solve image restoration problems. This type of approach relies on historical data collected by measurement instruments combined with machine learning models to restore images. The principle of the formulation is as follows:
Some researchers have considered using machine learning models to solve partial differential equations for practical image restoration problems. This type of approach relies on the rich a priori knowledge from other fields to model the image restoration process as a partial differential equation and then use the machine learning model to solve the partial differential equation approximately to solve the problem. Such methods, which use machine learning models to approximate the partial differential equation and then solve the actual image restoration problem, are usually referred to as physics-informed machine learning methods for image restoration. The principle of the formulation is as follows:
The main advantages of image restoration-informed machine learning approaches over traditional numerical methods are as follows: (1)No grid dissection required: traditional numerical methods use simple functions, such as spline functions, as approximators. These approximators have very limited nonlinear expression capability, so they need to do the best possible dissection on the definition domain of the partial differential equation. By best possible dissection I mean that the variation of the solution over the subdomain of the dissection is as small as possible and the properties are as simple as possible, making it possible to obtain good approximation results for simple functions as well. Image restoration-informed machine learning methods generally use deep learning models as approximators. Deep learning models have a much stronger nonlinear representation than simple functions. This allows image restoration-informed methods to solve partial differential equations well even without grid dissection. This saves the computational overhead and storage overhead associated with grid dissection. The computational principle is as follows [6]:(2)Simplicity of use: the same numerical method handles the process differs between different partial differential equations. For example, finite difference methods have to use different difference formats on different partial differential equations. Image restoration-informed machine learning methods process essentially the same process between different equations with the following basic principles:(3)Explanatory: a purely data-driven approach is to learn image restoration laws directly from the data using machine learning models, which is akin to a “black box.” An image restoration-informed machine learning approach learns the corresponding image restoration laws by approximating the partial differential equations. Partial differential equations are mathematical models that have been summarized by domain experts [7] to describe scenes scientifically. With the help of partial differential equations, a mathematical tool that contains a large amount of domain-specific a priori knowledge, to describe a given image restoration process, the image restoration-informed machine learning method has a better interpretation. The principle of the formulation is as follows:(4)Good generality: purely data-driven approaches learn image restoration laws directly from data and need to consider specific image restoration constraints in a given scenario. For example, in the problem of predicting the surface temperature [8] of a lake, the relationship between the depth and density of the lake needs to be considered to improve the prediction accuracy, but such considerations do not apply to the EM scenario. Image restoration-informed machine learning methods are an approximate method of solving partial differential equations, and scenario-specific image restoration constraints are often considered on the partial differential equations and not on the model. It is perfectly possible to use different partial differential equations to describe the lake surface temperature problem and the electromagnetism problem, respectively, and then use the same image restoration-informed machine learning method to solve the equations without having to design a special model for a particular scene. The underlying principle equation is as follows:
3.2. Partial Differential Equations Based on PINN
PINN considers the use of deep neural networks to approximate the unknown solutions of partial differential equations. This paper also focuses on considering this deep neural network-based approach for solving partial differential equations, i.e., image restoration-informed neural network approach [9]. In a real image restoration problem, we need the amount of image restoration at some moment, on some spatial point or some small region. This requires that our model should not only have good performance overall but also have high accuracy at any point or any subdomain. How to improve the existing image restoration-informed neural network methods to have high accuracy on arbitrary points or arbitrary subdomains is an important problem. We refer to this problem as the accuracy consistency problem for image restoration-informed neural networks or the accuracy consistency problem for short. The principle is shown in Figure 1.
It is worth mentioning that traditional numerical methods use simple function approximations on the results of the mesh discretization to eventually obtain an approximate solution. When the mesh is dissected finely enough, the mesh can be thought of as a “microelement.” Approximators, such as fractional polynomials, are approximated on “microelements.” The nature of the solution on the “microelement” is homogeneous, and using the same approximator will give similar prediction accuracy. Therefore, the prediction accuracy of classical numerical methods is consistent [9] over a given definition domain. In fact, through numerical approximation theory, numerical methods can obtain consistent a priori error estimates. However, the image restoration-informed neural network method goes to fit the unknown solution of the partial differential equation over the entire definition domain, which is difficult to guarantee that the accuracy of the solution is consistent. The principle of the formulation is as follows:
The P1NN algorithm has two important ideas. One is that it encodes the physical information embedded in the partial differential equation into the loss function through a deep neural network and then optimally solves it to solve the specific problem. The above process is the embodiment of this idea, and many subsequent variants of the PINN algorithm have inherited this inspiration. The other is the single network strategy, which is based on automatic differentiation techniques to construct the loss function [10]. The existing physically informed neural network methods are all single-net strategies, but of course, they can be changed to multinet strategies. The multinetwork strategy does have an efficiency advantage over the single-network strategy, but it does not mean that the multinetwork strategy is superior in all cases.
4. Application of Image Restoration in Environmental Art Design
This chapter focuses on image restoration techniques, mainly for the image in the process of transmission and acquisition; due to the acquisition of the image network instability, transmission equipment accuracy will make the image part of the missing damage or pixel loss, or the shooting process to capture unimportant items, the image of the defective part of the repair [11].
4.1. Image Restoration Target Detection
The determination of the unknown mutilated region of the image is done using target detection, where the mutilated region is detected as a target and the output of the detection is used as an input to the restoration model. The basic formulation principle is as follows:
The speed of detecting the defective region should meet the real-time requirement, and the accuracy of detection should be high. This chapter focuses on the analysis of image restoration techniques for microbias equations. The basic formulation principle is as follows:
Figure 2 shows the basic flow of image restoration.
The main steps of target detection are firstly to generate candidate regional schemes (regional schemes) and eliminate negative samples, then to classify and locate candidate regional schemes and further refine and generate final test results. The basic principles are as follows:
Image detection algorithm based on partial differential equation is one of the common algorithms used in two-stage target detection. One-stage target detection does not require candidate region suggestion and can directly detect the class probability and coordinate position of the target object with a simple process. SSD, Yolo, and CornerNet are common single-level target detection algorithms. Typical single-stage target detection algorithms are the Yolov4 target detection network and Yolov3 target detection network. The basic formulation principle is as follows:
4.2. Image Restoration Target Detection Network
4.2.1. Network Framework
The Yolov4 target detection network inputs the target image into this target detection network; firstly, the features of the residual image are extracted by the backbone network CSPDarknet53 [12], and then, the features are put into the enhanced feature extraction network (Spatial Pyramid Pooling, SPP) to separate the contextual features and then through the path fusion network (Path Aggregation Network, PANet) for feature fusion and, finally, multiscale prediction using head of Yolov3; the specific network model diagram is shown in Figure 3.
4.2.2. Loss Function
The loss function of the Yolov4 target detection network is mainly composed of confidence loss (confidence) function, anchor box position loss function, and classification loss (L-class) together. The function formula principle of loss function of Yolov4 target detection network is as follows:
4.3. Encoders for Image Restoration
In the area of deep learning image restoration in the image processing field, context encode proposed in the literature can directly and quickly predict the structure of the mutilated regions and images, but training is more difficult. For generative adversarial network (GAN), a research hotspot in recent years, the effect of repairing high-resolution images is not very good, and in the training process, the generator network and discriminator network have the disadvantage of not converging easily during training, so the encoder-decoder (encoder) structure, which is commonly used in the restoration field, is chosen (decoder). The structure is chosen. The basic formula principle of its operation is as follows [13]:
Encoder-decoder, a network model that can generate another output information based on the input information, can be applied to the problem of Seq2Seq; Seq2Seq can be applied in translation and question and answer systems, etc. To solve the Seq2Seq problem, the encoder-decoder is proposed, which is mainly applied to data compression and subsequently used for image noise reduction, data generation, etc. The general structure of the encoder-decoder is shown in Figure 4.
Encoding means converting the input information into a vector of fixed length, while decoding means converting the fixed vector generated by the encoder into the output information . The encoder-decoder structure has many typical applications, but its structure has many drawbacks and limitations; the biggest limitation is that the structure only has a vector that connects the encoder to the decoder, which is usually called the fixed vector length which is the semantic information of. When the model works, the encoder needs to compress all the input information into the fixed-length semantic information [14]. However, due to this single structure, it will lead to the content information of the input at the beginning being overwritten by the content information of the later input, and if the content of the input information is longer, this operation will make the decoder decode with lower accuracy. The basic principle of its operation is as follows:
To solve such drawbacks of encoder-decoder usually, the attention mechanism is invoked; this model will focus on the useful information within the sequence information in the output and produce the output immediately afterward. The flow of such a model is shown in Figure 5.
5. Experimental Results and Analysis
The purpose of image restoration is to use the information available in the image to recover the missing parts of the image, as much as possible close to the original image. After the evaluation experiments in Chapter 3, the final stubby region detection section uses a partial differential equation-based image restoration method for the target detection network, and the Yolov4 stubby region detection network can quickly and accurately detect stubby regions in the image with different embedding strengths, locations, and sizes. Once the stubby regions in the image are detected, the detection results are used for the image restoration work in this section. The images produced by the common encoder-decoder structure lack depth detail features and do not focus well on the texture features of the mutilated regions, and the restored images are not satisfactory. To solve such problems, this paper introduces a self-attention mechanism on the encoder-decoder structure and improves self-attention. The encoder-decoder network structure based on self-attention proposed in this paper can pay better attention to the features of the mutilated regions so that the restoration of the mutilated images achieves a good restoration effect. In recent years, attention modules can model remote dependencies and attention mechanisms have achieved great success in various image processing tasks. To achieve global reference for each pixel, the self-attention mechanism has emerged. Considering each pixel in the image as a random variable, the covariance between the image prediction pixel and other pixels in the image is obtained, and the weights in it are obtained based on the correlation between each pixel and the target pixel, which is the weighted sum between all the pixel values [15].
5.1. Experimental Results
The self-attentive module proposed in this paper is used to capture invariant features from stubby regions with different angles and locations. In the encoder-decoder network with the addition of the self-attention mechanism, the convolutional layer will retain the feature maps from the previous convolutional layers. Using the encoder-decoder and the self-attention layer, the residual features of the image are obtained to ensure the quality of the generated image. The overall mutilated image restoration network uses the self-attentive based encoder-decoder network to complete the restoration of the mutilated image and calculate the difference between the generated result and the true value image (corresponding to the intercepted part of the undamaged region). The efficiency of the overall mutilated image restoration network model constructed in this paper for different restorations is shown in Figure 6.
For the encoder, its main focus is to acquire the residual image of the input and generate the potential features of the image, i.e., the pixels of the image. The prediction is made as a global constraint using the constrained structured prediction capability of the encoder based on the AlexNet framework. The role of the decoder is to generate the missing part of the image using the potential features generated by the encoder, and after passing through the decoder, the target size is reached by upsampling. The encoder and decoder are usually connected by a channel connection layer in between, to better locate the missing regions; then, this paper introduces an attention mechanism layer. Compared with the standard encoder-decoder structure, the self-attentive mechanism proposed in this paper focuses on the features of the residual image, and the convolution layer retains the feature regions of the previous convolution layer; using this encoder-decoder can obtain the features of the residual region of the image, thus ensuring the quality of the generated image. Batch normalization is used in the encoder; batch normalization can adjust the numerical output of the neural network to make it more stable; in this paper, batch normalization is done for each channel after the convolution calculation and before the activation function. Re LU nonlinear activation function is used for the encoder, and leaky Re LU activation function is used for the decoder.
The deep learning model structure has a complex and large number of implicit layers; the model has strong learning ability, but the model is prone to overfitting (overfitting) after training. Overfitting is also prone to occur when the model has a large number of feature dimensions, a small number of data samples, and a high level of noise. Dataset augmentation can be used during dataset preparation to prevent overfitting. Duplicate, change, or delete data in the training data set to balance the proportion of samples of different types of features in the dataset. For example, panning, rotating, or scaling existing images in the dataset expands the number of datasets so that the model can better learn the features of different types of images [16].
Before starting the experiments in this paper, the restoration network dataset is preprocessed, which is used in environmental art design, where is the input image with the defective area and is the target image (the ground truth). First, the images in the repair network dataset are partitioned into and images, and the and images are intercepted relatively larger than the input image size of the repair network, and then, the larger dataset is intercepted with the appropriate network input image size as the input image. The preprocessing of the dataset for the restoration network is mainly done by splicing the input horizontally, which is done to augment the data for the limited training dataset and prevent the occurrence of overfitting in the image restoration network. However, the code splits it during the training process, segmenting it into single images like and . The specific effect is shown in Figure 7.
5.2. Analysis of Results
The dataset of the defective image restoration network is also composed of the training set and test set; a total of 10,000 defective images and 10,000 true images are selected, with a total of 14,000 positive and negative samples [17], including 7,000 positive sample images and 7,000 negative sample images, where 70.00% of the prepared dataset (14,000 images) is used as the training set for the training of the image restoration algorithm. The remaining 30.00% (6000 images) of 3000 negative samples are used as test data to validate the effect of repairing the defective regions in this study [18]. For the 7000 negative samples in the training set of the restoration network dataset, the specific number of allocations constructed is shown in Table 1.
The restoration method of this paper is first compared quantitatively on the restoration network dataset created in this paper [19], and the average value of the loss is 2.21% using the scheme proposed in this chapter, proving that the method of this paper achieves the highest numerical performance to accurately infer the correct content, and details of the mutilated image are the input image, as shown in Figure 8.
Compare several models of image restoration, where the efficiency of restoration is shown in Figure 9.
The application of the partial differential equation-based image restoration method in environmental art and design is extremely important; with such an efficient restoration method, it can make the design of environmental art come out better [20].
6. Conclusion
In today’s fast-growing information age, a large amount of data is used in the form of images for daily communication and for environmental art design due to equipment transmission errors or network instability problems that lead to network packet loss, so image restoration technology came into being. Image restoration techniques can enhance the quality of the image being processed and gain visual comfort. Traditional restoration methods have drawbacks such as time-consuming and poor restoration results.
To solve such problems, an image restoration method based on partial differential equations is proposed in this paper. The method mainly consists of two parts: stubby region detection and stubby region restoration. The mutilated region detection part involves constructing the mutilated image dataset and the validation set of different influencing factors. For mutilated region restoration part, in this paper, an encoder-decoder-based self-attentive mechanism for mutilated image restoration algorithm is proposed. A new self-attention mechanism is designed and combined with the coder-decoder to address the problem of ambiguity in the repair effect, a joint reconstruction loss and an adversarial loss function are proposed, and an Adam optimizer is selected for detail optimization. The mutilated regions detected by the detection network are appropriately extended range clipping, and the clipped positive and negative samples are preprocessed by horizontal splicing. The experimental results show that the network is effective in repairing the mutilated region with a small area, low embedding strength, and simple background. Cross-sectional comparisons with other image repair algorithms are made, and the results demonstrate that the self-attention-based encoder-decoder stumped image repair algorithm proposed in this paper can effectively repair the stumped region and achieve high efficiency. This technology will continue to improve its processing efficiency and repair details. In the future, the quality of image repair will be higher and higher, and the repair of details will be better controlled.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The author declares that there are no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.