Abstract
Dunhuang frescoes are one of China’s precious historical and cultural heritages, but this treasure trove of human art has become very fragile due to natural weathering damage and the threat of major natural disasters. Therefore, the preservation of Dunhuang murals has become an urgent task. Digital fresco conservation and restoration work can preserve fresco information permanently, and can use computers to virtually repair frescoes, assist in fresco conservation, assist in fresco copying, virtually evolve the fresco disease process, and virtually display fresco realism. Digital fresco conservation and restoration provides sufficient scientific basis and testing environment for the physical conservation and restoration process of frescoes, and minimizes the danger of conservation work. This paper presents a new approach to digital conservation and virtual restoration of frescoes. Based on research by Dunhuang artists on the mural restoration process, research on pigment composition, knowledge accumulated by the artists’ experience, and some of the better-preserved murals, digital image processing, computer graphics, artificial intelligence, and other technologies are synthesized to achieve the virtual restoration of murals. Aiming at the current situation of Dunhuang mural restoration, combined with the actual painting process, this paper puts forward a set of computer-aided mural restoration solution based on intelligent generation of line graph. It can be well applied to the actual restoration work. It is a new attempt of computer-aided restoration of ancient paintings.
1. Introduction
Among China’s vast and colorful treasure trove of frescoes, Dunhuang frescoes are unique and distinctive, with different styles and characteristics while following some basic rules over more than a thousand years and ten dynasties [1, 2]. Dunhuang frescoes are one of the precious historical and cultural heritages of China, and the art of Dunhuang frescoes is an outstanding representative of Chinese classical art civilization. However, this treasure trove of human art has become very fragile due to natural weathering damage and the threat of major bairan disasters [3]. Because fresco pigments contain lead and other components that are easily oxidized, and are affected by external factors such as light, oxygen, temperature and humidity, industrial gases, mold, bacteria (of which sunlight and temperature and humidity are important factors), these internal and external factors act simultaneously to make the frescoes constantly suffer from diseases, the main types of diseases are: ground tent peeling, hollow drums, cracks, scratches, crispy alkali, nail, herpetic peeling (dot peeling is included in (within the herpetiform shedding), frost, blistering nail, mildew, smoke, pigment layer peeling, residual pigment layer, residual white powder layer, and white powder layer peeling [4, 5]. The slow intensification of the disease then led to significant breakage of the murals. There are three types of damage: the first is discoloration, in which the color of a certain pigment slowly changes to another color under the action of light and other factors over a long period of time. The second is the fading damage, the original bright and shiny color due to the pigment particles off or adsorption of dust mixed with dust color and gradually fade. The third type is exfoliation, in which the pigment layer in the same area of the fresco is completely removed due to disease or natural causes, revealing the color of the wall and gradually expanding [6].
How to preserve the splendid culture of Dunhuang forever and how to rescue these endangered artistic treasures has been a struggle for every generation of Dunhuang scholars. When the wave of digitization came into everything at the end of the twentieth century, Dunhuang preservationists were ahead of the curve and began research on large-scale computerized storage and reproduction systems [7]. Geometric correction and color reproduction of high-precision stereoscopic photographs obtained by close-up photogrammetry were digitally scanned to create a huge database, so that all the pictures and texts were dissolved in a sea of binary digital; the thousands of poses on the walls and in the caves were instantly available for remote dissemination, lossless reproduction, and easy retrieval via the Internet. Based on the digitization of cultural relics, modern computer technology has brought a whole new concept and revolution to the conservation of precious Dunhuang mural art [8]. Digital image processing, computer graphics, and artificial intelligence are widely used in the conservation and restoration of cultural relics, including Dunhuang murals [9].
Over the past few decades, digital image processing technology has made rapid advances and its applications have exploded. In these short decades of history, it has been successfully applied to almost all imaging-related fields. In recent years, the field of digital image processing has grown rapidly [10]. Digital images have become an effective tool for scholars in psychology, physiology, computer science, and many other areas to study visual perception. The demand for image processing in military, remote sensing, meteorology, and other lesser and type of applications is also increasing [11]. The rapid development of digital imaging, image printers, and digital printing technology has made image processing equipment no longer expensive, providing excellent conditions for the development of image processing [12]. After years of hard work, computer workers around the world engaged in image processing research have come up with a number of systematic theories and algorithms. The classical image processing theory has become the theoretical foundation of the contemporary computer image processing field, which includes image digitization, digital image display, image operations and transformations, image restoration, image compression, pattern recognition, and processing of color and multispectral images and two-dimensional images. Based on the classical image processing theory, a large number of software and hardware products for image processing have been developed and put into use, such as the widely used and famous image processing software Photoshop from Adobe.
The current rapid development of image graphics processing, artificial intelligence, computational geometry, multimedia databases, and content-based information retrieval technologies provides a scientific basis and advanced means to achieve virtual restoration recreations of Dunhuang murals for the study of Dunhuang art, cave archaeology, and heritage conservation [13]. The use of various computer technologies and domain-related knowledge and experience (research by Dunhuang conservationists on the composition of fresco pigments, artists’ accumulated empirical knowledge, and some of the better-preserved frescoes currently available) to restore Dunhuang images and to estimate the fading and discoloration of images according to the current environment is of great reference value to Dunhuang researchers. Therefore, this paper hopes that the research will give a new and intelligent way to implement the virtual mural restoration and make some meaningful process demonstrations using the restored results [14].
This paper is organized as follows: Section 1 describes the research background of this paper and the main structure of this paper; Section 2 introduces the current status and international research in related fields and summarizes the research significance of this paper; Section 3 analyzes the restorative copying process of Dunhuang brushstrokes and proposes the restoration model of this paper. Section 4 provides an objective analysis and evaluation of the results under different parameters, and substitutes the optimal parameters into the restoration model. The proposed algorithm obtains the best restoration results in the comparison experiments based online drawings and expert guidance of broken murals. Section 5 summarizes the research content of this paper and provides an outlook on future research directions.
2. The Related Works
Computer art and related studies have become a branch of computer graphics, and SIGGRAPH conferences in recent years have set up the topics for discussion. The World Computer Graphics Conference held in the United States in July 1994 had a large proportion of computer art content and a special edition of selected computer art works [15].
Theoretical research on computer art covers various aspects: the nature of computer art creation and its social functions, the impact of computer technology on artistic functions and forms, and the roles of computer engineers, commercial artists, and academic artists in creation; however, there are still no mature conclusions. In terms of application technology development, the representative ones are based on computer 3D modeling and realistic display technology, using genetic algorithms of biological evolution, mainly for film and television creation, sculpture art, etc.; in terms of development trends, the emergence of new technologies brings new tools and methods to computer art creation and artistic expression, from simple browsing of works on a single machine to interactive appreciation by users in a network environment The means of creation has evolved from the behavior of individual artists to the direction of real-time collaborative creation by groups [16].
At present, if the restoration of cultural relics is done by conservation experts and artists, based on the historical documents and similar works in relatively good condition, the study of folk customs of different historical periods and the styles and techniques of different art disciplines, the drawing techniques and color assignment rules of cultural relics of each historical period are summarized, and then restoration copying is done manually before the actual restoration is possible [17]. Restoration copying is a very important part of this traditional manual approach. A comprehensive and deep understanding of the above-mentioned contents requires long-term research and accumulation, and it takes a long time to present this abstract knowledge in the form of copying. In addition, the non-repeatable and non-retraceable nature of manual copying makes this work more complex and difficult, requiring a high level of professional and comprehensive quality from the restorer [16]. Therefore, the use of new techniques and methods in this field is undoubtedly challenging and has great promise for application.
At present, there is a large gap between the theoretical research and applied technology of computer art, and the theoretical research is relatively advanced; the applied technology lacks the guidance of formed theoretical models, and the deep-level research on the simulation of the art creation process is still at the stage of theoretical exploration; the application focuses on the creation of new art works, while less consideration is given to the restoration and protection of traditional art works, and in terms of technology, it mainly relies on computer graphics, and less consideration is given to the applications focusing on the creation of new artworks and less consideration is given to the restoration and conservation of traditional artworks, relying mainly on computer graphics in terms of technology, with less consideration given to the integration of multimedia and intelligent technologies [18]. Classical image processing techniques still dominate the art restoration techniques. Nowadays, the more widely used commercial software, such as Photoshop, CorelDraw, provides users with some conventional tools for pattern creation and image processing. At the same time, more and more computer workers and artists are discovering that there are many problems in the field of computer art restoration that cannot be solved by classical image processing methods or are not satisfactorily processed. Generic commercial software for different fields of restoration requirements, their general functions also appear to be quite limited. Therefore, different fields of restoration have proposed more targeted image processing techniques. For example, IEEE Multimedia Computing and Systems Conf. 98 has presented several papers proposing new image processing techniques based on artistic restoration, and EUROGRAPHICS. SIGGRAPH, China Graph, and other important academic conferences in China have also discussed these topics in-depth [19].
In the last decade, image processing techniques are widely used in numerous scientific and engineering fields, such as remote sensing industrial inspection, medicine, meteorology, reconnaissance, communication, intelligent robotics, and are clearly changing the means of production and the way of life [20]. In all these applications, some image processing techniques will be encountered: enhancement, coding, reconstruction, segmentation, and recognition. A lot of work has been done in Greece, Italy, and Japan on virtual restoration of the cultural objects. The Kyushu Institute of Technology in Japan has studied virtual exhibitions about the Terracotta Warriors in Xi’an, China, and techniques for restoring the color of the armor on the terracotta warriors. Their research focused on the virtual restoration of a two-dimensional model of the terracotta warriors and horses, and then the colors were filled in manually using tools such as Photoshop. In contrast, work on the restoration of ancient paintings has been carried out more often in Greece and Italy, and from the information available a few days ago, work on the treatment of ancient images has focused on the removal of the oxidized discolored layer of ancient paintings (decontamination) and the elimination of cracks [21]. In these two areas, a lot of research and applications have been carried out.
Research and practice in recent years have shown that the computer-assisted digital conservation and restoration has been a great success. This method has no risk of damaging cultural relics, and has the advantages of efficiency and repeatability that traditional methods do not. Artificial intelligence and image processing techniques can bring the original appearance of ancient frescoes back to life. Based on research on the restorative mural copying process by Dunhuang artists, combined with research by Dunhuang conservation experts on the composition of mural pigments, the knowledge accumulated by artists, and some of the better-preserved murals, this paper aims to investigate a practical, computer-aided restoration method that is consistent with the characteristics of mural restoration, using a combination of image graphics processing, artificial intelligence, and other computer technologies. This will help the artist to generate certain restoration results on the computer and provide the restoration staff with restoration solutions and reference suggestions. The frescoes generated by the assisted restoration are combined with virtual modeling and roaming technology and multimedia information management, which can be applied comprehensively to various aspects of endangered cave relics, such as site investigation, disease labeling and categorization, disease distribution analysis, disease research and discussion, and heritage conservation and restoration. Also it aims to open up new methods and provide effective research tools for the conservation and study of endangered cave relics; and to promote the research and development of multimedia, virtual reality, and image processing technologies. Computers provide such operations to improve the efficiency of fresco conservation work. At the same time, the provision of some related services can also reduce the number of people entering and leaving the cave and the length of stay, which is conducive to the conservation of cave relics.
3. Uncertain Logistics Transportation System Model and Online Route Planning Scheme
Due to the limitations of traditional materials and technology, as well as the impact of external natural environment and air pollution, the weathering and aging problem of oil painting is very serious. If the mural has a history of about 30 years, it needs to be protected and repaired. The study of computer-aided mural restoration techniques refers to the traditional methods and processes of restorative copying, and aims to overcome the limitations and difficulties of the method. As mentioned earlier, we have developed a prototype system for virtual restoration of ancient frescoes that focuses on fading and discoloration of frescoes intelligently and has achieved relatively successful results. However, an in-depth survey of Dunhuang artists’ fresco restoration work revealed a strong need for easy computer application to the fresco restoration process. In particular, the use of computers to generate relatively accurate line drawings would greatly improve their work efficiency. In this context, we apply computer technology to traditional restoration copying and address the current problems in restoration copying from two broad perspectives.(1)Reduce the workload of the Dunhuang artist by utilizing computer-generated line drawings for all or most of the site. Provide a search engine of line drawing elements for base and feature points to assist the Dunhuang artist in determining restoration strategies.(2)The use of computer-assisted color restoration of murals has provided Dunhuang artists, archaeologists, and conservators with an intelligent set of color restoration solutions.
Starting from these two problems, this paper designs a new model and process for computer-aided fresco restoration. The specific restoration process can be divided into the following steps.(1)Image segmentation and edge detection: there are many styles of line drawings in Chinese painting, but for aiding restoration, the main purpose of this paper is to provide a relatively accurate line drawing for the artist. Therefore, the edge detection method can be used to obtain a preliminary contour line drawing without the stroke style, in order to prepare for the generation of an accurate line drawing.(2)Replacement of line drawing elements: since most of the murals to be restored have missing parts, it is impossible to get the complete line drawing requested by the artist by directly using segmentation and edge detection. Therefore, those missing parts need to be added interactively using the computer. In this paper, a library of replacement line drawing elements is created to provide editing, entry, query, and replacement of the elements. A preliminary line drawing is obtained by replacing and drawing the severely missing parts.(3)Post-fine revision: after the replacement of the main damage parts is completed, there is an interactive fine revision process. The restorers can make some fine adjustments to the details of the first draft of the line drawing according to their domain knowledge. The result is a line drawing that can be used for color assignment.(4)Divide the area of the meter according to the line drawing to establish the fill skeleton, then use the brush model and to draw. Or use texture-based synthesis of art style learning for color restoration.
3.1. Image Segmentation and Edge Detection
Image segmentation is a major problem in image processing and also a classical problem. Although the image segmentation problem has attracted a lot of researchers to make great efforts since the 1970s, there is no general method and no objective criterion to judge the success of segmentation so far.
The first step in computer-aided restoration of Dunhuang murals is to generate line drawings of Dunhuang murals, and one of the first steps done is edge detection and segmentation of color images: segmentation of the images is also required for later regional coloring of the murals. Unlike grayscale images, color also includes useful information such as hue and saturation, and color image segmentation can be achieved by computing statistical features of the color information in the image.
Image segmentation refers to the differentiation of different regions in an image that have a special meaning. These regions are disjoint, and each region satisfies the consistency of a particular region. The segmentation of an image , where , is to divide the image into subregions that satisfy the following conditions :(1), i.e., all subregions make up the whole image;(2) is the connected area;(3), i.e., there is no common element in any two subregions;(4)The region satisfies a certain uniformity condition, which means that the difference in gray value between pixel points in the same region is small or the gray value changes slowly.
Commonly used image segmentation techniques can be classified into four categories: feature thresholding or clustering, edge loosening, region growing or region extraction, and recursive pixel classification. Although these methods segment grayscale images with good results, they often fail to achieve the desired results for color image segmentation.
In the current situation where it seems impractical to perform the image segmentation task entirely by a computer, the approach of allowing the computer to realize the segmentation task with the help of a human, i.e., human-computer interaction, has attracted widespread attention. Satisfactory results may be obtained if the human role is organically incorporated into the image segmentation process, i.e., when qualitative problems are to be solved, appropriate hints are given by the human, while the precise computational processing work is performed by the computer. Based on the above considerations, this paper uses region growing and region merging methods to segment mural images.
Let the color of pixel i to be judged be shown by the vector , the mean value of cluster k be represented by the vector , and E{arg} be the expected value of arg. The common distance measures to determine whether X belongs to cluster k are Euclidean, Mahalanobis, maximum likelihood judgment, etc.
The Euclidean distance is the most common measure of the proximity of two colors and is defined by the following equation:
If the vectors in class k obey a multidimensional Gaussian distribution, the Mahalanobis distance metric can be used, which is defined aswhere is the covariance array of the class, which can be calculated by the following equation:
The most human likelihood method of clustering minimizes the average error of the clusters. Let the joint probability function of X belonging to class k bewhere is the probability of occurrence of class k, and is the conditional probability of occurrence of class k when X occurs. According to the Bayesian judgment rule, when the discriminant function is minimal for k, the maximum likelihood will be obtained as follows.
If , then the vector X is classified into class k = c.
In practical application, we adopt the Euclidean distance formula as the criterion to judge the similarity of two colors, and the three components are combined with HSL color space and RGB color space.
Since the white motion separated mural images do not meet the practical requirements and it is mainly the variegated colors that are to be processed. Therefore, we use the interactive region growing method to extract the color regions. Only regions that satisfy the color similarity metric are considered as target regions. During the region growing process, the location and size of seed points are selected interactively by the user. The distance metric for region point clustering is
The color value of the pixel point is , the color value of the expansion point of the region adjacent to it, the average color value of the growing region, and the color value of the seed point are and respectively; and are the corresponding three thresholds, and their sizes should generally satisfy.
For frescoes, in addition to the influence of noise, the intensity of the colors used in painting, the mutual integration of different colors, coupled with the wear and tear of the fresco surface, peeling, etc., are important reasons for this color area and contains numerous small areas.
The key to region merging is to decide how much of the region can be considered as fragmented areas that do not have physical meaning. Choosing too large a size will leave out some color areas that should be segmented, while choosing too small a size will not achieve the goal of removing fragmented areas. Experimental results show that the size of the fragmented area is 5 × 5 pixels.
Of course, the size of the fragmentation area does not have to be a constant 5 × 5. Generally speaking, if the value is too small, the segmentation result is poor; while if the value is too large, meaningful color objects may be missed. It is possible to set a threshold value that can be adjusted interactively by the user.
3.2. Establishment and Matching of Feature Models of Line Drawing Elements
The same iron line depiction is different between Northern Wei and Western Wei, the characteristics of Northern Wei are stable and Western Wei are dynamic. As shown in Figure 1, it is impossible to depict it with a simple function, yet a description of the morphology of the various compositional elements is essential to the restoration of the fresco line drawings. The style of Dunhuang fresco line drawings is generally smooth, with each stroke being made in a single stroke. The versatility and variability of the Bezier curves can be easily approximated to show the colorful curves of the compositional elements.
Bezier curves have many excellent properties that make them convenient and easy to implement in the curve formulation. A Bezier curve segment can be fitted to any number of control points. These control points are approximated and their associated positions determine the number of Bezier polynomials. In general, a Bezier curve times can be defined by n + 1 control points :where is the Bernstein polynomial:
A complete line drawing element can be defined as a set of a fixed number of strokes S. The number of strokes is the full set of the number of strokes needed to draw a line drawing of the same type of element, analyzing the characteristics of line drawings from different periods for different compositional elements of Dunhuang wall paintings. For each stroke, we also define a switch quantity. The stroke lines consist of the set of control vertices mentioned in the above section, the corresponding pressure parameters, and the line width parameters.where is the i-th stroke in the element model, for a complete eye element model above, as shown in Figure 2.
For each stroke curve of , it can be defined as:where is a switch quantity to represent the different types and styles of eyes, since we are defining a complete eye line drawing model, not all eyes have these characteristics, for example, some eyes are single eyelids and some eyes have no lower curtain line. So we define a switch quantity to personalize the representation of each special case. means that this curve does not need to be drawn; means that this curve needs to be drawn. indicates the width of this stroke curve. is the coordinate of the control vertex of the stroke curve, where m is the number of days of the control vertex of the stroke curve, and this number is fixed for each particular stroke.
is the parameter that controls the pressure at the apex and end points.
3.3. Line Drawing Coloring
It is not easy for an artist unfamiliar with the art of fresco painting to debug a restored image in the Dunhuang style. If we can transfer the color and texture styles of the better-preserved fresco images to the frescoes to be restored, then we will get better restoration results. The basic accepted forms of similarity and analogy are.
This shows that if the attributes s1 and t1 in the source set S and the target set T are similar, it can be inferred that their other attributes s2 and t2 may also be similar. The idea of image analogy is to “transfer the style of one image to another image.” This approach does not require the use of a specific, concrete filter and the iterative setting of the parameters of the brush model, and a general approach can yield a large number of images with different visual effects.
We summarize the object-oriented image art style learning algorithm in the following equation:
Among them, indicates color objects containing different art styles (either the whole image with a uniform style or a color object of an image), and T is the control image, which in this case should be the image representing the color object after processing based on the line drawing of the mural. is the image of the restoration result output. indicate the feature value of the target image and the p-point of the result image. In the algorithm processing, it is necessary to copy the q points in the sample map to the p points of the result map, and for this purpose a data structure s(.) indexed by p is created.
The matching algorithm uses KNN and Ashikhmin algorithm, according to the two algorithms to obtain the points to be selected from the sample map, respectively, and then select the best point from the points to be selected, copy the point into the result map, and update the value in s(p) at the same time.
The well-preserved image is divided into different color objects as the object of artistic style learning, and then the corresponding relationships between these color objects and different objects in the control chart are specified, and these corresponding relationships are represented on different color blocks of the control chart. Through the above methods, we can restore the virtual color of Dunhuang murals while maintaining the original artistic style of Dunhuang murals.
4. Experimental Results and Analysis
In order to verify the effectiveness of this paper, three broken murals are restored and compared with the most mainstream Criminisi algorithm and Wang algorithm in image restoration algorithms, and the experimental results are analyzed. The mainstream Criminisi algorithm is a sample block repair algorithm based on matching. It is mainly divided into two aspects: (1) for the block selection method (i.e., filling order); (2) Block filling method (i.e., how to select matching blocks and how to fill them) in terms of filling order. The filling sequence is determined by using gradient features. The filling order is determined according to the DCT coefficient of the block to be filled. Figures 3 and 4 show the Peak Signal to Noise Ratio (PSNR) curves and Structural Similarity (SSIM) curves for ten images with different block sizes to be restored, respectively. It can be seen that the size of the block to be repaired has a significant effect on the restoration results, and the trend of the curves shows that the larger the repair block, the lower the PSNR and SSIM, for the following reasons: (1) when repairing the texture region, the larger the repair block is, the less texture blocks are repaired, i.e., the more structural blocks are left, which increases the difficulty of subsequent repair; (2) when repairing the structural region, the larger the block to be repaired, the more larger the block to be repaired, the more complex the structural information is, so it may not be possible to find the best matching candidate block, which will affect the results.
Figures 5 and 6 show the PSNR and SSIM curves of the three images with different numbers of candidate blocks, respectively. It can be seen that the number of candidate blocks does affect the restoration results. Analyzing the trend of the curves, the restoration results show an increasing trend with the increase of the number of candidate blocks, but then tend to fall back, which indicates that the more the number of candidate blocks, many irrelevant candidates may be introduced, which will affect the restoration results.
As can be seen in Figures 7 and 8, for each image, the proposed restoration scheme reaches the maximum PSNR and SSIM, and the average PSNR and SSIM values of the proposed scheme also reach 28.97 and 0.9631, which are much higher than the 26.85 and 0.9529 of Criminisi and Wang algorithms, and 26.09 and 0.9446.
As can be seen in Figure 9, Wang’s algorithm takes the longest time, which is due to the fact that the method uses global search in searching for candidate blocks, and this paper also has a significant advantage over Criminisi algorithm, which is due to the fact that the method uses multiple scales in restoring images and requires solving the energy equation. In conclusion, the experiments demonstrate that the proposed scheme in this paper has significant advantages both in terms of restoration results and time.
5. Summary and Outlook
Based on the research of Dunhuang artists on the restoration process of murals, this paper makes comprehensive use of pigments. The knowledge accumulated by artists’ experience and some well-preserved murals, digital image processing, computer graphics, artificial intelligence, and other technologies realize the virtual restoration of murals. In view of the current situation of Dunhuang mural restoration, combined with the actual painting process, a set of computer-aided mural restoration solution based on intelligent line drawing generation is proposed. It can be well applied to the actual repair work. This is a new attempt of computer-aided restoration of ancient paintings. This study can be well applied to the actual restoration work, and is a new attempt of computer-aided restoration of ancient paintings. Although preliminary results have been achieved, there are still some problems to be solved. There is still more interaction when editing and inputting line graphic elements. How to generate line graphic elements efficiently is a problem that needs to be solved in the further work.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by Anhui Provincial Education Department Scientific Research Project: Research on digital protection and virtual museum construction of “Huizhou Temple Murals of Ming Dynasty” (No. SK2021A0260); Anhui Provincial Education Department Scientific Research Project: research on color digitization analysis and knowledge reuse of “Phoenix Painting” in Anhui province (No. SK2021A0263).