Abstract
Many ancient wall paintings are confronted with the threat of irreversible damages and in urgent requirement of restoration. This work provides the superpixel segmentation method and pigment identification method for the visible spectral image of ancient wall paintings to guide the scientific restoration of the paintings. The superpixel segmentation method for the visible spectral image is an extension of SLIC (Simple Linear Iterative Clustering) for the RGB image by redefining the feature of the visible spectral image. It can extract the outline of wall paintings and limit the pigment filling area in restoration of wall paintings. 44 kinds of commonly used pigments with size variations are selected to construct a visible spectral reference database for pigment identification. The pigment used in each superpixel is identified by searching the database in a specifically constructed feature space to find the nearest reference sample. This can provide guidance to pigment selection in restoration of wall paintings. At last, the methods are validated using the visible spectral image captured from Mogao Grottoes in Dunhuang by using a multispectral imaging system.
1. Introduction
As an important language and vision element to artistic creation, color has symbolic and expressive applications in paintings to present the artistic effect of the paintings and to convey the subjective emotions of the artists. Affected by various factors such as temperature, humidity, illumination, insect pest, microorganism reproduction, human activities, and natural disasters [1], however, the color of ancient wall paintings is changing and fading at an irreversible and ever-increasing speed. The constitution of a scientific and integrated system to protect the color of ancient wall paintings has become more and more urgent. It is fortunate that the rapid development of digital technique in recent years has provided new approaches and ideas for the color preservation of ancient wall paintings.
Multispectral imaging technique is one of the most extensively and thoroughly investigated digital techniques in recent years for the digitalization protection of cultural heritages [2–4]. Multispectral imaging systems can be constructed by using a RGB camera with two continuous spectrum light sources and some broadband color filters, or a monochrome camera with some narrowband color filters or LED light sources. It is low in cost and convenient to construct. Moreover, the visible spectral image captured by the system usually has high spatial resolution, and each pixel of the image records the visible spectral reflectance of the paintings at the corresponding position. As is well known, visible spectral reflectance is an intrinsic property of materials and can be utilized to characterize the color of materials that is robust to illumination and observer changes [5]. It means that multispectral imaging technique can record the high-spatial-resolution information and authentic color information of the materials simultaneously. Therefore, it is widely employed in digital acquisition, archives, and exhibition of cultural heritages since it can endow cultural heritages with eternal virtual life. Our team has also constructed a multispectral imaging system and applied it to the digital acquisition of ancient wall paintings [6]. So far, we have obtained many visible spectral images from Mogao Grottoes of Dunhuang. For details about the images, please refer to our report [6].
In order to improve the visual effect and to prolong the entity life of the ancient wall paintings, it is necessary to implement some restoration to the ancient wall paintings. The selection and filling of pigments is of critical importance to the restoration of ancient wall paintings in that pigments are the material base of the ancient wall paintings to present color and they determine the outline and color of the restored paintings. Subjective judgment of pigments and their filling by restorers often results in destruction of the ancient wall paintings. A scientific guidance to restorers is required for the selection and filling of the pigments in the restoration. Visible spectral images captured by the multispectral imaging technique records the two-dimensional spatial information of the ancient wall paintings. The outline of the paintings to determine the filling areas of pigments can be extracted from the images. On the other hand, the type of the pigments to guide the selection can be inferred from the spectral reflectance of the visible spectral image since spectral reflectance is an intrinsic property of materials and it carries the chemical composition information of the materials. Now that many visible spectral images have been acquired from cultural heritages, it is interesting to exploit the approach to determine the type of pigments and their filling area from the visible spectral image to guide the scientific restoration of the ancient wall paintings.
The outline of the ancient wall paintings can be determined by segmentation of their visible spectral image. Earlier research studies involving the segmentation of the visible spectral image mainly gave service to the pigment identification of colored relics. Some research studies [7–10] regarded each pixel of the visible spectral image as one unit and identified the pigments pixel by pixel. While this can provide relatively accurate pigment identification results, it is difficult to locate the corresponding position of each pixel in the colored relics. So it cannot determine the filling areas of the pigments in the colored relics. Other research studies [11–14] applied traditional image segmentation methods to partition the visible spectral image into multiple regions and then implemented pigment identification for each region. In the image, each segmented region may contain multiple simply connected regions. The identification accuracy of these methods is worth discussing since the number of the simply connected regions in the segmented image is difficult to be predefined. Moreover, they cannot guide the restoration of colored relics effectively since the scale of the simply connected regions is also difficult to be predefined.
In this work, we first propose the superpixel segmentation method of the visible spectral images to obtain the outline of the images. Each superpixel is a simply connected region and applied as a guidance to determine the filing area of the pigment. Then, we offer the identification method of the pigments based on visible spectral reflectance of the segmented images to guide the selection of the pigments in restoration of the paintings. The contents of this paper are structured as follows: Section 1 introduces the backgrounds that give rise to this work. The superpixel segmentation method of the visible spectral image is described in Section 2. Section 3 gives the pigment identification method of the ancient wall paintings based on visible spectral reflectance. The conclusions and expectations of this work are presented in Section 4.
2. Superpixel Segmentation
Image segmentation aims to partition an image into multiple regions such that the pixels in the same region have the similar values while they are different in different regions. Because visible spectral reflectance is the “fingerprint” of materials, the pixels in the same region of the segmented visible spectral image represent that the pigment is identical in this region. It means that the boundary of each segmented region can be applied to guide the filling area of pigment in restoration of ancient wall paintings.
Image segmentation is a commonly used technique in digital image processing and analysis. Researchers have proposed many methods so far for image segmentation, for instance, thresholding method [15], edge detection method [16], region growing method [17], clustering method [18], and graph cutting method [19]. Superpixel segmentation is an oversegmentation of images and targets to group pixels in an image into atomic regions. It has been widely explored in recent years because its superpixel boundaries align well with the boundaries of the natural object. The existing method of superpixel segmentation can be grouped into two types: graph-based methods and gradient-ascent methods. Graph-based methods treat the image as an undirected edge-weighted graph. Each pixel of the image is regarded as a node of the graph. The edge between two nodes is weighted by the similarity of the nodes. Superpixels are obtained by minimizing an energy function defined over the graph. Normalized Cuts [20], Graph Cuts [21], and Entropy Rate [22] are typical representatives of this kind of method. Gradient-ascent methods usually start with a rough partition of the image and refine the partitions iteratively until the segmented superpixels satisfy some pre-required convergence criterion. Watershed [23], SLIC [24], and SEEDS [25] are typical representatives of gradient-ascent methods. Recently, researchers proposed some content-adaptive superpixel segmentation methods to further improve the adherence of the superpixel boundaries to the object’s boundaries. Each type of the methods has its own advantages and shortcomings. The effectiveness of the methods often hinges on the specific application environment. Most of the methods are oriented to the superpixel segmentation of RGB images. In this work, we will extend superpixel segmentation to visible spectral images by redefining and constructing the feature of visible spectral images.
2.1. Mathematical Modelling
We assume that dataset is a visible spectral image with pixels, where and denotes the number of rows and columns of the image, respectively; denotes the spatial domain of the image; and the integer pair denotes the spatial coordinate of the pixel. The visible spectral reflectance of the pixel at is formulated as
Here, represents the spectral value of the pixel at wavelength . It is notable that the visible spectral wavelength usually ranges from 400 nm to 700 nm at 10 nm intervals in consideration of data volume and accuracy for color computation. In other words, the visible spectral reflectance can be regarded as a vector.
The superpixel segmentation of the visible spectral image is the task that partitions the image into spatially simply connected, nonoverlapping, and nonempty regions, . The union of all the regions is the entire image. Each region corresponds to one superpixel, and the pixels in the same region have the similar attribute. The set form of the segmentation can be expressed as follows:(1).(2), and .(3), .(4) and , . is a similarity criterion.(5), , is a simply connected region.
2.2. Superpixel Segmentation for the Visible Spectral Image
After comprehensive comparisons of all the superpixel segmentation methods for the RGB image, we found that SLIC is a most outstanding method among them. Its computation and storage complexity is low while its outline adherence is high. It provides parameters to adjust the regularity of the segmented superpixels. Moreover, it has excellent scalability and can be easily extended to the superpixel segmentation of the visible spectral image. Thus, we select SLIC to the superpixel segmentation of the visible spectral image in this work by reconstructing its feature.
SLIC can be regarded as a variant of k-means clustering. It starts from -given centers and works in a feature space that is constructed by the weighted combination of pixel value and its spatial position. The pixels whose features are most similar to that of a certain center are classified into a cluster. Then, the centers are adjusted iteratively according to the clusters until the difference between the new centers and the previous centers reaches a predefined converge threshold. Because the pixel features of the RGB image are different from that of visible spectral image significantly, it is necessary to reexamine the pixel feature of the visible spectral image. Given a visible spectral image with and , we define its feature as a weighted combination of visible spectral reflectance and its spatial position, i.e.,where is a weighting factor applied to adjust the contribution of visible spectral value and spatial position to the feature and then adjust the regularity of the superpixels and is a parameter determined by the size of the image () and the number of cluster centers (), i.e., , to normalize the spatial measure. Accordingly, the similarities of the pixels in a visible spectral image can be measured by their Euclidean distances in the feature space spanned by equation (2).
In accordance with the principle of SLIC mentioned above, the extended SLIC for superpixel segmentation of the visible spectral image can be described as follows:(1)Select initial centers in the image. The centers are sampled on a regular grid of step in the spatial domain of the image.(2)Search the pixels in the neighborhood of each center and assign each pixel to the cluster whose center has the minimum Euclidean distance to the pixel in the feature space.(3)Update the center of each cluster using the average of all the pixels in the cluster.(4)Estimate the difference between the new centers and the previous centers. If it dissatisfies the predefined threshold, repeat (2) and (3). Otherwise, end the iteration.(5)Label the clusters and each cluster corresponds to one superpixel.
The extended SLIC is carried out on the visible spectral image of ancient wall paintings captured from the south wall of cave 296 in Mogao Grottoes. The image is obtained by using a multispectral imaging system developed by using a RGB camera with two continuous-spectrum light sources and two optimal broadband color filters. The size of the image is . The number of the superpixels is set to 500. After a series of tests, the optimal weighting factor is set to 0.5 and the coverage can be reached after about 15 iterations for most visible spectral images. The results are shown in Figure 1. It is notable that the visible spectral image is converted into the RGB image for visual presentation.
(a)
(b)
(c)
3. Pigment Identification
Different pigments have different chemical compositions and then present different absorption and reflection characteristics. It is not a stretch to infer that the visible spectral reflectance of the pigment carries the chemical composition information of the pigment. It can be utilized as a tool to identify the pigments used in ancient wall paintings when it is reasonably treated.
As a nondestructive analysis technique, visible reflectance spectroscopy has been widely used in the pigment identification of colored relics. So far, many methods have been proposed to treat the visible spectral reflectance for pigment identification. For instance, Dupuis et al. applied the wavelength corresponding to the peak value of the visible spectral reflectance to identify the pigments used in “Worship of the Child” [26]; Wang et al. applied the derivative of visible spectral reflectance to identify the pigments used in “Dazu Stone Carvings” and “Hanyang Mausoleum Lacquer Bowl” [27]. However, most of the pigments used in ancient wall paintings are mineral pigments. Painters have already grasped the rule that the tone of their paintings can be enriched by varying the particle size of the mineral pigments. So the treatment of visible spectral reflectance for pigment identification of ancient wall paintings requires taking the influence of particle size on the reflectance into account. In addition, though researchers have constructed the spectral reference database of commonly used pigments for identification [28, 29], these databases do not cover the visible spectral reflectance of the pigment with size variations. Thus, we will construct a new spectral reference database including the pigment size variations to qualitatively investigate the relationship between pigment size and its visible spectral reflectance and then give an identification method taking the size variations into account.
3.1. Spectral Reference Database
For the pigment identification of ancient wall paintings in Mogao Grottoes, 45 kinds of commonly used mineral pigments with different sizes are selected to construct the spectral reference database. The preparation process of the samples is in strict accordance with the procedure that makes the wall paintings of Mogao Grottoes. A total of 144 pigment samples are prepared employing gelatin solution as the binder. Here, the gelatin is obtained by boiling bones, skin, and tendons of animals such as donkey, cattle, and horse, which is the same with the old traditional technique for those grottoes.
The binder solution is prepared by dissolving 10 g gelatin into 90 g water: First, disperse 10 g gelatin into 40 g normal temperature water and stir it until gelatin cannot be further dissolved. Then, pour another 50 g water with temperature of 75°C into the solution. Continue stirring until all gelatin is dissolved.
Each sample includes four layers: substrate, ground layer, plaster layer, and pigment layer. An industrial board is selected as the substrate with the first layer peeled off to simulate the vegetable fiber in the ground layer of the ancient wall paintings. Fine sediment taken from the bed of the Dangquan River before the Mogao Grottoes is uniformly dispersed into the gelatin solution and then painted on the substrate to prepare the ground layer. Plaster layer is painted by plaster dispersed into the gelatin solution on the naturally dried ground layer. Finally, mineral pigments is uniformly dispersed into the gelatin solution and painted on the naturally dried plaster layer. In particular, 0.35 g pigment is dispersed into 175 ml binder solution. Part of the samples is shown in Figure 2.
After natural drying of the pigment layer, the visible spectral reflectance of each sample is measured by an X-rite SpectroEye. All the data are collected then to develop the spectral reference database for pigment identification of ancient wall paintings.
3.2. Spectral Data Analysis
Three commonly used pigments, Azurite, Malachite, and Hematite, with five particle size levels for each pigment are selected to investigate the influence of chemical composition and particle size on visible spectral reflectance. The visible spectral reflectance of the pigments are shown in Figure 3, in which 3#, 5#, 7#, 9#, and 11# are identifiers of traditional particle size degree for mineral pigments. A larger identifier represents a smaller size.
As shown in Figure 3, the same kind of pigments has the same visible spectral curve shape along wavelength, while different kinds of pigments have different visible spectral curve shapes along wavelength. It illustrates that the geometric shape of visible spectral reflectance mainly depends on the chemical composition of the pigments since chemical composition determines the selective absorption of pigments to wavelength. The particle size of pigments has little effect on visible spectral curve shape and mainly influences the amplitude of the visible spectral curve since it determines the back-scattering property of the pigments according to Mie theory. It is interesting that a smaller pigment size corresponds to a larger visible spectral amplitude, while pigment with a larger particle size has a relatively smaller amplitude.
3.3. Feature Extraction
Apparently, the curve shape variations of visible spectral reflectance can be utilized as the feature to identify the pigments used in ancient wall paintings. Therefore, the work in this part is mainly focused on removing the influence of pigment size on spectral amplitude and extracting the feature of spectral curve shape.
The influence of pigment size on spectral amplitude can be removed by normalization of visible spectral reflectance. The aim of normalization can be described as, with the prerequisite that geometric shape of visible spectral reflectance is maintained, making the normalized visible spectral reflectance of the pigment with the same chemical composition overlap each other. In this work, we first zero-center the spectral reflectance:
And then unitize it as
The geometric shape of visible spectral reflectance can be represented by the normalized spectral curve itself. It can also be represented by the slope and curvature of the normalized curve, namely, the first- and the second-order derivative of the curve:
The slope and curvature are also normalized using equations (3) and (4) to keep them dimensionless. Then, the curve itself, its slope, and curvature are weighted and combined to represent the feature of spectral curve shape for pigment identification. That is,where , , and are the weighting factors for the feature and can be used to adjust the contributions of three components to the feature. Because , , and are 31, , and vectors, respectively, the feature of equation (7) is an vector.
3.4. Feature Matching
The proximity of two points in the feature space can be measured by their Euclidean distance. A smaller Euclidean distance corresponds to a more similar chemical composition between the pigments. Thus, when we obtain the visible spectral reflectance of ancient wall paintings, we can map it into the feature space and search in the spectral reference database to match the pigment that has the nearest distance to the spectral reflectance from ancient wall paintings. Then, we regard the corresponding pigment as that used in ancient wall paintings. According to the principle, the process to implement pigment identification can be described as follows:(1)Normalize visible spectral reflectance according to equations (3) and (4)(2)Map the normalized spectral reflectance into feature space according to equations (5)–(7)(3)Compute the distances between the spectral reflectance in reference database and that from wall paintings(4)Find the pigment that has the nearest distance in the reference database
It is obvious that the identification result is mainly determined by the completeness of the spectral reference database. Therefore, it is necessary to increase the kinds of pigments for the reference database as much as possible.
4. Results and Discussion
Each pixel of the visible spectral image contains a spectral reflectance. So the pigment of the image can be directly identified pixel by pixel. However, this cannot guide the restoration of ancient wall paintings effectively since it is difficult to locate the position of a pixel in the paintings. Instead, because each superpixel includes about pixels, it occupies a relatively large area in the paintings. Moreover, superpixel can limit the boundaries for pigment filling. It is meaningful to the restoration of ancient wall paintings. Thus, in this work, we regard superpixel as a basic unit and implement pigment identification superpixel by superpixel. The average of all the visible spectral reflectances in each superpixel is used as the target to determine its pigment used in wall paintings. Because the normalized visible spectral reflectance of the pigment with different particle sizes in the reference database overlaps each other, we also use their average as the reference for pigment identification.
According to the implementation process of pigment identification, we set the weighting factors in equation (7) as and and then identify the pigment used in Figure 1 superpixel by superpixel. Some identification results are shown in Figure 4. It shows that the pigment used in red area A is hemitate; in dark gray area is slate gray; in blue area is lapis lazuli; and in green area is malachite.
4.1. Complexity Analysis
As mentioned above, the extended SILC is a variant of k-means clustering. It is well known that the time complexity of the classical k-means algorithm is O (MNKI), where MN is the number of pixels in the image, K is the number of clusters predefined, and I is the iteration number required for convergence. However, the time complexity of the extended SLIC is linear to the number of the pixels in the visible spectral images and only O (MN) since the number of its iterations is constant and its search process is limited in the neighbourhood which means that a pixel falls in the neighborhood of less than eight cluster centers and the distance computation is no more than eight times. As for the pigment identification method proposed, its time complexity is linear to the number of the samples in the spectral reference database.
5. Conclusions
Multispectral imaging technique endows ancient wall paintings with eternal virtual life. The state of a painting, however, is suffering from various irreversible damages and in urgent requirement of restoration. In order to guide the scientific restoration of ancient wall paintings, we provide the superpixel segmentation method and pigment identification method based on visible spectral image of ancient wall paintings captured by using the multispectral imaging system. We extend SLIC to the superpixel segmentation of the visible spectral image by redefining the feature of the visible spectral image. It can effectively extract the outline of wall paintings and then limit the pigment filling area in restoration of wall paintings. The pigment identification method can remove the influence of pigment size on visible spectral reflectance and guide the pigment selection in restoration of wall paintings. This work can be integrated with the multispectral imaging system to provide a digital preservation to the color of ancient wall paintings.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work was supported by the Key Scientific and Technological Project of Henan Province (no. 192102310028); the General Project for Humanities and Social Sciences Research of the Education Department of Henan Province (no. 2019-ZZJH-128); the Startup Foundation for Doctors of Henan University of Animal Husbandry and Economy; and the Science and Technology Innovation Team of Henan University of Animal Husbandry and Economy.