Abstract
Porous materials have become increasingly common in people’s daily lives as the industry has advanced. Porous materials have numerous applications in the petroleum and chemical industries, as well as in everyday life. The study of diffusion, thermal conductivity, and percolation properties of porous materials has an important engineering application background and scientific value. The microstructure of materials affects their properties and attributes, so the description and visualization of the microstructure of porous materials is of great importance in the study of materials science. Due to the specificity of the internal structure of porous materials, many scenarios require 3-dimensional reconstruction of porous materials in practical engineering. In order to improve the effect of 3-dimensional reconstruction of porous materials, a 3D reconstruction method based on the improved generative adversarial neural network (GAN) is proposed in this paper for SEM images of porous materials. First, scanning electron microscope (SEM) images of porous materials are acquired, and then the acquired SEM images are preprocessed, including denoising and determining the boundary. Second, an improved GAN-based image super-resolution reconstruction model (ISRGAN) is used, and then the preprocessed images are fed into the ISRGAN model for training. Thus, multiple intermediate layer images are generated. Third, the 3D reconstruction of the intermediate layer images is performed using the slice combination method. The relationship between the unit cell pixels and the porosity is analyzed in the experiments to verify the effectiveness of the 3D reconstruction method used in this paper, and it is concluded that the porosity tends to be stable when the unit cell pixels converge to 110 and converge to the porosity of the real sample. The experimental results validate the feasibility and effectiveness of the method presented in this paper in the 3D reconstruction process.
1. Introduction
Porous materials are commonly found all around us, and the material plays a significant role in structure, cushioning, vibration damping, insulation, sound dissipation, and filtration. High porosity solids have low rigidity and high density, so naturally porous solids are often used as structural bodies, such as wood and bone. Humans have also developed many functional uses for porous material use. The study of porous materials began with zeolites. Zeolite is an ore that was first discovered in 1756. Cronstedt, a Swedish mineralogist, discovered a class of natural silica-aluminate ores that boil when burned. Molecular sieves have a homogeneous microporous structure and have a preferential adsorption capacity for polar and saturated molecules. As a result, molecules with different degrees of polarity, saturation, molecular size, and boiling point can be separated. The relative pore content of porous materials is variable. According to the pore size, below 2 nm is called microporous, 2 nm–50 nm is mesoporous, and above 50 nm is called macroporous. It can also be divided into porous metal, porous ceramic, porous plastic, etc. based on the material. Also, based on the size of porosity, it can be divided into low and medium porosity materials and high porosity materials. The former is mostly closed type, and the latter will present three types, which are honeycomb material, open cell foam material, and closed cell foam material. Porous material is a composite composed of solid phase, and pores are formed through the solid phase, and the most distinctive feature that distinguishes it from ordinary dense solid materials is the presence of useful pores. The most basic parameters of porous materials are the indicators that directly characterize their pore properties, such as porosity, pore size, and specific surface area. In addition, the properties of porous materials also depend heavily on the pore morphology, pore size, and its distribution.
The microstructure of porous materials is very important for engineering analysis as well as practical engineering applications, and traditional two-dimensional images cannot meet the engineering needs. Therefore, 3D reconstruction techniques based on porous materials are widely studied. Three-dimensional reconstruction is the process of recovering three-dimensional space from a two-dimensional image using a primitive map, that is, studying the relationship between the three-dimensional coordinates of points, lines, and surfaces in three-dimensional space and the two-dimensional coordinates of corresponding points, lines, and surfaces in two-dimensional image in order to achieve quantitative analysis of the object’s size and the fold mutual position relation. Currently, the main imaging techniques are computed tomography (CT) technology [1, 2], SEM [3, 4], focused ion beam-scanning electron microscopy (FIB-SEM) [5, 6], and nuclear magnetic imaging (MRI) [7, 8]. Compared to the images produced by other imaging techniques, SEM images have several advantages: first, it has high resolution. The resolution of the SEM secondary electron image can reach 3 nm, the resolution of the ultrahigh resolution SEM secondary electron image is up to 1 nm, which is two orders of magnitude higher than the limiting resolution of optical microscopy of 200 nm. Second, the depth of field is large. At a magnification of 100×, the depth of field of an optical microscope is 1 μm, while a SEM can reach 1 mm. But even when the magnification reaches 40,000 times, the scanning electron microscope can still have a depth of field of about 1 μm. Third, the magnification is continuously adjustable in a wide range, and the image has sufficient signal brightness at high magnification. Fourth, the sample is relatively easy to produce. Fifth, dynamic observation is possible. SEM techniques are widely used for microscopic morphology and structure of solid samples, microregional elemental composition, line distribution, and surface distribution of samples, which are widely used in nanotechnology, materials, physics, chemistry, and environmental science [9, 10].
At present, three-dimensional reconstruction technology has been very mature. The current methods of 3D reconstruction modeling are mainly the following three: (1) direct model construction using traditional geometric modeling techniques, that is, the use of modeling software to construct 3D models [11, 12]; (2) dynamic reconstruction using 3D scanning equipment to scan real objects to directly get the information of object space points, and then reconstruct the model [13, 14]; (3) static reconstruction, based on digital pictures of 3D reconstruction, using images to recover the geometric shape of the object, that is, reconstructing the model using two or more images of the real object taken by cameras and digital cameras from various viewpoints [15, 16]. Reference [17] developed a system for picture-based 3D reconstruction. Reference [18] developed an enhanced 3D modeling system using algorithms such as an uncalibrated structure from motion and camera self-calibration. Reference [19] proposed a 3D reconstruction system for buildings. Reference [20] created a human–computer interactive reconstruction system that employs a series of panoramic views of an object, that is, n pictures of the object from all angles, and then processes these images to reconstruct its 3D solid. Reference [21] used a self-calibration method of the camera to reconstruct the building’s morphology. From these studies, it can be seen that 3D reconstruction techniques have been maturely applied in various industries. For 3D reconstruction of porous materials, reference [22] proposed an improved convolutional neural network for 3D reconstruction of porous materials for 3D reconstruction of porous materials. Reference [23] compares the performance of several algorithms for 3-dimensional reconstruction of porous materials, including the genetic algorithm, particle swarm optimization, differential evolution, firefly algorithm, artificial bee colony, and gravitational search algorithm. The experimental structure shows that the genetic algorithm gives the best results. Reference [24] proposed the ST-CGAN network based on the GAN network to extract effective information from 2D images to construct 3D effects. Reference [25] proposed a 3D reconstruction of CT images of rocks using GAN networks with good results. Since the SEM images of porous materials have the advantages of high resolution and the magnification of the images are continuously adjustable in a wide range, this paper focuses on the SEM images of porous materials and uses an improved GAN network for 3D reconstruction of porous materials. Similar techniques are widely used [26].
2. Related Knowledge
2.1. Introduction to Porous Materials
There are many porous materials, such as porous metals and porous ceramics. Compared with dense metal materials, porous metals have a lot of good characteristics, such as low thermal conductivity, high heat transfer and heat dissipation capacity, sound absorption and good sound insulation, excellent wave transmission, good electromagnetic wave absorption, fire resistance, and thermal shock resistance. Commonly used porous metal materials are bronze, nickel, titanium, aluminum, stainless steel, and other metals and alloys. Porous ceramic is a new type of ceramic material, and the manufacture of its use began in the late 1950s. Initially, it was used only as a bacterial filter material. Porous ceramic materials have good permeability, low density, high hardness, large surface area, and small thermal conductivity, so they are widely used in metallurgy, chemical industry, environmental protection, energy, biology, food, medicine, and other fields. The most widely used porous ceramic preparation process is organic foam immersion slurry replication technology, which produces a mesh ceramic which is a porous material consisting of ceramic network surrounded by connected voids.
A porous material is a collection of units of solid matter that are combined in some way. These units have pores between them, and these pores allow the circulation of substances such as gases and liquids. The most important characteristic of porous materials is that they have many pores compared to other materials with a tighter structure, and the size and shape of the pores vary from one porous material to another. The appearance of porous materials that people see with the naked eye is actually the appearance of all the units gathered together. Porous materials rely on solid material units to maintain their shape and properties, so the sum of interconnected units in porous materials is usually called the solid skeleton. From the above analysis, it is clear that porous material is a material form composed of solid skeleton and pore space together. The space between the units in a porous material is called the pore of the porous material. The shape of the units is very irregular and thus the pore shape is extremely complex; it is so complex that it is difficult to describe the geometry precisely by mathematical means. The pores are homogeneous at the macroscopic level and randomly distributed in the porous material at the microscopic level. Some pores are interconnected and they play a major role in the permeability of the medium. Some pores are isolated and closed by the solid skeleton and become dead-pores, while some pores are partially isolated and only one end is connected to other pores and are called dead-end-pores, which can store fluids but contribute little to the permeability of the medium. Some pores are located between units and are called intergranular pores in geoscience. Some pores are located inside the unit and are referred to as intragrain pores in geoscience. The larger scale pores in the medium are often in the form of fissures, cuttings, cracks, or cavities. Regardless of the type, pores are interwoven in porous materials to form a complex network, and it is not meaningful to study individual pores in isolation.
2.2. Generating Adversarial Neural Network (GAN)
GAN was proposed 8 years ago [27]. The principle of GAN is similar to copying a painting in people’s daily life. When copying more and more times, the more similar the painting becomes at the end. At the highest level, the copied painting is exactly the same as the copied painting. In the GAN network, the operation of drawing is replaced by model training. The GAN network is used to learn data distribution patterns and generate data that are similar to the original data distribution. The GAN network structure consists of two parts, a generator and a discriminator, and the GAN training process is depicted in Figure 1.
The GAN is trained alternately with the generator and discriminator during the training process. The red dashed line indicates the distribution of the real data, the blue solid line indicates the generator output distribution, and the purple solid line indicates the recognition curve of the discriminator. The horizontal line is a uniformly sampled region, the horizontal line x is a partial region of the real data, and the upward-facing arrow represents the mapping x = B(z). B shrinks in the high-density region and spreads in the low-density region. To train the GAN network, the discriminator is trained by first fixing the generator parameters. When the discriminator has been trained, adjust the parameters and train the generator. After the generator is trained, the discriminator is trained again. The generator and discriminator are trained alternately and finally reach the equilibrium state. The data distribution generated by the generator is consistent with the real data distribution in the equilibrium state. GAN’s mathematical expression is as follows:where is the expected value of the distribution function, y is the true data, O(y) is the true sample distribution, z is the noise input to the G-network, B(z) is the data generated by the G-network, N(z) is the noise distribution defined in the lower dimension, and A(y) is the probability of whether the data is true, with closer to 1 indicating more realistic data. A(B(z)) is the likelihood of the D-network determining whether the data generated by the G-network is correct. The generating network wants to generate a more realistic picture, that is, the larger the A(B(z)), the better, when V(A,B) is smaller. For the discriminative network, the stronger the discriminative ability is, that is, the larger the A(y) is, the larger the V(A,B) will be at this time. Fix B first and find the optimal solution A:
The optimal A is obtained by bringing the optimal A to :where K represents the Kullback–Leibler divergence and J represents the Jensen–Shannon divergence. denotes the difference between the two distributions with a minimum value of −2log2 and a maximum value of 0. B is optimal when .
2.3. SRGAN Model
When the magnification of the image is relatively low, the image super-resolution reconstruction algorithm incorporating traditional deep learning convolutional neural networks is advantageous. When the magnification of the image is more than four times the size of the image itself, the image processed by this technique will be blurred or lack of image details. The reason for this is that this algorithm mostly uses two techniques, interpolation convolution or interpolation padding convolution. On the other hand, since traditional neural networks usually use mean squared difference MSE as the loss function during model training, a high peak signal-to-noise ratio can be achieved, but the disadvantage is that the images generated using this technique tend to lose high-frequency details, causing the images to become smooth, thus affecting one’s visual experience. In view of this, an image super-resolution reconstruction model based GAN (SRGAN) based on generative adversarial networks is introduced in the literature [28]. In the model training process, the new method employs a perceptual loss function, and then achieves image super-resolution through upsampling and convolution. The experimental results show that the recovered image with the SRGAN model has more high-frequency details, ensuring the image’s quality. Figure 2 depicts the SRGAN model.
As shown in the SRGAN model in Figure 2, the model mainly consists of several core components, including the generator, discriminator, VGG network, and loss function. It has been shown that the amount of feature information extracted by the network is closely related to the depth of the network structure. In other words, a deeper network structure has to be constructed in order to extract more feature information. However, the cost of doing so is that the problem of gradient dispersion or gradient explosion will occur. The use of residual blocks, on the other hand, can solve the problem and thus ensure the effective transfer of gradient information. In view of this, first, we further develop the optimization of the generator based on ResNet. It has several residual blocks, and each residual block has two 3 × 3 convolutional layers. Then, the batch normalization (BN) layer is located behind the convolution layer, which utilizes Relu as the activation function, while the two 2× subpixel convolutional layer are used to expand the feature information inch. Finally, the output 3-channel image is generated by using a 3 × 3 convolution layer. In this way, with this model, super-resolution images can be obtained even with the input low-resolution images. Second, the discriminator contains eight convolutional layers, and any one of them is connected with a batch normalization layer BN. Along with the deeper and deeper layers of the network, the number of features becomes more and more, and the size of the features keeps getting smaller, so LeakyRelu is utilized as the activation function. Finally, the batch normalization layer BN is followed by two fully connected layers, with a sigmoid activation function that is used to determine the likelihood that the judgment is a natural image. The role of the VGG network is that it outputs the super-resolution image SR generated at the end of the generator to the already trained network on ImageNet. The loss function of the SRGAN model refers to the loss function D_Loss for training the discriminator and the loss function G_Loss for training the generator, respectively, where the latter includes content loss and adversarial loss.
3. 3D Reconstruction Method for SEM Images of Porous Materials
3.1. Execution Flow of the 3D Reconstruction Method
The execution flow of this paper with the 3D reconstruction method is shown in Figure 3. First, the SEM images of the porous material are acquired. It is important to note that the acquired SEM images show the entire porous material as much as possible and are continuous. Second, the initial SEM images are often noisy and have unclear boundaries. Therefore, the initial SEM images need to be preprocessed. The common means of preprocessing is binarization or filtering techniques. Third, for 3D reconstruction, the intermediate layer images are very important. In this paper, an improved SRGAN model (ISRGAN) is used to generate the middle layer image. The middle layer image is characterized by a high similarity to the actual acquired SEM image. It is reflected by its void feature distribution which is the same as the acquired SEM image. Fourth, 3D reconstruction is performed using the generated intermediate layer images. The reconstruction is done by using layer-by-layer extraction and stacking in this way.
3.2. ISRGAN Model
The SRGAN model is characterized by a very large number of residual convolution layers. This characteristic, although the final result obtained is better, is very time-consuming to train and requires high performance of the hardware. The authors of [29] proposed a convolution-deconvolution network structure that can effectively approach the image super-resolution reconstruction problem. The convolution-deconvolution network structure is shown in Figure 4.
The network structure shown in Figure 4 is completely symmetric. The figure shows that the convolution layer is followed by downsampling and the deconvolution layer is followed by upsampling. The convolution operation is multiple inputs and one output. The deconvolution operation is the opposite, with one input and multiple outputs. The deconvolution is actually the process of expansion, so the operation can get more comprehensive information and thus obtain more detailed information of the image. In this paper, the ISRGAN model is obtained by introducing the convolution-deconvolution structure on the basis of the SRGAN model. The structure of the model is shown in Figure 5.
The generator of the ISRGAN model contains seven convolutional and seven deconvolutional layers each. The parameters of the generator are set as shown in Table 1.
In addition to the improved generator of the model, the discriminator is also improved in this paper. The discriminator is mainly used to determine the generated forged image and the real image to calculate the minimum variance of the pixel space between them. In order to recover better image information, this paper assists in extracting image features by training the VGG19 network, and its structure is given in Figure 6.
The feature map of a layer is extracted from a previously trained VGG19, and the generated fake image FEATURE MAP is compared to the real image MAP to determine the Euclidean distance between the generated image and the real image feature representations. LM calculates the degree of matching between pixels and the loss function and expression of the degree of matching of a certain feature layer is as follows:where Li,j, Di,j is the feature map size, is the feature map obtained by the jth convolution before the i-th maximum pooling layer, and MT is the real image. Equation (5) shows the discriminator’s loss function in the GAN:
When y is real data, the result is the output of the real data minus the average of the output of all the fake data, and then the sigmoid is taken over the result. When y is fake data, the result is the output of the fake data minus the average of the output of all the real data, and then the sigmoid is taken over the result.
The loss functions of the SRGAN discriminator and generator are modified as shown in equations (6) and (7), respectively:
Finally, the generator’s loss function is depicted as follows:where is the balancing factor, whose role is to balance the loss values of the loss function.
3.3. 3D Model Reconstruction
According to the 3D model construction process given in this paper, once the ISRGAN model generates all the intermediate layer images, then the intermediate layer images are overlapped from the top to the bottom according to the stacking method, as shown in Figure 7.
The 3D model reconstruction method used here is the slice-combination method. The operation schematic of the slice-combination method is shown in Figure 8.
It should be noted here that the boundaries of 3D models constructed by overlapping multiple intermediate layer images are usually rough and can be seen to overlap multiple layers; therefore, smoothing is essential. Commonly used smoothing techniques include weight smoothing, mean filtering, median filtering, Gaussian filtering, and bilateral filtering. Mean filtering is highly efficient and simple in thought. Therefore, in this paper, the mean filter is chosen to smooth the boundary.
4. Experimental Results and Analysis
Image size affects the performance of the training model. An image that is too large will consume memory, while the one that is too small will not be able to characterize the internal information. To quantify the size of the dataset used in this paper, the relationship between different image sizes and the running time of the algorithm was experimentally compared. The settings of each parameter of the model in this paper are shown in Table 2.
In this paper, SEM scanned images of coal-based porous carbon are selected as the input data for 3D reconstruction. The 3D reconstruction algorithm’s performance is measured by whether the porosity approximates the real porosity of the sample. Different cell sizes have different effects on porosity. Based on the Avizo software, the cell sizes on the whole 3D model are extracted and the cell porosity is calculated. Experiments were conducted on three samples with different cell edge lengths, and the porosity comparison for each sample was obtained by calculation as shown in Table 3 as well as Figure 9.
In the porosity variation curves of the three samples shown in Figure 9, all three samples have a large variation in porosity until the cell size is 100. Samples 1 and 3 have a stable porosity region at a unit size of 100, and sample 2 has a stable porosity region at a unit size of 110. This indicates that the optimal unit cell size is different for different coal-based porous carbon samples. In this paper, a 180 × 180 × 180 pixel unit cell was finally used as the study object. The experimental results show that the 3D reconstruction model used in this paper is capable of obtaining the closest porosity to the real object. The reason for the good results of 3-dimensional reconstruction based on SEM images of porous materials in this paper is the perfect simulation generation of the intermediate layer images of the 3D model using the ISRGAN model. The experimental results show that the generation of the middle layer image is effective, and the information of the void part in the middle layer image can be generated as much as possible and close to the real image. The experimental results show that the ISRGAN model used in this paper is very effective and superior. The method can also be applied to other porous materials for 3D reconstruction in the future.
5. Conclusion
The pore structure of porous materials is very complex, and its representation has been a difficult problem in this research area. Adequate analysis is needed before modeling to derive a suitable stochastic model expression; otherwise, the reconstructed model will not truly reflect the geometric characteristics of porous materials. In this paper, we have conducted an in-depth study on the selection of scanning images, preprocessing, 3D mathematical modeling, and model validation in the modeling process and also implemented the corresponding algorithms mainly in the following aspects. First, by analyzing and comparing the advantages and disadvantages of various scanning images, we finally determined that using SEM images is the most beneficial for 3D reconstruction. The reason is that SEM images have higher resolution, large depth of field, continuously adjustable magnification in a wide range, relatively easy specimen preparation, and dynamic observation. The SEM images of porous materials were acquired, and the acquired SEM images are preprocessed, including noise removal and determination of boundaries. Second, for the generation process of the intermediate layer images, an improved deep learning model for image super-resolution reconstruction is used in this paper. This model is an improvement of the SRGAN model, which has 16 layers of residual convolution in the generator, and although the deeper network can extract more details of the characteristics, the large network structure is not conducive to training, the requirements for hardware and other experimental environments are relatively high, and the stability of the training process is difficult to be guaranteed. The ISRGAN model, on the other hand, circumvents these problems. The preprocessed images are fed into the ISRGAN model for training. Thus, multiple intermediate layer images are generated. Third, the 3D reconstruction of the middle layer images is performed using the slice combination method. In order to validate the efficacy of the 3D reconstruction method used in this paper, the experiments examine the relationship between unit cell pixels and porosity. It is concluded that the porosity tends to be stable when the unit cell pixels is over 110, and it is converged to the real porosity of the sample. The experimental results validate the feasibility and efficacy of the method presented in this paper in the process of 3D reconstruction. However, the samples reconstructed in 3D based on image processing techniques will have information loss and addition, and how to extract the desired information features without loss is the direction of our next stage of research.
Data Availability
The labeled dataset used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no competing interests.
Acknowledgments
This work was supported by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China (20KJB410005) and the Scientific Research Foundation of Nanjing Vocational University of Industry Technology (YK18-03-03).