Abstract
Current development in a deep neural network (DNN) has given an opportunity to a novel framework for the reconstruction of a holographic image and a phase recovery method with real-time performance. There are many deep learning-based techniques that have been proposed for the holographic image reconstruction, but these deep learning-based methods can still lack in performance, time complexity, accuracy, and real-time performance. Due to iterative calculation, the generation of a CGH requires a long computation time. A novel deep generative adversarial network holography (GAN-Holo) framework is proposed for hologram reconstruction. This novel framework consists of two phases. In phase one, we used the Fresnel-based method to make the dataset. In the second phase, we trained the raw input image and holographic label image data from phase one acquired images. Our method has the capability of the noniterative process of computer-generated holograms (CGHs). The experimental results have demonstrated that the proposed method outperforms the existing methods.
1. Introduction
As holographic display can precisely retrieve the wavefront details of a three-dimensional entity, it emerges as a potential future display technology. Using a spatial light modulator (SLM), the holographic display typically shows the digitally produced interference pattern called the computer-generated hologram (CGH) to resolve electronically the wavefront of the coherent light source. Computer-generated holography (CGH) is a methodology for calculating the holographic interference pattern that produces a random optical field by using digital computers. There are various iterative methods that have been used for generating CGH, including the Fresnel method [1] and the Gerchberg–Saxton (GS) method [2], the optimal-rotation-angle method [3], iterative finite-element mesh adaption [4], quantized kinogram by genetic algorithm [5], and vectorial beam shaping [6]. Computer-generated holography has been widely used in many areas, including three-dimensional augmentation reality [7], holographic projection [8], holographic dynamic display [9], virtual and optical information security [10], and optical tweezers [11]. However, iterative methods have some issues and one of the major issues concerning the generation of CGH is the heavy computation power required; it is time-consuming and uses a lot of computer processing power, so we need to save computation time and cost, as the computational cost is high [12].
In recent years, deep learning plays a key role in the technological revolution and is widely used in industrial and scientific research fields. So, deep learning is becoming a hot topic in the field of optics, as this method is faster to compute and could provide a new framework in holographic image reconstruction. There are several deep neural network-based projects that have been proposed and applied in optics fields. Generative adversarial network (GAN) [13] is a type of neural network architecture for generative modeling and has grown widespread attention. So, GAN is constructed of two separate neural network models: one is called the generator or generative network model and the second is called the discriminator or discriminative network model. Both models need to be trained, so the generator model learns to generate samples, while the discriminator model learns to discriminate or differentiate generated samples.
In this paper, we proposed a generative adversarial network (GAN) based on GAN-Holo, a novel generative adversarial network-based generative holography using deep learning. Since we need to generate hologram from the image, so by taking advantage of the deep neural network, a noniterative method is introduced based on a generative adversarial network (GAN), where our model GAN-Holo is trained by using a series of MNIST images as input and hologram images as ground truth. However, in our proposed method, the training datasets were generated by the traditional iterative method, that is, the Fresnel method as shown in Figure 1. After training GAN-Holo, the MNIST image can be used as the input data for the GAN-Holo model, and the hologram image can be predicted in the GAN-Holo output. To obtain optimal results, experiments with various hyperparameter settings are carried out.
The contribution of this paper is that the GAN-Holo model generates a reasonable hologram image quality with image resolution 128 x 128 of the reproduced intensity pattern and faster as compared to iterative CGH methods with the use of less computational power which is verified by experiments and simulations.
The rest of the article is organized as follows: Section 2 analyzes previous related work. Section 3 explains the proposed method in the form of Phase 1 (Principle of Fresnel) and Phase 2 (GAN-Holo). Section 4 discusses the experimental evaluation of GAN-Holo and presents the results. Section 5 concludes the paper.
2. Literature Review
In the research field of optics and computer vision, deep learning models were used several years ago. The authors in [14] presented a novel deep learning approach to generate a hologram. In this deep learning approach, the author proposed a noniterative method using ResNet to generated the holograms and then compared it with the iterative method (Gerchberg–Saxton). In [15], to the best of our knowledge, the authors presented a method to employ deep neural networks to reduce losses in a JPEG compressed hologram, as the reconstructed image of JPEG compressed hologram suffers from extreme quality loss because, during the compression process, certain high-frequency features in the hologram will be lost. The proposed framework in [16] is called deep learning invariant hologram classification (DL-IHC) and the authors introduced a self-attention to convolution neural network (CNN) to implement a classification hologram of deformable objects. The authors in [17] presented a method of generating a multidepth hologram and the strategy of its training by using a deep neural network. The author trained the deep neural network in such a way that, as an input, it takes images of different depths and, as an output, it calculates the complex hologram. To train the network with high performance and accuracy, the proposed network and the dataset compositing were structured with a different technique. Rather than a simple image and hologram, the dataset consists of multiple input images and their holograms. So, the strategy of the proposed dataset enhances the accuracy of reconstructed images by the network called deep learning holograms (DLHs).
In [18], the authors proposed a noniterative algorithm based on a convolutional neural network with unsupervised learning called DeepCGH that computes to enhance holograms. In [19], a holographic vision system (HVS) was proposed based on deep learning. The proposed method HVS is based on a hologram classification to identify holograms and evaluated with the dataset of handwritten numerals. In [20], the authors proposed a method that generates a binary hologram based on deep learning. This novel deep learning approach generates binary hologram in the noniteration method. The generated binary holograms by neural network compared with the previous methods and the results were faster than previous work with enhanced quality. In [21], the authors presented computer-generated holography (CGH) which is based on deep learning. The proposed method generates holograms quite fast with high quality, so the performance of the proposed technique is outperform because the baseline methods utilize more computation power and time. In [22], the authors proposed deep learning-based digital holographic microscopy (DHM). The network was trained with thousands of blurred microparticle holograms flowing in a microtube. The convolution neural network (CNN) estimates the depth of the microparticles, Segnet and Hough transform used for the detection of in-plane of microparticles. After training, the performance of the proposed method to track the 3D motion of microparticles was faster than the iterative method.
The proposed framework in [23] for denoising digital holograms based on deep learning is called the spectral convolution neural network (SCNN). The proposed technique uses just a single hologram to manage different speckle-noise levels effectively instead of using traditional filters to remove the speckle noise. This novel technique is faster than conventional approaches without losing output performance. In [24], the authors introduced a novel technique for the reconstruction of an object from a single shot lensless measurement. The proposed approach presents a technique for mutually developing convolutional neural networks and coded aperture to reconstruct an object. This novel method includes a deep learning system in which the coded aperture is put as a first convolution layer is linked to the coded aperture and the reconstruction network. In [25], the authors proposed a novel algorithm denosing fringe patterns. The proposed algorithm recovers high-quality fringe patterns compared with other denosing algorithms. In [26], the authors presented a deep learning-based method for denoising digital holographic speckle pattern interferometer (DHSPI) wrapped phase. The method proposed is very effective to extract the required information and reduce speckle noise. The application of the proposed method is to detect defects in topography maps.
The authors in [27] presented a novel approach based on Super-Resolution Generative Adversarial Network (SRGAN) called Enhanced Super-Resolution Generative Adversarial Network (ESRGAN), which is capable of generating an image with better quality. In this approach, the authors thoroughly studied how to improve the SRGAN and their losses. Moreover, by introducing Residual in Residual Dense block and to improve the perceptual loss made the ESRGAN achieves better quality with realistic results than SRGAN. In [28], the author proposed an adversarial learning method for robust satellite image superresolution reconstruction based on GAN called Edge-Enhancement Generative Adversarial Network (EEGAN). The presented method consists of two networks: the first network extracts features and gets an intermediate high-resolution image and the second network enhances the edge of the image then to combine the intermediate image and the enhanced image gives a result with a clear image.
3. Proposed Method
In this section, the proposed model GAN-Holo is presented. GAN-Holo consists of two phases as shown in Figure 2: phase 1 and phase 2. Phase 1 consists of the principle of the Fresnel method and phase 2 consists of the GAN-Holo method. A detailed explanation is given below.
3.1. Principle of the Fresnel Method
As we use the traditional Fresnel zone method to create the dataset for training our proposed model GAN-Holo, in this section, we will discuss the Fresnel method. As shown in Figure 1, such holograms are classified as Fresnel zone plates.
To produce the Fresnel zone, it is important to consider the distance between each pixel of the Fresnel zone plate and the point of source. We can deduce the phase of the wavefront from this distance:where is the wavefront and is the wavelength of the illuminating light. is a random point on the Fresnel zone plate which is generated by the random object point . Then, we take the cosine of two-phase differences to generate a hologram later.
3.2. GAN-Holo
GAN has already been applied in many applications such as videos, images, labels, and text, as well as in several other nonapplication specific processes. In this paper, we proposed GAN-Holo architecture to generate hologram images, which is based on a generative adversarial network (GAN).
GAN-Holo architecture for generating hologram can be defined as follows: GAN-Holo is a deep convolutional neural network architecture, which consists of two networks including generator (G) and discriminator (D) as shown in Figure 3(a), competing against each other. The role of G is to create data in such a way so that it can fool the discriminator, and D is the one that discriminates between two classes of real and fake images. GAN-Holo structure that is going to be implemented for generating hologram can be defined as follows: as there are two types of deep neural networks, G and D, G maps random Gaussian noise z under the condition of observed image x to the hologram h:
(a)
(b)
(c)
In training the generator network, we aim to fool the discriminator, so that the discriminator cannot be able to decide whether the generated hologram image is real or fake:where G is trying to force D to accept the generated hologram images as real outputs. But at the same time, D is trained to classify fake hologram images from the real ones:
In the above equation, the first part represents the training of the discriminator that is to train the discriminator network with real input images and real label images (holograms), while the second part is to train the generator network to create fake hologram images.
GAN consists of two “adversarial” networks: the one generative network G that generates the hologram images and the second discriminator network D that estimates the probability of the generated hologram image coming from G or training data. Both G and D are trained at the same time: we refine parameters for D to maximize and refine parameters for G to minimize , as these two are players for the max–min game with the value function V (D, G) and the objective function of this max–min game would be as
In Figure 4, the black curve (pdata) represents real data points, the green curve (pg) represents distribution generated by GAN-Holo, and the blue curve (D (x)) is the confidence of the discriminator. In Figure 4(a), the pg is almost similar to pdata and D is not good to discriminate. D is trained enough to discriminate pdata from pg in Figure 4(b). After training G, G (z) is guided by the gradient of D to change those regions which are the same as the original data. In Figure 4(d), after a few more steps of training pg = pdata, at this point, both G and D cannot improve more and the discriminator cannot differentiate between the real data and the generated data by G.
The network architecture of GAN-Holo is illustrated in Figure 3(a). GAN-Holo consists of a generator (Figure 3(b)) and a discriminator (Figure 3(c)). The generator consists of a contracting path (upper side) and an expensive path (downside). The generator consists of a total of 22 convolution layers. The contracting path follows the architecture of a convolution network. The contracting part consists of repeated two 3 x 3 convolutions, each followed by ReLU (rectified linear unit), batch normalization, and 2 x 2 max pooling for downsampling. After each max pooling, the number of feature maps gets double, as shown in Figure 3(b), the generator is starting with 64 feature maps and then 128, and so on. Upsampling of feature maps happens in the expansive path that halves the number of feature channels followed by a 2 x 2 convolution, which is called upconvolution. From contracting path, a concatenation occurs with the corresponding copy and crop feature map (skip connection) and two 2 x 2 convolutions. Each convolution is followed by ReLU (rectified linear unit) and batch normalization. The copy or crop feature map is necessary to prevent the loss in every convolution. The final layer consists of 1x1 convolution which is used to map feature vector to the desired number of classes. The discriminator in Figure 3(c) is a standard convolution neural network architecture with 8 convolution layers.
4. Experimental Demonstration
In this section, we present the dataset used in the process of our training and testing setup. The real ground-truth holograms were generated by the Fresnel method using MATLAB. The training datasets are composed of images “x” which were a handwritten digit dataset and hologram “h” and the number of pairs was 64,000, which were split into 80% for training and 20% for validation. All input 2D images and output hologram images were scaled to 128 × 128 resolution. The pixel counts for a small percentage of the dataset for testing setup were conducted after the training process. For optimizing the network, a learning algorithm called “Adam” was used with an initial learning rate of 0.0001, batch size 8, and the number of epochs was 100. The code for GAN-Holo was implemented with PyTorch v1.2.0 library and Python 3.5 on Ubuntu operating system and the code for the dataset was implemented in MATLAB.
To display the hologram on a system shown in Figure 5, we use He–Ne Laser with the power of 15 mW. The SLM is phase-only liquid-crystal-on-silicon (LCoS) SLM (Holoeye Pluto VIS) with 1920 × 1080 pixels and an 8 μm pixel pitch. The refresh rate is 60 Hz. The SLM is capable of working in phase-only modulation mode with a modulation depth of more than 2π for visible light.
To evaluate the performance of GAN-Holo for generating hologram, we used data for testing the trained GAN-Holo that is not included in the training data. In Figure 6, there are four sets of images with a different handwritten digit predicted by the GAN-Holo. The four target intensity patters in Figure 6(a) consist of three digits; those were randomly chosen from MNIST handwritten digit dataset, the ground truth of the target intensity pattern of the hologram in Figure 6(b), the reconstructed images of hologram in Figure 6(c), the hologram of the GS method in Figure 6(d), and their reconstructed images of hologram in Figure 6(e), the hologram images were predicted by the training network GAN-Holo in Figure 6(f) and the reconstructed images of hologram in Figure 6(g). The root mean square error (RMSE) between the reconstructed images of the hologram of target intensity patterns in Figure 6(c) and the reconstructed image of the proposed method results in Figure 6(e) is 0.07. The RMSE between Figures 6(c) and 6(g) is 0.09.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
For generating a single hologram, when using a computer with IntelCoreTM i7-7820X [email protected]×; 13, the GeForce GTX 1080 (NVIDIA) and memory 16 GB, the computational time based on GAN-Holo and the Fresnel method was 5 ms, 16 ms, and 94 ms, respectively. Therefore, the GAN-Holo method compared to the conventional iterative methods calculates holograms with a shorter computational time. The time took for training GAN-Holo was 20 h.
5. Conclusions
In this paper, we proposed a GAN-Holo method based on a deep convolution neural network for the generation of CGHs, which is a noniterative method. After comparing our results with the traditional iterative method, that is, the Fresnel method, the demonstration showed that our proposed method compared with the iterative method has reasonable results with 3 times shorter computation time. Furthermore, the proposed method, that is, GAN-Holo for calculating the CGH, which is based on the generative adversarial network has stable performance and there is no need to adjust parameters.
Data Availability
No data were used to support this study. We have conducted the simulations to evaluate the performance of the proposed technique. However, any query about the research conducted in this paper is highly appreciated and can be asked from the principal author Aamir Khan upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest.
Acknowledgments
Aamir Khan personally thanks the Department of Communication and Information Engineering, Shanghai University, Shanghai, China, that extended their full support by providing all key resources during the implementation and all afterward phases of this project. He would also specially thank Prof. Zhang Zhijiang and Yingjie Yu because of their continuous encouragement and massive support academically, financially, and socially during this project. Last but not the least, he is thankful to Dr. Muhammad Amir Khan for help and guidance in writing each and every section of the research paper. This work was financially supported by the National Natural Science Foundation of China (Grant no. 51775326).