Abstract
In recent years, some indoor landscapes have the common problems of stacking elements, improvisation, and lack of artistic beauty; how to perfectly integrate the landscape design elements into the indoor environment space is particularly important. High dynamic range images (HDRI) are more and more widely used in image processing systems; based on SoC FPGA design, the author implements a 20-bit high-dynamic real-time video image processing system, so that according to the characteristics of the indoor environment, extracting landscape design elements, the author can identify categories of graphic design elements to apply in interior spaces. According to the design requirements of ecological, modern, and sustainable development of interior landscape, based on Zynq SoC FPGA chip, the author realizes the software design of high dynamic real-time processing system and the programmable logic design based on Verilog; the realization of module functions such as image processing module register configuration, image receiving and processing, and image data output is realized. Simulation results show the author’s objective quality score is above 0.98, which is 118.95% of the average of other classical algorithms, which verifies the correctness and superiority of the author’s algorithm and system.
1. Introduction
Since the reform and opening up, with the prosperity and development of the economy, the process of urbanization has continued to advance, housing constructions are everywhere, and people’s living standards are getting better and better. However, people’s daily life is full of many unhealthy factors: air pollution, water pollution, noise pollution, etc., are threatening human life and health all the time; the ecological environment has suffered unprecedented damage. For their own health and home environment, the application of green environmental protection and ecological design in indoor and outdoor spaces is imminent.
Indoor space is the place where people live the most in their lives, and it is also one of the most important places. Whether people work or live, they spend most of their time in indoor spaces; the importance of interior space to us cannot be overstated. The comfort of an indoor space directly affects our mental and physical health. How to make indoor space healthier and comfortable, suitable for people to live and work, it is particularly important to introduce landscape design elements into the interior. The country vigorously advocates ecological protection, and people are eager to get close to nature; many conditions and phenomena show that, in today’s society; it has become a trend and trend to introduce landscape into indoor space; it is not only pleasing to the eyes but also extremely beneficial to people’s physical and mental health. However, due to many factors, the application of landscape design elements in interior design is not very perfect; it is more about the stacking of elements and the single use of a certain element for interior decoration purposes.
At present, the acquisition, transmission, and processing systems of high dynamic images are relatively rare, and the existing chips have obvious limitations, unable to meet high dynamic image processing requirements; the existing classical dynamic image processing technology is shown in Figure 1. In special lighting scenarios, or in application scenarios with specific requirements such as automatic driving image acquisition and interior landscape design, the dynamic range of the scene that often needs to be processed is huge; the images collected by ordinary video image processing systems in these application scenarios are not good enough, the obtained image is either overexposed, causing the loss of details in the bright parts of the scene, or it is too dark and the image cannot be seen clearly; this requires a specific image acquisition, transmission, and processing system to meet the needs of these applications, so the high dynamic image processing system has a good application prospect and market value.
2. Literature Review
In the 21st century, with the improvement of people’s awareness and the continuous improvement of science and technology, the disciplinary vision and research scope of interior design are broader. In the pursuit of quality of life, people have higher and higher requirements for the design sense and cultural level of indoor space; designers pay more attention to the quality of human settlements, environmental ecology, artistic style, and regional characteristics [1]. As people yearn for tranquility and closeness to nature, it is particularly important to introduce landscape design elements into interior spaces. However, the improvement of requirements has not been accompanied by the improvement of quality, and some interior designs cannot be coordinated with the overall space, make the landscape design elements appear blunt and discordant, or blindly copy the design language and landscape symbols of other countries; there is no connection with the local culture and the site environment, making the whole interior design seem weird and abrupt [2].
The development of design affects the process and continuation of human civilization; as early as the last century, the United Nations held a theme conference on “designing for a sustainable future,” the meeting put forward a number of policies conducive to sustainable development [3]. Today, under the leadership of the concept of sustainable development and the concept of transforming from industrial social civilization to ecological social civilization, it has become the theoretical basis and guiding ideology for the development and decision-making of the entire design industry. At present, the high dynamic image processing technology mainly focuses on two aspects: high dynamic image synthesis and high dynamic image enhancement processing. High dynamic image synthesis technology is aimed at obtaining a high dynamic image through algorithmic synthesis. LIU and Yang [4] solve the camera’s response function according to the multiexposure image sequence of the same scene and use this function and the image sequence to synthesize high dynamic images. Law [5] performs Laplacian pyramid decomposition on the image, using the local contrast, brightness, and saturation of the image to weight and fuse the layers of the pyramid; a high dynamic image is obtained. Jiang et al. [6] utilize multiexposure images to synthesize high dynamic images through a convolutional neural network approach. Rashwan et al. [7] obtained the number of strong exposure frames and weak exposure frames in an imaging cycle; the high dynamic images are obtained by means of time delay integration, respectively.
High dynamic image enhancement processing technology, namely, tone mapping algorithm, is designed to compress the dynamic range of high dynamic image, so that it can be displayed by ordinary imaging equipment. Guo et al. [8] believed that the human eye is only sensitive to the relative light and dark changes of the scene; therefore, the brightness of a fovea can be counted only according to the brightness level of the focus point of the fovea, and then the logarithm of these brightnesses can be taken to perform histogram equalization, and the resulting image can be obtained. Allahyar and Kazemi [9] perform different gray-level mappings according to the pixels of different brightness areas; this method can preserve the details of the bright and dark areas of the scene, but it will lead to blurred images. Zhang and Zhou [10] simulate the process of image deepening and lightening based on traditional photography knowledge and can achieve better image display effect, but this method requires too much computation. Suchocka et al. [11] used bilateral filters to layer the image and compress the dynamic range of the basic layer that reflects the overall brightness of the image, the detail layer that reflects the details of the image is retained, and then the compressed base layer and the detail layer are recombined to obtain the processing result; the image detail is good, but it causes halos to appear at the edges of the image. Feng et al. [12] proposed to generate a multiexposure sequence from the left viewpoint; it is combined with the original left-view underexposed image to form a multiexposure sequence, and a generative adversarial network is used to achieve high dynamic range image fusion.
Many scholars have carried out research on high dynamic image hardware processing system. Ying and Tian [13] designed a high dynamic image real-time synthesis system; a 2D lookup table structure is designed based on FPGA and Block RAM resources, which can realize 1080 p high dynamic image synthesis at 60 fps. Allahyar and Kazemi implemented a high dynamic image processing pipeline, using GPU to directly synthesize multiple ordinary images into a high dynamic image; then, the tone mapping algorithm is implemented on the GPU, which greatly improves the image frame rate compared to the CPU platform. Kareem and Hameed [14] implemented a global combined local high dynamic image display algorithm on FPGA; the system inputs a grayscale image with a size of 1024x768, which is displayed in real time on a standard monitor after high dynamic image processing. You and Li [15] implemented a high dynamic CMOS camera based on FPGA, which can collect high dynamic images with a dynamic range of 96 dB and output images of pixels at 24 fps. Pratama [16] performed hardware acceleration on the maximum image entropy high dynamic image compression algorithm and implemented a 12-bit depth high dynamic image dynamic range compression system using FPGA. Theoretical research and engineering application research of high dynamic images are developing and progressing, but the following problems still exist: (1)The main research purpose of the high dynamic image enhancement processing algorithm is to preserve the details of the image processed by the algorithm as much as possible; the real-time performance of the algorithm is not the focus of the design, so its calculation is usually complicated and difficult to apply in hardware real-time systems(2)The existing image processing systems with a high degree of integration can process images with a low depth; it is impossible to collect and process high dynamic images with more than 20 bits, so the application scenarios that require higher dynamic range of images and existing systems are difficult to meet the requirements
The author designs and implements a 20-bit high-dynamic real-time video image processing system based on SoC FPGA, so that according to the characteristics of the indoor environment, extracting landscape design elements, the author can identify categories of graphic design elements to apply in interior spaces. Design requirements for ecological, modern, and sustainable development of interior landscapes. Based on Zynq SoC FPGA chip, it realizes high dynamic real-time, software design of processing system and programmable logic design based on Verilog; the realization of module functions such as image processing module register configuration, image receiving and processing, and image data output is realized. Figure 2 shows the general design flow chart of the video image processing system implemented by the author.
3. Research Methods
3.1. High Dynamic Image Processing System Architecture
The author’s research goal is to design and implement a high dynamic image processing system, in order to better extract and integrate graphic design image elements into interior landscape design; the video processing system design module is shown in Figure 3.
The image acquisition part is mainly responsible for the acquisition and transmission of high dynamic images, and the images are acquired by the image sensor and then transmitted to the core processing chip of the system. The image processing part is mainly responsible for some necessary processing of the collected images, including image reception and high dynamic enhancement processing. The image output part mainly outputs the processed result image through the data interface, which can be directly displayed or output to the host computer, such as PC [17].
The tone mapping algorithm of high dynamic images is the basis for realizing the core functions of the system, so it is necessary to investigate the tone mapping algorithm; the author adopts some classical tone mapping algorithms to provide a theoretical basis for the design of high dynamic image processing algorithms.
3.1.1. Tone Mapping Algorithm Based on Photography
The author first proposes a dynamic range compression algorithm based on photography; the idea of the algorithm is to average the original images; for the brightness value, reconstruct the real brightness of the scene; then the dynamic range of the image is compressed by a global compression method; for an image with a particularly large dynamic range, it is necessary to perform local dynamic range compression processing on the image by simulating automatic exposure and shading. First, take the original average logarithmic brightness for the image, as shown in where is the luminance component of the input image.
According to the average logarithmic brightness of the scene and a brightness key value parameter, the real brightness value of the scene can be reconstructed, as shown in
The parameter controls the brightness level of the entire image, and the larger the parameter, the greater the average brightness of the image.
After reconstructing the image brightness, a global dynamic range compression algorithm is used to compress the image brightness to [1,0], as shown in where is the smallest luminance value that will be mapped to pure white.
For images with a particularly high dynamic range, the above methods are not very effective; inspired by photography, the author simulates the automatic exposure shading operation, finds a region for each pixel based on its contrast, and introduces the above-mentioned brightness key value to this area to control the average brightness of this area. When the above algorithm deals with images with low image depth, the processing effect is better and the complexity is lower, but when processing images with high image depth, the algorithm cannot obtain good results, and it is necessary to introduce the part of simulating automatic exposure and shading, and the complexity of the algorithm will be greatly increased [18].
3.1.2. Tone Mapping Algorithm based on a Hierarchical Model
The main idea of the layered model is to decompose an image into a base layer and a detail layer, as shown in where is the input image, is the detail layer of the input image, and is the base layer of the input image.
The base layer of the image shows the brightness of the image with a large dynamic range, and the detail layer is the detail texture of the image jumping and changing; according to this property, the base layer can be separated for dynamic range compression while retaining the detail layer [19], as shown in
is the result image, is the detail layer of the original image, and is the base layer after dynamic range compression.
The author adopts bilateral filter to divide the image into base layer and detail layer; its filter window and normalized weight are shown in where is the center window position, is the position of any pixel in the window, is the filter window, and is the Gaussian kernel function.
After converting the image brightness to logarithm, the above filter is used to decompose the image into a base layer and a detail layer, as shown ine where is the logarithmic brightness of the input image and BF is the bilateral filter. Detail layer is obtained by subtracting the base layer from the logarithmic luminance.
After the base layer is obtained, the dynamic range of the base layer can be compressed [20], as shown in
Among them, is the compression parameter, which controls the compression degree of the base layer.
After compression, the base layer and the detail layer are merged to restore the luminance component of the image, as shown in
3.1.3. Tone Mapping Algorithm Based on Gradient Compression
The author proposes an algorithm based on gradient compression; the main idea is that the part of the image with large brightness changes has a higher gradient; however, some details of the image have low gradients, so the parts with large gradients in high-dynamic images can be compressed, keep the part with smaller gradient, and achieve dynamic range compression while preserving image details [21]. The algorithm implementation environment refers to the system shown in Table 1.
First, a compression method of dynamic range is proposed in a one-dimensional case; its purpose is to compress the amplitude of large-scale changes as much as possible, while preserving small changes. This is achieved by acting on the derivative of luminance with an appropriate spatially varying decay function, as shown in where is the spatial attenuation function; the attenuation effect of this function on large-scale changes is higher than that of small-scale changes [22].
Then, this method is extended to the two-dimensional case, and the gradient is processed; only its size is changed, but its direction is not changed, as shown in where is the image gradient and is the decay function.
In order to better detect image edge information to avoid halo phenomenon, the author introduces a multiresolution edge detection method to perform Gaussian pyramid decomposition on the image, as shown in where is the -th level of the pyramid and is the gradient of the -th level pyramid. Each layer of the pyramid corresponds to a decay function, as shown in
In the formula, controls the gradient compression range, those larger than are compressed, and those smaller than are amplified.
Finally, after merging the decay functions layer by layer, by solving the Poisson equation, find the brightness image whose gradient approximates ; finally, exponentialize to get the output image. The image processed by this algorithm shows more image details, and the details in the bright and dark places are rich, but the complexity of the algorithm is too high.
3.1.4. Histogram Equalization Algorithm
For high dynamic images, the simplest display method is to linearly map the dynamic range of the high dynamic image to the dynamic range of the display device, but most of the results of linear mapping of high dynamic images are not ideal. Figure 4 shows the processing effect of a high dynamic range image after linear mapping, only a small amount of highlighted areas in the image remain in the output image, and other darker areas are completely black. The reason for this phenomenon is that the gray level of most pixel values in high dynamic images is small; linear mapping is equivalent to discarding the lower bits of the image data; in the processed result, most of the pixels are set to 0 or close to 0 due to their small values, resulting in poor image processing effect.
Figure 5 is a schematic diagram of the histogram shape of a typical high dynamic image; the tone mapping algorithm needs to merge the sparse and less pixel-distributed areas and expand the more densely distributed areas.
Histogram equalization is a commonly used image enhancement algorithm; it has a good image enhancement effect on images with uneven pixel distribution. The idea of the histogram equalization algorithm is to transform the histogram of the processed image into an approximately uniform distribution, in order to achieve the purpose of enhancing image contrast [23]; it can solve the problem of uneven distribution of image gray scale. Since the tone mapping algorithm is essentially a compression of the dynamic range of luminance, so the author converts the RGB image to HSV color space, only the V component representing luminance is processed, the advantage of this is that the hue and saturation of the original image can be kept unchanged, and the extraction of the V component is shown in
After the luminance component of the image is obtained, its histogram is counted according to where is the luminance component of the input image and is the histogram of the image.
After obtaining the histogram, calculate its grayscale mapping function, as shown in where is the maximum gray level of the output image, usually 255 and is the maximum gray level of the input image, such as a high dynamic image with a depth of 20 bits; the value of is 1048575.
3.2. Second-Level Histogram Equalization Algorithm
In the previous section, the author discussed that the histogram equalization algorithm has a good processing effect on high dynamic images, and because of its simple calculation, the result is the output of the look-up table; it is more suitable for implementation in high dynamic image processing system. However, in the actual development of the system, there will be a problem that the histogram occupies too much memory; for a high dynamic image with an image depth of 20bit, the gray level range is [0, 2-1]; therefore, the memory space required for an image statistical histogram with an image resolution of 720p is 16 Mb; the on-chip memory of general embedded processing chips usually cannot meet this requirement, if the histogram information is stored in off-chip DDR memory, additional DDR access time overhead is required; it is not easy to meet the real-time performance of the entire high-dynamic image processing system. Therefore, the author proposes a two-level histogram equalization method, which avoids the problem that the histogram of high dynamic images occupies too much memory; it can achieve the processing effect of approximating the histogram equalization algorithm in the case of greatly reducing the memory usage.
In order to reduce the memory required to count the primary histogram, the method adopted by the author is to count the histogram after taking high bits of the input image; for example, for a high dynamic image with a depth of 20 bits, if the upper 13 bits of the input image are taken, the memory space required for the statistical histogram is 32 KB (256Kb); the memory space occupied by it is greatly reduced. First, take the high -bit statistical histogram for the input original high dynamic image, as shown in where is the high bit of the input luminance image , is the number of bits of the input image, is the high-order digit for the statistical histogram, and is the first-order histogram.
After obtaining the high-order histogram of the input image, perform equalization calculation on the histogram to obtain the first-level grayscale mapping table, as shown in where is the maximum brightness of the output image, is the maximum brightness after taking high bits of the input image, is the grayscale mapping table after one equalization, and is the number of pixels of the image.
After the first-level grayscale mapping table is obtained, the output image and its grayscale histogram can be obtained according to the grayscale mapping table. According to the image mapped from the first-level grayscale mapping table, it can be seen that some details of the image are lost; this is due to the high bits of the input image when the histogram is counted; therefore, a large number of pixels are merged into the same gray level. In the histogram, the phenomenon of losing details is reflected as many “valleys,” and a large number of pixels merged into the same gray level are reflected as higher “peaks” in the histogram and the “valleys” between the two “peaks” in the figure.” It can be seen as an interval.
In order to enrich the details of the image, grayscale expansion can be performed on these intervals, make the pixels accumulated on the “peaks” fill these “valleys,” and redistribute the pixels accumulated to a certain gray level to achieve the purpose of detail enhancement. For each interval in the histogram, it can be regarded as an image with uneven grayscale distribution, so the histogram equalization can be performed on this part of the interval again.
Two parameters need to be determined when doing secondary balance for these ranges: The interval is in the grayscale range of the output image, and the interval is in the grayscale range of the input image. The interval in the grayscale range of the output image can be determined by the first-level grayscale mapping table; in the first-level equalization, since the high bits of the input image are taken, all pixels are naturally merged according to the high bits of their gray levels and then transformed to a certain “peak” value, so a certain “peak” in the histogram, in fact, contains the different grayscale values of the input images that are merged because only the high K bits are taken; if conventional histogram equalization is performed on a high dynamic image, the number of pixels of the gray level on the “valley” to the left of the “peak” value may not be zero; therefore, the grayscale range of the output image can be determined according to the “peak” value and redistributes pixels that pile up to “peak” values to gray levels on “valleys,” increasing the detail of the image.
4. Analysis of Results
4.1. Analysis of Brightness Histogram Effect
According to the improved algorithm above, after the pixels are redistributed, the image processing result and its histogram are shown in Figure 6.
It can be seen from Figure 6 that the brighter part of the figure and the darker background areas are clearly displayed in the same image, and the details are relatively intact. From the histogram of the luminance component of the output image after the histogram equalization process, it can be known that the grayscale distribution of the luminance component of the histogram equalization output image is more uniform, it can display the high-brightness area and low-brightness area of the image in the same image and maintain the relative relationship between the brightness values of the image, the effect of high dynamic image enhancement processing is achieved, and the computational complexity of the entire algorithm flow is low.
Based on this result, the author improves the histogram equalization. In order to keep the average brightness of the image after histogram equalization unchanged, a method based on the average brightness of the image is proposed, which divides the image into two parts for histogram equalization, respectively, in order to preserve the details of the image after histogram equalization as much as possible; it is proposed to find several minimum values according to the histogram of the input image, and according to these minimum values, the image is divided into several intervals for histogram equalization, respectively. The improved effect of histogram equalization obtained after processing is shown in Figure 7.
4.2. Peak Signal-To-Noise Ratio
Peak signal-to-noise ratio (PSNR) is an objective standard for evaluating image approximation effects and used to analyze the quality difference between the processed image and the original image. Usually, the larger the value of the peak signal-to-noise ratio, the closer the two images involved in the comparison are.
The histogram equalization memory optimization method was designed by the author; in the first equalization, the high value has an impact on the processing results, so the author uses different values for testing. Figure 8 shows the processing result of the church high dynamic image, and the number of processing intervals in the figure is 4. When the value of increases, the results of the first-level equilibrium and the second-level equilibrium; the PSNR value is also increased relative to the result of ordinary histogram equalization; the larger the value of , the closer the processing result is to the ordinary histogram equalization algorithm. Meanwhile, for different values, compared with the PSNR value of the second-level equalization processing result and the conventional histogram result, compared with the first-level equalization processing result, and compared with the conventional histogram results, the PSNR values are significantly improved, indicating that the two-level equalization method can achieve the effect of approximating the conventional histogram equalization.
The author selected 5 public high dynamic images of representative scenes for approximation effect analysis; the PSNR values of the two-level equalization processing results relative to the conventional histogram equalization processing results are shown in Table 1; when comparing, the second-level equalization process is performed on all intervals of the first-level equalization result. Table 2 shows that when the value of increases, the approximation degree between the results of the second-level equalization and the histogram equalization is higher, and when the value of is greater than 13 bits, the author’s algorithm has a better approximate effect on the conventional histogram equalization.
In the secondary equilibrium process, the number of intervals will also affect the processing results, Table 3 shows the corresponding PSNR values when the of the first-level equalization module is 13 bits, and the second-level equalization module takes the number of different intervals; the more intervals there are, the closer the effect is to the normal histogram equalization, but the larger the memory space occupied; it can be seen from the table that when the number of intervals is greater than 4; the 5 high dynamic images can achieve a better approximation effect.
4.3. Structural Similarity and Objective Quality Evaluation
Structural similarity SSIM (structural similarity) and objective quality assessment TMQI (Tone Mapped image Quality Index) measure the brightness, contrast, and structure of two images, in order to judge the similarity of the images; the larger the value of SSIM and TMQI, the higher the similarity of the two images.
The author passes the output image of conventional histogram equalization and calculates the structural similarity with the image output by the secondary equalization algorithm. Figures 9 and 10 show the first-level equalization of the church high dynamic image, the value is from small to large, and the number of processing intervals in the figure for the calculation results of SSIM and TMQI is 4. It can be seen from the figure that after the secondary equalization processing, the output image processing effect and the effect of ordinary histogram equalization have been greatly improved, and when the value of is larger, the result of the secondary equalization output image is closer to the result of the conventional histogram equalization output image.
Tables 4 and 5 are the representative high dynamic images of 5 scenes; in the first-level equalization, the calculation results of SSIM and TMQI with different number of digits are used; when comparing, second-level equalization processing is performed on all sections of the first-level equalization output. It can be seen from the table that the larger the value of , the closer the result of the output image is to the result of the conventional histogram equalization output image, and for these seven high dynamic images, when the value of is greater than 13 bits, the approximate results of SSIM and TMQI values are better.
As can be seen from the table, the author’s algorithm ranks first in the TMQI score of the processing results of all images, it shows that the author’s algorithm preserves the structure of the original high dynamic image well, and the image is more natural; at the same time, the TMQI score of the author’s algorithm for all test images reached above 0.98; it is 118.95% of the average value of other classical algorithms, indicating that the author’s algorithm has good robustness and superiority.
5. Conclusion
For indoor space, it runs through all aspects of human society, daily life, work, entertainment, study, etc. which are all related to it. Therefore, people pay more and more attention to the comfort, health, and artistry of the indoor environment.
The pace of urbanization has never stopped, and the environmental problems brought about by rapid development have become more and more serious, people are eager to get in touch with nature, and returning to nature has become particularly important. The application of landscape design elements in interior design has become an effective method to solve this problem. The entry of landscape design elements into the interior space not only improves the relationship between buildings and people but also shortens the distance between people to a certain extent and eliminates the barriers between them.
Based on this requirement, the author designs and implements a high dynamic real-time video image processing system. A two-level piecewise histogram equalization algorithm is proposed for high dynamic image enhancement processing. Improvements have been made to the problem that high dynamic images occupy too much memory resources in the application of histogram equalization; on the premise of greatly reducing memory resource consumption, the image processing effect of the second-level histogram equalization algorithm is close to the conventional histogram equalization. The specific functions of the high dynamic image processing system are realized based on SoC FPGA. Through FPGA programmable logic and C language software design, the system’s chip register configuration, image receiving and processing, high-dynamic image processing, and image data output modules are realized, and the design goal of the system is achieved.
In addition, there are still the following shortcomings, which can be improved in the future: The author uses the MIPI signal level conversion circuit and can also perform level conversion through the chip, such a scheme can support higher rate MIPI signals, and the signals are more stable. When designing the circuit board, a variety of output interfaces are designed, and these interfaces need to be debugged later. The system can also integrate other tone mapping algorithms to meet more application requirements and can continue to expand in the future.
Data Availability
No data were used to support this study.
Conflicts of Interest
The author declares that there is no conflict of interest with any financial organizations regarding the material reported in this manuscript.