Abstract
Now, more and more mobile smart devices are emerging massively; energy consumption of these devices has become an important consideration due to the limitation of battery capacity. Displays are the dominant energy consuming component of battery-operated devices, giving rise to organic light-emitting diode (OLED) as a new promising display technology, which consumes different power when displaying different content due to their emissive nature. Based on this property, we propose an approach to improve image energy efficiency on OLED displays by perceiving image content. The key idea of our approach is to eliminate undesired details while preserving the region of interest of the image by leveraging the color and spatial information. First, we use edge detection algorithm to extract region of interest (ROI) of an image. Next, we gradually change luminance and saturation of region of noninterest (NON-ROI) of the image. Then we perform detailed experiment and case study to validate our approach; experiment results show that our approach can save 22.5% energy on average while preserving high quality of the image.
1. Introduction
With the rapid development of big data, mobile computing, and internet of things, more and more mobile smart devices are emerging in our life. Smartphones as one of typical smart devices have become very popular in recent years, and more than 1.9 billion of them are being used worldwide today. Gartner predicts that they will grow from 1.9 billion in 2015 to over two billion in 2018 [1]. Most of the smart devices are powered by lithium-ion batteries, while the batteries are limited in size and capacity; thus low energy consumption is an urgent concern for mobile smart devices.
Modern smartphones are equipped with a wide range of I/O components and sensors, such as CPU, displays, Wi-Fi NIC, graphics, Bluetooth, GPS, and audio. Many researchers put forward various strategies to reduce energy consumption of mobile devices, such as Lee and Kim [2] who presented a new approach for energy-efficient real-time HAR on smart mobile devices and Peng et al. [3] who obtained the optimal transmission rate threshold at each detection slot time to reduce network energy consumption of mobile devices. Among all the components of mobile devices, displays are the most power-hungry component, which consume 38–50% of total energy [4, 5]. Unlike liquid crystal display (LCD) panels which require high intensity backlight, the new emerging organic light emitting diode displays (OLED) emit light by their pixels themselves, which do not need an external backlight as the illumination source. Thus this brings us a new opportunity for energy saving, since energy consumption of each pixel of the OLED depends on the content displayed. Each pixel of an OLED display emits three channels of the color: red, green, and blue. Dong and Zhong [6] have pointed that energy consumption of the three colors is different, for example, the black color consumes the lowest energy since the luminance of the three channels is zero, and the white color consumes the highest energy since all the luminance of the three channels is fully filled. Many studies [7–13] show that OLED display technology is widely used in various kinds of smart devices, which will become the mainstream display technology in the future for mobile smart device; thus it is meaningful to explore low energy consumption methods for mobile OLED displays.
Many attempts have proposed to reduce energy consumption of OLED displays from different aspects, mainly through dynamic voltage scaling of OLED displays [7–9, 14, 15], context-aware dimming [10–12, 16–18], and color remapping [13, 15, 19–23]. Among these techniques, dimming is a simple and effective method for energy saving, which reduces luminance of the pixels displayed on OLED, while color remapping is implemented by replacing the energy-saving color schemes with the energy-hungry color schemes. Dong and Zhong [6] first proposed power modeling and optimization for OLED displays, and they noticed that OLED consumed drastically different power when displaying different colors due to their emissive nature. Dimming technique was put forward to reduce energy consumption of OLED displays in [10–12, 16–18], and the essence of dimming technique is to reduce the luminance of three color components (R, G, and B) in each pixel of the OLED, so we could conclude that dimming technique is also color remapping. Compared with color remapping, dimming technique is simple and an effective way, which is used widely in mobile games [11] and videos [22]. Inspired by this, we try to improve image energy efficiency for smart devices by dimming NON-ROI of the image.
In this paper, we present our approach by eliminating undesired details while preserving the region of interest of the image. Our approach consists of two major steps. First, we use edge detection algorithm to extract the ROI of an image; we make use of the classic Canny edge detection algorithm and bilateral filter to get the region of interest of the image. Second, we change the luminance and saturation of NON-ROI of the image gradually. We change the luminance and saturation of the image from the salient region to image boundary so as to ensure maximum image quality, which can reduce the maximum power consumption of NON-ROI area while ensuring that the average structural similarity (MSSIM) of the image is meeting the user’s vision requirements. Finally, we perform detailed experiments and case study to validate our approach; experiment results show that our approach can save 22.5% energy on average while preserving high quality of the image, which demonstrates the effectiveness and efficiency of our approach.
The organization of this paper is as follows: related work is presented in Section 2, and our approach is elaborated in Section 3; experiment evaluation and discussion are given in Section 4, and we conclude our research in Section 5.
2. Related Works
2.1. OLED Energy Model
OLED is a new emerging display technology which provides wider viewing angels and better power efficiency than traditional LCD and which is widely used in commercial applications such as displays for various kinds of smart devices and portable digital devices. The main difference between OLED displays and LCD displays is that OLED displays do not require external lighting and the pixels of OLED are emissive by themselves. Each pixel of an OLED consists of three color components, namely, red, green, and blue. Dong and Zhong [6] first proposed power modeling and optimization for OLED displays; they modeled the power contributed by a single pixel, specified in (R, G, and B), shown as formula (1). In order to measure the energy consumption of one image, we also make use of the generic OLED power model proposed by Dong and Zhong.where , , and are power consumption of red, green, and blue devices of the pixel, respectively. And the power consumption of an OLED display with pixels is (2). is the static energy consumption which is dominated by a driven current of the control chips and which can be estimated by measuring the energy consumption of a completely black screen. We can get , , and by measuring the energy consumption for each individual channel with different intensity levels. Figure 1 shows the energy consumption of three colors with different intensity on a μOLED-32028-P1 AMOLED display.
2.2. Energy Saving
Smart devices have become a part of everyone’s life, which offer more and more services for our daily life. Particularly for smartphones, there are millions of applications in Google Play and App Store, and these rich applications require battery to provide more energy. The capacity of lithium-ion batteries is still constrained by the size and weight due to smart devices’ mobility; thus energy saving of smart devices has become an urgent concern. Many studies have been made to reduce energy consumption of smart devices from different aspects; particularly for OLED, there are mainly three ways for energy saving, namely, dynamic voltage scaling [7–9, 14, 15], color context aware dimming [10–12, 16–18], and color remapping [13, 15, 19–23].
Shin et al. [7] first introduced dynamic driving voltage scaling (DVS) of OLED panel technique; the idea is to scale down the supply voltage and, in turn, dramatically reduce the wasted power caused by the voltage drop across the driver transistor as well as internal parasitic resistance; thus energy is saved on the OLED display panel with only minor changes in the color and luminance of the image; their experiment shows that their method saves up to 52.5% of the OLED energy while keeping the same image quality for the Lena image. Based on their work, Chen et al. [8] proposed a new fine-grained dynamic voltage scaling method; the key point is partitioning the OLED panel into multiple display areas and adjusting the supply voltage based on the displayed content, and they designed a DVS-friendly OLED driver to enhance the color accuracy of the OLED pixels at the scaled supply voltage. Their experimental results show that, compared to existing global DVS technique, FDVS technique can achieve 25.9%~43.1% more OLED power saving while maintaining a high image quality measured by Structural Similarity Index (SSIM = 0.98). Also in [15], Song and Park presented a decoding model that allows buffering frames to let the CPU run at low frequency to reduce the energy required for video decoding. These two methods focus on the hardware structure of the OLED, which are compatible with our method, since our method focuses on the image. In our study, we also use SSIM to validate the image quality to ensure the effectiveness of our method.
Since power consumption of each pixel of the OLED depends on the color displayed, previous energy saving methods [9–13, 19–22] mainly change the color or luminance of the displayed image. As illustrated in Figure 1, we can clearly observe that blue is the most power-hungry color and energy efficiencies of different colors are different. Dong et al. [13] first took a commercial off-the-shelf QVGA OLED module to adapt GUIs for energy saving; they designed a color-adaptive web browser for mobile OLED displays in [20], and the browser renders web pages with energy-saving color schemes under user-supplied constraints. Similarly, Li et al. [21] proposed an approach for automatically rewriting web applications so as to generate more energy-efficient web pages for mobile smart devices. Wang et al. [19] put forward an approach to reduce power consumption on OLED displays for sequential data visualization by replacing autogenerated color schemes with the most energy-saving color schemes. They first create a multiobjective optimization approach to find the most energy-saving color schemes for given visual perception difference levels and then apply the model in two situations: predesigned color schemes and autogenerated color schemes. These methods are all the specific implementation of color remapping technique, while remapping color is computing intensive and complicate for image processing. Dimming is a simple and effective way, which is widely used for game and video processing in [10–12, 16–18].
Dalton and Ellis [10] first used dimming technique to reduce energy consumption of the displays; they used a web camera to detect the user’s face, keeping the laptop’s display on as the user is present and turning it off when the user leaves, while this dimming method is coarse-grained. Then fine-grained dimming attempts [11, 12, 16–18] were proposed; Wee and Balan [11] put forward a technique which makes use of saliency, by reducing the brightness of game areas which are not of interest currently to the game player to reduce the power consumption of OLED displays. They assumed that the region of interest was the center of the screen and used a rectangle representing the ROI and then computed a series of dimming boxes (rectangle) from the ROI to the edge of the screen. While our ROI is obtained by edge detection algorithm, which is more accurate than the assumption that ROI is center of the screen, the shape of our dimming boxes is based on the shape of the ROI, which is more helpful to preserve the quality of the image. Betts-LaCroix [12] put forward dimming selected areas of the OLED display according to use’s perception, which also results in power savings for OLED display.
Also in [16–18], all authors used dimming technique for power saving. The key idea of [16] is that they also use the notion of saliency to save display power by dimming portions of the applications that are less important to the user, while they used a simple ROI model which assumes that user attention is directed mostly towards the top or bottom portions of the screen when using an application. Choubey et al. [17] noticed that if distance between two point size light sources is less than 0.04–0.05 mm, they will appear as single source for human visual system. Based on human eye’s visual acuity, even if some subpixels are turned off, they will not be perceived. Thus a display content and human visual acuity aware technique to reduce OLED panel power consumption was proposed. The crucial step is turning off selective subpixels in specific regions of display. This approach focuses on the manipulation of OLED according to the display content and human visual acuity, while our approach is focused on the image itself. The common point between their method and our method is reducing the power consumption of pixels in specific region of the display. Lin et al. [18] introduced an alternative low-power technique called image pixel scaling, which leverages the flexibility provided by OLED technology to scale down the pixel values of different-shaped regions; this approach is similar to ours. And they proposed the design, algorithm, and implementation of a novel framework called CURA for quality-retaining power saving on mobile OLED displays. Their method is able to display an image without adversely impacting the user’s visual experience, while they implemented attention region segmentation, region distortion assessment, and boundary effect elimination to target the ROI of the image, which led to more computational overhead. Compared to their method, our approach is more simple and fast in image processing, while maybe the effect of power saving is not better than theirs.
Most of these studies reduce energy by dimming or turning off the selected areas of the OLED when displaying content, while our approach is different from these methods, since we focus on the image itself and we try to improve image energy efficiency for mobile smart devices initiatively.
3. Our Approach
For a specific image, our approach consisted of two main steps, namely, region of interest (ROI) abstraction and region of noninterest (NON-ROI) dimming. First, we use edge detection algorithm to extract ROI of the image and then adjust the luminance and saturation of NON-ROI of the image smoothly. In our paper, we use classical Canny [24] algorithm for ROI abstraction, dimming NON-ROI of the image by adjusting luminance and saturation (ALS) algorithm; detailed steps are presented below.
3.1. ROI Abstraction
The region of interest of an image is also known as salient region of the image, which shows the main content and the most interesting area of an image. ROI abstraction is the process of preserving the salient region while eliminating undesired details of the image. Usually we use edge detection for feature detection and extraction; edge detection is designed to detect edges or a significant change in discrete regions of a digital image. Typically used edge detection algorithms have Canny edge detection, wavelet transform detection, and fuzzy theory detection and so on. In this paper, we use classical Canny algorithm for image edge detection. Canny operator is considered to be the most classic edge detector, and many of them do comparative analysis always refering to this standard. Canny operator is originally designed for gray image edge detection, with the increasingly wide range of applications of color images, which is also applied to edge detection of color images. In the process of edge detection for color image, we need to convert color image to gray image; main steps of the algorithm are described below.
(1) Image Graying. The process of turning a colorful image into a gray image is called image graying. Detailed steps calculated the average value of the three components (R, G, and B) of each pixel and then assigned the average value to the three components of each pixel. Considering human physiological characteristics, we convert a color image to a gray image according to formula (3). This step is not required for gray images.
(2) Image Smoothing. Smoothing is blurring the image to remove noise. Canny algorithm uses Gauss filter to smooth the image. In subsequent applications, Tomasi and Manduchi [25] find that Gauss filter obviously blurs the edge and protective effect of high-frequency details is not obvious. In this paper we use bilateral filter for image smoothing, which is a nonlinear, edge-preserving, and noise-reducing for images and is based on the combination of the spatial proximity of the image and the similarity of each pixel, considering the spatial information and the gray similarity. Moreover, it can also achieve the goal of edge-preserving and noise-reducing. The formula of bilateral filter is as follows:
is the filtered image and is the normalization factor. I is the original input image to be filtered, x is the coordinates of the current pixel to be filtered, is the window centered in , is the range kernel for smoothing differences in intensities, and is the spatial kernel for smoothing differences in coordinates.
(3) Nonmaximum Suppression. This step is to find the location of the gray intensity with the sharpest changes in the image and then makes the fuzzy edge become clear through nonmaximum suppression. The gray intensity with sharpest changes is the great changes of the gradient direction of the image; gradient and direction of one pixel can be calculated by calculating the gradient and direction in and ; the formulas are (6) and (7), where and are the gradients in the and directions, respectively. Then according to the gradient and the angle calculated, take nonmaximum suppression and preserve the local maximum gradient points to realize the edge thinning, which will reduce the edge pixels after these steps and reduce the difficulty of determining the edges.
(4) Double Thresholding. After nonmaximum suppression, edge pixels are quite accurate to present the real edge. However, there are still some edge pixels at this point caused by noise and color variation. In order to get rid of the spurious responses from these bothering factors, it is essential to filter out the edge pixel with the weak gradient value and preserve the edge with the high gradient value. The simplest way to discern between these would be to use a threshold; thus only edges which are stronger than a certain value would be preserved, so the algorithm uses double thresholding to finish this process.
(5) Edge Tracking by Hysteresis. Strong edge pixels are certainly preserved in the final edge image since they are extracted from the true edges in the image. However, for the weak edge pixels, there are still some debates on these pixels which can be extracted from either the true edge or the noise/color variations. In order to get an accurate result, weak edges caused by the noise/color variations should be removed. Generally a weak edge pixel caused by true edges will be connected to a strong edge pixel while noise responses are unconnected. Then blob analysis is applied by looking at a weak edge pixel and its 8-connected neighborhood pixels to track the edge connection.
3.2. NON-ROI Dimming
Adjusting luminance and saturation of the NON-ROI of an image is the key step for energy-saving, in order to ensure the luminance and saturation of the image having a smooth transition. We adjust them from the salient region to the image boundary gradually in order to ensure maximum image quality. In this paper, we take luminance adjustment and saturation adjustment simultaneously, reducing the maximum power consumption of NON-ROI region while ensuring the average structural similarity (MSSIM) of the image to meet user’s vision requirements.
(1) Luminance Adjustment. For a given pixel, its luminance is weighted sum of three-color (R, G, and B) pixel values; the formula is (8), where is weighted sum function, and for gray image the weighted sum function is (3). From the formula, we know that the greater the value of the three colors, the higher the luminance of the pixel. When we reduce the luminance of one pixel, which directly reduces the value of each color component of the pixel, the luminance adjustment can be expressed as (9).
is the original luminance of one pixel and is the luminance after adjusting. is the adjustment parameter and its range is between 0 and 1. If the value of is 0, it means we do not adjust, and if the value of is 1, it means adjust completely and the pixel becomes complete black. is the number of the adjustment regions of noninterest region of the image, and the reasonable range of N verified by experiment in this paper is 3–15. is the region number of the pixels located, and we assume the value of the first noninterest region adjacent to the domain of interest is 1, and the region number of the following region is increasing in turn, and the last region of the noninterest is . We continuously increase the value of from the ROI of the image to the image boundary in adjusting process.
(2) Saturation Adjustment. Saturation describes how a pure color is mixed with achromatic components, which is also known as the purity of color. Bright image always has higher purity, and bleak image has lower purity. The goal of adjusting saturation is to reduce the contributions from the power-hungry color components and enhance the power-efficient ones while maintaining overall luminance. From the OLED power model described in Section 2.1, we know that blue is the highest energy consuming color, so we first consider reducing the blue component of one pixel; thus our saturation adjustment method is described as follows:
Here Δs describes the color difference of three primary colors, in practice which is usually around 60. is the adjustment amount, is number of grading regulation, and is the region number of the pixels located. We also assign the value of the first noninterest region adjacent to the domain of interest to be 1; the region number of the following region is increasing in turn, and the last region of the noninterest is N.
Now we present our dimming algorithm, which consists of the above two adjustment strategies, which is able to adapt to different images since it is continues to change the adjustment parameter according to the MSSIM of the image. Now, we present our algorithm, namely, ALS (adjusting luminance and saturation of NON-ROI of the image). and are the ROI coordinate matrix and the NON-ROI coordinate matrix of the image; is the gradual changing matrix set which is obtained by dividing into matrix sets; is one pixel in ; and we assume the number of each is . is the luminance adjustment parameter and is the saturation adjustment amount. The pseudocode of ALS algorithm is shown in Algorithm 1.
| ||||||||||||||||||||||||||||||||||||||
4. Results and Evaluation
In this section, we first introduce configuration of our experiment. Next, we implement our dimming algorithm to process images for energy saving. In order to compare the energy saving effect of our algorithm, we present the other two adjust methods: AL (adjusting luminance of the whole image) and AS (adjusting saturation of the whole image), and our approach is ALS. Then, we present one example of processing the image “Seagull,” listing the energy saved and some important attributes of the image to illustrate our algorithm. Finally, we randomly select 200 images to verify the generality of our method.
4.1. Experiment Settings
First, we need to get the energy consumption model used in our experiments, which is composed of three estimation functions , , and (in formula (1)). Since these functions depend on specific displays, we measure them on an aμOLED-32028-P1 AMOLED display module with HOIKI 3334 power meter; detailed configuration of our experiment platform is shown in Table 1. Then we calculate the energy consumption by tracking the electrical current values, and we use KA3005P DC power supply to provide stable and controllable voltage, and leverage HOIKI multifunction power measuring instrument to power consumption of the displays. We change the intensity levels from 0 to 1 for testing each color channel in measuring , , and . In each test, we fill the OLED with corresponding color for 60 seconds to calculate the average energy consumption and detailed results are shown in Figure 1. From Figure 1 we know that the power consumption of each color component is a nonlinear; in order to simplify the calculation, we use least squares curve fitting to get linear relation between the color component and its power consumption as shown in Figure 2. Then we implement the three algorithms to validate the effect of energy saving, and we select one image “Seagull” [26] which is selected from the Google gallery (https://image.google.com/) to validate our algorithm.
4.2. Results and Discussion
Figure 3 illustrates the effects of the above three algorithms in processing image Seagull: (a) is the original image, (b) is adjusting luminance of the whole image, (c) is adjusting saturation of the whole image, and (d) is the image using our approach (ALS). Table 2 shows some important attributes of images (a), (b), (c), and (d). We assume the SSIM of original image is 1, and original luminance and saturation of the image are fixed values (we use “—” to stand for the value), and a negative value in the table indicates how much is reduced on the basis of the original value. Table 2 shows experimental results of our method in terms of power consumption and image similarity with the other two methods. First column is the approach name; second column is grading regulation; third column, fourth column, and fifth column are the SSIM of the ROI, NON-ROI, and whole region of the image; sixth and seventh columns are the luminance of the ROI and NON-ROI of the image; eighth and ninth column are the saturation of the ROI and NON-ROI of the image, and the last column is the power consumption of the image.
(a) Original
(b) AL
(c) AS
(d) ALS
From Table 2 and Figure 3, we observe that the images processed by the three methods have lower power consumption than original image. The original energy consumption of the image is 2374.836 μw, when the is 0.95 for the image after using three methods; the luminance of image (b) is reduced by 18.76% and energy consumption is reduced by 26.68%; for image (c), the saturation of image is reduced by 21.47% and energy consumption is reduced by 25.96%; and for image (d) the luminance is reduced by 24.24% and saturation is reduced by 19.68%; energy consumption is reduced by 26% when is 9. At the same time, from Table 2, we observe that when increases gradually, the MSSIM of the image increases, while power consumption of the image does not change significantly. Thus the brightness and saturation of the image change more smoothly, so the quality of the image cloud be maintained. Our method preserves the SSIM of ROI completely of the image in dimming process, from Table 2 and Figure 3, which shows that our method is able to reduce the same energy while maintaining high quality of the image.
In order to verify the generality of our method, we performed a case study to assess the effectiveness and users’ acceptance of our energy-saving method. We randomly selected 200 images from the Google image database for statistical analysis; for each image, we take the same steps to deal with it. First, we recode energy consumption of the original image and then recode energy reduction of the image processed by our ALS method. Finally, we calculate the energy saving ratio of all the images and the result is presented in Figure 4. Figure 4 is the distribution of the power reduction ratio of 200 images processed by our ALS algorithm; all the MSSIM of the image maintain a value more than or equal to 0.93. From the figure, we can observe that 41% of the experimental samples save energy consumption by 20%–30% and 24% of the samples save energy consumption by 30%–40% and only a few samples have low energy saving since the content of these images is fully filled. The average energy saving of all the test samples is 22.5%, which proves the effectiveness and generality of our method.
Based on the quantitative results of Table 2 and Figure 4, we clearly see that dimming method is effective for energy saving. And our ALS algorithm can be more acceptable than results of uniform dimming methods (AL and AS), since our approach keeps the ROI of the image and adjusts the luminance and saturation gradually, which makes the change of the image more smoothly. In conducting the case study, we find that when white or blue are the main colors of an image, the energy saving effect is obvious, while the energy saving effect is not obvious when the main color of an image is gray. What is more, our algorithm is very effective in dealing with the image with single ROI and the region of ROI is limited; since the region of ROI is limited, we can change much of the region of NON-ROI of the image. Our algorithm is not effective for the image fully filled with content, since the NON-ROI of these kinds of image is very limited. And for the image with no single ROI, we segment an image into a set of ROI regions and for each region applying our algorithm to calculate the maximum change that each region can tolerate according to the average SSIM of the image, which will cause more computing in detecting ROI of the image and calculating the maximum change of NON-ROI of the image.
5. Conclusion
In this work, we propose a new approach to improve image energy efficiency for mobile OLED displays. First we use edge detection algorithm to extract the ROI of an image and then adjust the luminance and saturation of NON-ROI of the image gradually. By this way, we can reduce significant amounts of energy consumption while preserving high quality of the image. Experiment results show that our approach can save 22.5% energy on average while preserving image quality. Our method is simple and effective for energy saving and can be used for video and other multimedia applications.
Competing Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported in part by the State Key Program of National Natural Science Foundation of China under Grant no. 61332001, the National Natural Science Foundation of China under Grant nos. 61272104 and 61472050, and the Applied Basic Research Program of Sichuan Province under Grant nos. 2014JY0257, 2015GZ0103, and 2014-HM01-00326-SF.