Abstract
Electronic displays and cameras can provide an intuitive, simple communications interface without dependence on additional wireless interfaces or the Internet infrastructure. In this paper, we design a complementary color barcode-based optical camera communication (CCB-OCC) system to provide an easy-to-use communication capability from an electronic display-to-camera (D2C) link. The proposed method encodes information into specially designed color barcodes and transmits it in a format perceptually invisible to humans but detectable by camera-equipped devices. In addition, we propose a new transmission packet design that contains pilot symbols to synchronize symbol packets and estimate the D2C channel link for calibrating captured images caused by irregular differences between the sending color and the receiving color in the D2C link. Experimental results verify the feasibility of the CCB-OCC scheme for short-range communications to offer additional information which shows a new possibility in designing a D2C communication system with robust to environmental change, easy-to-use, and simple implementation.
1. Introduction
With the quickly growing amounts of smart content in our daily lives, people rely more and more on smart devices. At the same time, various devices equipped with displays have become a readily available information source for human users. Therefore, we can expect displays to convey information to people while further delivering other meaningful and customized content to their smart devices. In particular, there is an exploding interest in content-delivering technology via links between various types of displays and cameras integrated into smart devices, which are called “display-to-camera (D2C) links.” Accordingly, this technology makes it possible to provide various personalized public information services for institutions, events, advertisements, etc., by using a digital display in institutions, public spaces, and commercial places.
The most commercialized application of the D2C link is to obtain information from quick response (QR) codes [1,2] by using the scanning app of a mobile phone. As a general rule of thumb, QR codes should be used minimally and should not disrupt or dominate the original visual content. However, this causes reduced throughput and limited communication distances. To overcome these drawbacks, it is necessary for digital marks to expand their size and to use additional dimensions, such as colors and shapes.
As a congruent solution to the spectrum congestion of RF-based technologies, optical camera communication (OCC) [3–6] has become a new trend, operating in the visible light spectrum and using a camera as a data receiver. The data can be modulated by variations in light from electronic displays and can be detected from captured images. By taking advantage of the inherent characteristics of OCC technology and QR codes, we can obtain supplemental channels for data communication over a D2C link. In this concept, the camera is used not only for capturing images with visual information but also for recognizing data transmitted from the electronic display.
Recently, a lot of research into D2C communications has been conducted. InFrame [7] and its updated version [8] provide an overlay of data patterns based on “superpixels” and pixel blocks. For this, temporal luminance modulation is used with smooth transitions between different data patterns. HiLight [9] applies frequency modulation of a video transparency (D-channel) to transmit data over a D2C link. DisCo [10] is based on a rolling shutter camera to execute the decoding by translating the temporal sequence into a spatial pattern. Secure barcode-based visible light communications (SBVLC) [11] addresses geometric models and physical security enhancement mechanisms for barcode communications. RainBar [12] proposed code locator detection and localization scheme to provide robust color recognition and its updated version RainBar+ [13] is an online robust color barcode-based VLC system with real-time feedback. In [14,15], “SoftLight” scheme is presented over screen-camera links, which uses color modulation schemes with channel coding and “Soft hint” for data decoding through barcode layout. To the best of the authors’ knowledge, the abovementioned techniques do not consider complementary color-based packetization of pilot and data symbols that are imperceptible to the human eye, along with optical channel estimation between a display device and a camera; thus, performance may vary depending on the ambient optical noise or the characteristics of the transceiver.
In this paper, we propose complementary color barcode-based optical camera communication (CCB-OCC) providing a robust and easy-to-use communication capability via D2C link. The proposed method encodes information into specially designed, invisible color barcodes called “complementary color-based encoding” and transmits them from electronic displays to cameras in smart devices. Note that the proposed scheme is similar to the conventional RGB LED-based CSK modulation scheme [16], based on the x-y color space chromaticity diagram, defined by the international commission on illumination in CIE 1931 color space. CSK modulation uses three different light sources of triangular shape for the constellation diagram, which results in white tonality in average. However, since the constellation of CSK has an irregular shape, the undesirable, possibly time-varying, color can be perceived unless a strong assumption is given that each symbol occurs with the same probability. Unlike the conventional CSK scheme, the proposed complementary color-based scheme performs constellation signaling based on the round-shaped Hue circle. In addition, the data modulated color is sent and soon afterwards the complementary color of the color is sent, and thus, the probability of generating the primary and complementary colors becomes exactly . Therefore, the proposed scheme can achieve a desired white tonality in average without the strong assumption of equally likely symbols.
In addition, we also propose a new transmission packet design which is configured in the form of a pilot symbol for both synchronization and D2C channel estimation followed by a data symbol. After finding the start point of data symbols in the packet through synchronization procedure, the estimated channel information is used for calibrating distorted images caused by different hardware and software characteristics of the display and camera and by ambient light noise, viewing angle, and so on. Each symbol is transmitted with complementary pairs that are perceptually invisible to the human eye but detectable by camera-equipped devices. Experimental results prove that the CCB-OCC scheme can be a viable solution for short-range communications that offer additional information that is invisible to humans but can be detected by a camera.
The remainder of this paper is organized as follows. The system model of the complementary color barcode-based OCC scheme is presented in Section 2. Section 3 addresses packet synchronization, channel estimation, and the decoding process. Section 4 discusses the experimental results. Section 5 presents the conclusions.
2. System Model of Complementary Color Barcode-Based OCC
In the CCB-OCC scheme, the transmitter encodes data into color barcodes and displays the packet of color barcodes on its display. The receiving side uses a camera to capture successive images including the color barcodes and decodes the captured images to obtain the transmitted data. The concept and the system model of CCB-OCC, consisting of a display transmitter and a camera receiver, are presented in Figures 1 and 2, respectively. The white stripe in Figure 1 means the color barcode region containing transmit data. This image region is perceived as white bar in human eyes, but when captured with a camera, it is perceived in the form of a color barcode due to the rolling shutter effect, and thus, it can be used for data reception.
In the transmitter, the main objective is to encode binary data into color barcodes in a complementary manner and display them on an electronic display with unnoticeable variation in color intensity. For this, a bit-to-color mapping process is performed to convert an input bitstream into a specific color. The digital bits are mapped to color according to the predetermined symbol constellation design. The colorized symbol is composed of complementary color pairs, connected with a pilot symbol for synchronization and channel estimation, and packetized for transmission from a display. The images captured by the camera in the receiving device are sequentially stored in a buffer, and the color barcode region is extracted via difference image. Since the packet consists of pilot and data symbols with the mixture of red-green-blue (RGB) colors, the pixel value of pilot symbols is searched for packet synchronization. Since the RGB component along the line-scanned image is found using histogram analysis, the wireless optical channel between the electronic display and the camera is estimated from the image frame including the pilot symbol, and then, the remaining data symbols can be decoded using the estimated channel information.
2.1. Constellation Coding and Data Mapping
CCB-OCC adopts a new color barcode design that features an image frame that is perceptually transparent to the human eye but detectable by camera-equipped devices. This can be done by sequentially showing complementary colors where any two colors are located opposite each other on the Hue circle. As seen in Figure 3, complementary colors on the Hue circle differ by 180 degrees. Considering blue, green, and red as primary colors, blue and yellow, green and magenta, and red and aqua are complementary pairs. The condition of a complementary color is presented as follows:where are the values of the red, green, and blue channels of the primary color, and are the values of the red, green, and blue channels of the complementary colors. If two colors are combined, they become white, so the values of the red, green, and blue channels are 255, 255, and 255, respectively. If two successive colors in complementary pairs change rapidly on the display device, the human eye will not recognize the original color, and it will be recognized as white. Using this phenomenon, the proposed color barcode patterns are not visually appealing and significantly distract from the overall production quality of the display content.
Table 1 and Figure 3 present the code constellation and bit-to-color mapping of the CCB-OCC scheme according to the Hue circle. To minimize the number of bit errors in the process of converting from decoded symbols to received symbols, we consider gray coding in the code constellation.
Figure 4 presents the successive primary and complementary color pairs observed by the camera device. When the color barcode image from a monitor operating at is captured with a camera sensor at , at least two colors appear in every image frame. As seen in the figure, the RR’GG’BB’ color pattern, where R, G, B mean the primary red, blue, and green color symbols and R’, G’, and B’ mean their complementary colors, is appeared. Theoretically, the number of colors observed in the color barcode in a single received image is the ratio of the transmission speed to reception speed, that is, the ratio of the data encoding rate to the camera capture rate. As shown in Figure 4, six colors of RR’GG’BB’ are observed along the consecutive three image frames because the reception speed is half of the transmission speed. Since the main color and complementary color of the color barcode region are observed as a continuous color change by the rolling shutter effect, the synchronization of packets is important for channel estimation and data decoding. The transmission frequencies can be controlled by varying the refresh rate of the display device. When a display with a higher refresh rate is utilized, the number of colors appearing in a single received image will further increase, thereby increasing the data rate.
In this way, the color-mapped symbol is aggregated in the form of a data packet structure and is transmitted via the display screen. As shown in Figure 5, the packet structure of the proposed scheme was defined with pilot color symbols and data symbols with complementary color pairs. These symbols are encoded in the color barcode region of each image frame transmitted through the display device. The pilot symbols are composed of six consecutive frames of RR’GG’BB’ color pattern. And the subsequent data frame is composed of multiple complementary color (CC) pairs of data. These frames with encoded data are aggregated and displayed on the screens, and a smartphone can receive the data by capturing the images with its camera and decoding the image stream.
2.2. Synchronization
D2C communication systems require synchronization to ensure successful decoding from the consecutive captured frames. The proposed scheme modulates a bitstream to well-defined colored pixel blocks, which are then demodulated accordingly to retrieve the original bitstream. From the received frame patterns, our frame synchronization approach is to find the consecutive synchronization frames with RR’GG’BB’ color pattern. Since the combination of modulated colors of the data payload does not produce the synchronization color pattern, the receiver examines the last six consecutive colors to find the start and end points of the packet.
Figure 6 presents the example of detecting pilot symbols for packet synchronization. As can be seen, pilot symbols of RR’GG’BB’ colors are received from -th to -th image frames. Since the color barcode region from the received image includes a continuous color change, the packet is synchronized based on the point that is the maximum value of the RGB histogram rather than expressing the exact packet boundaries. Note that RGB histogram of the proposed scheme is used not only for packet synchronization, but also for channel estimation and data demodulation. Once RR’GG’BB’ color pattern is determined, the subsequent data symbols with complementary color pairs are decoded.
The CCB technique is designed to add data communication functions while maintaining the original functions of the display and camera. Therefore, the remaining parameters except the camera’s frames per second (fps) and the display’s refresh rate, which have the greatest effect on the rolling shutter effect, are perceived as a kind of noise from the receiver’s point of view. Therefore, the histogram analysis and channel estimation technique of the received image can overcome the influence of the remaining parameters.
2.3. Optical Channel Estimation
In terms of transmitting information, different electronic display types have a different color appearance and brightness distribution. Receivers with different lenses and camera sensors can have different structures for processing color features. Furthermore, when the camera sensor takes a photograph, a color variation occurs due to external factors, such as ambient light luminance and capture angle. This can cause a common problem where the color barcode image taken by one user is significantly different from another’s captured image, thus leading to different decoding performance.
To overcome this problem, the proposed scheme adopts a transmission packet composed of a pilot symbol and a data symbol constructed and stored in a display buffer, and image frames are sequentially displayed on the display device. When a color corresponding to the data to be transmitted is represented on the display, there is a clear difference in the color obtained from the image captured by the camera sensor at the receiving end. In this environment, channel decoding should be performed based on the reference color information included in the pilot frame in order to decode the data transmitted in the form of color from the display screen.
As shown in Figure 4, a rolling shutter effect occurs due to the difference between the transmission speed of the display device and the reception speed of the camera. As a result, a barcode with a plurality of colors is generated on one line in the captured image. In consideration of this phenomenon, the proposed scheme can obtain a representative value of the pilot and data symbols through histogram analysis. Even if there are multiple colors in a single color barcode, we find the histogram that most closely resembles the transmitted signal. In the color values of the CCB-OCC symbols in Table 1, it is easy to note that there is only one component per symbol representing the maximum 255 gray levels. Therefore, the maximum value among the color components representing the histogram maximum, and the color components representing the complementary color, can be found and selected as the reference pixel value at that position.
Since the color barcode region in the image has a significant change in value between all adjacent frames, the color barcode region can be easily extracted via the difference image. By finding the combination of RR’GG’BB’ in consecutive image frames stored in the receiver buffer, the starting position of the packet can be determined, and the pilot and data symbols can be obtained. Through histogram analysis, we note that the color barcode region in the pilot frame is dominated by red, green, and blue monochromes, which are not mixed with various colors. Hence, the average value of an RGB component in the barcode area is calculated and used as a pilot symbol for the RGB component.
From the pilot frame, the relation between the transmitted RGB value and the received RGB value can be shown as follows:where is the channel matrix, is the transmitted color in the pilot i frame, and j is red, green, and blue colors. Similarly, is the captured color in the pilot i frame, where j is the red, green, and blue channels. In the pilot frame, since the color of the signal to be transmitted is previously known, the channel matrix can be obtained through an inverse matrix operation. Using the estimated channel and the obtained pixel values in the color barcode region in the data frame, the corrected red, green, and blue values affected by the wireless optical link between the display device and the camera can be obtained, so they can be utilized for data decoding.
As shown in Figure 6, histogram analysis on the RGB component shows a plurality of colors in one color barcode of pilot frames. In order to estimate a channel for a communication link between display cameras, a reference value for an RGB component at the receiving end should be determined. To do this, the histogram corresponding to the representative color of each pilot frame is analyzed, and the reference value for the RGB component is determined as the value at the maximum point. Here, the reference value is obtained by placing a predetermined window at an image position representing the maximum point of the histogram. Figure 7 shows the window area to obtain the reference value of the RGB component extracted from each pilot frame. Here, the average pixel value in each window area is selected as the reference RGB component. Through this, the relationship between the color information of the signal to be transmitted and the color information received by the camera sensor can be grasped, and this can correct the received data to predict the original signal to be transmitted. From the experimental example, the obtained RGB components of the reference color in the pilot frames were (228.5, 71.9, 95.9) for the pilot R frame, (39.5, 232.7, 82.3) for the pilot G frame, and (55.8, 61.0, 235.2) for the pilot B frame. Then, equation (2) can be rewritten as follows:
Using an inverse matrix operation, the channel matrix between the display and the camera is obtained as follows:
This channel matrix compensates for received color values to reduce the color mismatch between the display and the camera devices.
3. Decoding process
In this section, we present the decoding process of the CCB-OCC scheme. Figure 8 shows a color barcode region within a received data frame following the pilot frame. As can be seen, the change in color appears in a single image frame due to the rolling shutter effect. By analyzing the histogram for RGB components, the peak position at each RGB component can be observed. The pixel location of the peak point in the G component is 425, and the peak points for the B and G components are approximately 1000 pixel locations in the color barcode region. Since complementary symbols are formed in pairs and both colors can be shown in a single frame, the pixel location for detecting data was selected as 425, and the value of the RGB component at this point was (40.7, 198.9, 101.1). By multiplying the inverse function of channel matrix H, the value of the color symbol intended to be sent by the display device can be estimated as follows:
Converting the result of an expression consisting of RGB components to a Hue component yields approximately 140 degrees. According to the bit-to-color constellation, this symbol has the minimum Euclidean distance from 135 degrees, and thus, the received symbol can be decoded to “010”.
The peak position of the R and B components observed in the histogram show complementary colors and are approximately at the 1000th pixel location, and the corresponding angle on the Hue circle is 298 degrees. According to the bit-to-color constellation, this symbol has the minimum Euclidean distance from 315 degrees, which is the complementary color of 135 degrees. In this way, consecutive data symbols in the packet can be decoded.
4. Experimental Result
In this section, we conduct extensive experiments to evaluate the performance of the CCB-OCC scheme. The experimental results are showed under a variety of settings on the CCB-OCC scheme configurations and environmental factors including sender-receiver distance, view angle, and modulation order. Our receiver devices was a Samsung Galaxy Note 9 (device A), which is an off-the-shelf, normal Android smartphone equipped with a standard camera featuring a video capture rate. On the transmitting side, the resolution of the display device is , and experiments were conducted indoors under normal lighting conditions.
Figure 9 shows the performance of the achievable data rate of the CCB-OCC scheme according to the distance between the display and the camera. To observe how the characteristics of the receiver affect the communication performance in D2C link, we compared the performance by experimenting with another receiver device, which is a Samsung Galaxy S9 (device B) with same camera parameters with device A. In addition, experiments of transmitting 4 bits per symbol were also performed to observe the communication performance when using different constellations. The minimum distance to capture the entire region of the electronic display from the camera of device A was , and the experiment was conducted by increasing the distance by . When using device A within the range between and , a maximum of and a minimum of were obtained. As can be seen, if the distance between the display and the camera lies between and , no significant difference in data rate is observed. Note that the distance more than will aggravate image distortion and blur, making it more difficult to extract the precise point of channel estimation and decoding from the RGB histogram.
Even with changes of capturing conditions using different receiving devices (devices A and B), similar achievable data rate performance was achieved. Individual receiving device presents a different color value for each RGB channel, but the proposed channel estimation technique can compensate for the change in the received value so that the data can be detected successfully. By using different constellation of 4 bits per symbol, the achievable data rate was enhanced. Although the distance between symbols is slightly reduced compared to constellations using 3 bits per symbol, the high resolution of commercial cameras allows the color barcode to successfully distinguish 16 colors.
As can be seen in Figure 10, we investigated the impact of view angle on the performance of the achievable data rate of the CCB-OCC scheme. Here, the distance parameter was set to . We experimented with varying yaw and pitch angles when capturing the display with a camera device. Table 2 presents the performance of the achievable data rate (ADR) according to various angles. Note that a smaller angle indicates a better alignment between the display and the camera, leading to higher achievable data rate. As the angle between the display and the camera increases, the number of pixels occupied in the color barcode region is reduced. This makes it impossible to take full advantage of the rolling shutter effect within a single image and makes it difficult to obtain an accurate reference value for the RGB component. When data are decoded using an incorrect channel information, the error occurrence rate increases. Therefore, the proposed technique is suitable for short-range communication applications that can fully utilize the rolling shutter effect and limit the alignment angle in the D2C link.
Note that a commercial off-the-shelf PC monitor has a refresh rate, but the latest gaming displays can reach . Pursuing a faster refresh rate is important for games because the electronic display can keep up with the rapid movement of the player. Since the future development trend for monitors will fully support high-resolution and high refresh rate, we believe that the proposed scheme can be adopted in various display-based applications in near future.
5. Conclusions
In this paper, we designed and implemented a robust visual communication system using CCB-OCC in a D2C link. We introduced the complementary color-based modulation concept, where the primary and complementary colors are generated with equal probability to make pilot and data symbols that are imperceptible to the human eye. And new transmission packet design that contains pilot symbols to synchronize and estimate the D2C channel link for calibrating corrupted images caused by different characteristics in the display and camera and capturing environment was proposed. Experimental results show that the CCB-OCC scheme can provide robust achievable data rate performance with regard to sender-receiver distance, view angle, and modulation order. Therefore, we believe that our proposed scheme shows a new possibility in designing a D2C communication system with robust to environmental change, easy-to-use, and simple implementation.
Data Availability
The simulation parameter data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This research was supported by the 2019 Yeungnam University Research Grant.