Abstract
Spectral fingerprinting of chemical indicators, using computer screens as light sources and web cameras as imaging detectors, is an emerging approach for chemical sensing with the potential to coexist in common consumer electronic devices. The migration of this technique to mobile phones is key to extend this sensing approach to the most ubiquitous and familiar type of instrumentation. Here, we investigate the feasibility and performance of spectral fingerprinting on reference samples using a standard mobile phone as a complete measuring platform, where the screen provides controlled illumination while the front camera is the imaging detector. Key elements for the execution of such experiments are the software design, the definition of the sample layout, the type of alignment between the phone and the sample, and the influence of ambient illumination. This paper demonstrates the feasibility of reflectance fingerprinting on standard mobile phones and identify the operating conditions of the key parameters that produce an adequate evaluation performance.
1. Introduction
Chemical sensing based on ubiquitous platforms such as consumer electronic devices is an emerging strategy for decentralized biomedical determinations [1] and (bio)chemical sensing [2, 3].
In this sensing paradigm, the users already own the measuring platforms, which are devices such as DVD units, flat bed scanners, or computers sets, and the detection is made possible through an additional sensing element specially designed to be optically readable with such platforms.
Flat bed scanners [3, 4], DVD and blueray drives [5, 6], computer sets in combination with web cameras (the so called computer screen photo-assisted technique, CSPT [2, 7]), and mobile phones [8] have been demonstrated for different chemical sensing applications.
Different platforms have been exploited for different detection principles. In the case of flat-bed scanners and mobile phones, existing demonstrations are purely colorimetric detections, while blu-ray devices have been used in their three available narrow bands of excitation [6]. The CSPT concept, in particular, has been demonstrated to support diverse optical phenomena, from spectral fingerprinting of the excitation emission matrix of fluorescent indicators [2, 9, 10], and reflectance measurements [11], to label-free detection such as ellipsometric imaging [12] and surface plasmon resonance [13] experiments.
In CSPT, linearly polarized wide band and partially overlapped spectral radiances can be combined to produce varied spectral excitations, which in combination with the partially overlapped wide-band detection supported by the web cameras create multiple bands for spectral analysis [10, 14]. The method builds upon spectral reconstruction strategies [15, 16] and it is adapted for the classification of fluorescent indicators used for chemical sensing [2, 10].
The necessary components to perform CSPT experiments are today present in standard mobile phones, which are perhaps the most attractive platform due to its ubiquity, affordability, and mobility, and in this work, we investigate the performance of CSPT fingerprinting on such devices. We aim at CSPT fingerprinting in the simplest possible configuration, which is just a phone aligned with a sample and the screen programmed as a light source operating in combination with the phone’s front side camera to produce a CSPT measurement.
Here, we examine the limitations and possibilities of CSPT fingerprinting in this configuration, and we address the most relevant issues such as illuminating conditions, sample size, density of sensing and/or reference areas, effect of ambient light interference, alignment conditions, and software design.
2. Experimental
2.1. Setup
The measuring platform in this work is an affordable and standard mobile phone Nokia 6220 Classic, which provides the controlled light source and the imaging and acquisition capabilities for CSPT experiments.
The screen is 54 mm diagonal, 24 bits color resolution TFT display with QVGA format (320 × 240 pixels). Figure 1(a) shows the spectral radiance for 7 colors displayed on the screen of a regular LCD screen (Dell Latitude D800) commonly used for CSPT experiments, while Figure 1(b) corresponds to the mobile phone screen, both operating in standard conditions of contrast, intensity, gamma, and color settings.
(a)
(b)
The spectral radiances are measured with a 12 bits resolution fiber optics spectrophotometer (Ocean Optics USB 2000) pointing directly to the screen and set to identical configuration, while they display the 7 considered colors.
The displayed colors can be specified by triplets of weights for the red, green, and blue channels of the screen. The triplets used to define the illuminating colors in this work are: red (100), magenta (101), blue (001), cyan (011), green (010), yellow (110), and white (111), where 1 indicates the maximum intensity.
The perception of these same colors can be induced by an infinite number of different spectral radiances, whenever they produce the same intensity under the integrating bands of the detector. The spectral radiances of the Nokia 6220 Classic, although different in shape, have equivalent intensity to the LCD screen and cover the spectral range required by the CSPT experiments.
For image detection, the Nokia 6220 counts with two imaging devices, a main 5 megapixels camera in the back side of the device, and a CIF camera (352 × 288 pixels, used as QCIF 176 × 144 pixels video) on the upper left corner of the screen, which is aimed to operate during video calls.
For colorimetric determinations in ambient light it is certain that, the back-side camera is the best choice, however, in the CSPT case, which utilizes the screen as a light source, the front side camera is the most conveniently located since it can produce the CSPT measurement without any additional sample holder or light deflector.
The considered setup is just a mobile phone held by hand with its screen pointing toward a sensing device and running a software that changes the colors in the screen while simultaneously capturing the image of the sample.
In this work, the sample is a color array that can be reproduced with the RGB specification of the printed colors. Instead of a chemically sensing array, a printed array enables to disentangle the performance of the measuring setup from the response of eventual sensing elements, thus allowing a more accurate assessment of the platform alone.
In order to systematically study the measuring configuration, we used a simple mechanical stand that reproduces the considered measuring conditions and allows controlling the screen to sample distance (z in Figure 2(a)) as well as the alignment between the sample and the phone’s screen.
(a)
(b)
Since the screen and camera position are not symmetric (Figure 2(a)), the conditions for homogeneous illumination do not coincide with the best conditions for detection, a situation that becomes stressed for shorter z distances, where the illumination is, however, more intense. Accordingly, a compromise between these different variables must be reached.
We compare in this work two possible alignments. In one case, the image of the sample is centered in the captured image (image centering, IC), while in the second case the sample is centered on the screen, in the captured images. Since this alignment varies for different distances, we call this alternative dynamic centering (DC).
The sample in CSPT spectral fingerprinting experiments is typically a collection of chemical indicators that change their spectral properties upon exposure to a target environment [2]. The measurement can evaluate transmitted [2], emitted [2, 7, 10], or reflected light [11], and in the present case, it is reflected light the most convenient for the chosen configuration.
It is the ability to discriminate, in this case spectral reflectances from the sample, which is used for the evaluation in this kind of experiment. Similar to the visual inspection of quick tests, where the reference concentrations, represented by colors in a label are compared with the color developed by a test strip [11] upon exposure, CSPT can perform these evaluations automatically. Thus, the ability of this platform to discriminate different spectral reflectances, is what determines the chemical sensing capabilities of the system and is what we investigate in the present study. Accordingly, the sample is a printed layout displaying the largest possible range of colors that can be found in chemically sensitive reflectometric strips. This includes pure colors (maximum saturation) covering the hues of the visible spectrum compatible with CSPT [17], and the same colors at decreasing saturation and at decreasing lightness, such as described by the HSV color model [18].
A main concurrent factor in the design of a sample layout is its physical size and the number or sensing and/or reference areas that can be incorporated in the device. In conventional CSPT platforms, using microstructured layouts hundreds of areas down to 100 m in diameter can be visualized [19, 20]. In the present simpler setup, we test the limits of the camera resolution and illumination capabilities with a sample layout that incorporate 36 regions (2.5×2.5 mm regions in a 66 ROI array) in a 15 × 15 mm2 total area (Figure 2(a)). Images of the sample area captured for IC and DC alignments are shown in Figure 2(b) and discussed in the next section.
2.2. Software and Processing
The Nokia 6220 Classic is an S-60 platform on a Symbian OS, which is compatible with Java ME MIDP 2.0.
The software for controlling the illuminating sequence (sequence of colors displayed on the screen) and the acquisition was written in Java ME [21] in Net Beans IDE, an open-source integrated development environment.
Displaying color sequences on the mobile screen can be implemented with standard functions on Java ME; however, even basic camera control, such as shooting snapshots in automatic exposure mode, requires extended libraries such as JSR-234 [22]. JSR-234 prompts the execution to inform about the use of the library and require a Yes/No input to proceed, thus making it unsuitable with the automatic acquisition of multiple images.
The alternative solution we implemented requires a single prompt, which launches a fixed length video acquisition, and runs in the background of a canvas that displays the illuminating sequence. The result of the measurement is a single 176 × 144 pixels frame, H.263 compressed 3gp file acquired at 15 fps. This less than 400 kb data set is suitable with transmission to a remote site for data evaluation of the type describe below, and does not require to rely on the mobile phone itself for the processing.
The ability to control the camera in Java ME varies between different brands of mobile phones and it is not directly linked to the category of the devices.
Data analysis is carried out in Matlab 2008b, for which we previously convert the acquired video file into a 176 144 pixels frames, 1fps uncompressed avi file using Mpeg-Stream 1.9.2 for Macintosh.
The avi files are loaded in Matlab for analysis of illumination profiles and gridding procedures defining regions of interest (ROI), which are used to compose spectral fingerprints of the ROIs along the illuminating sequence [2, 8, 11, 13] and to classify the fingerprints using principal components analysis [23] (Statistics Toolbox).
3. Results and Discussion
In the ubiquitous sensing paradigm the chemical sensing functions should coexist with the original purpose of the platform, and hence permanent modifications of the devices are not acceptable. Accordingly, the whole measurement strategy must be adapted to negotiate these requirements.
An example of this type of approach is the incorporation of references embedded within the sensor, so that a calibration is automatically performed for each measurement [11], rather than assuming that the CSPT platforms are stable enough to operate between periodic calibrations. The evaluation of this type of tests consists of the automatic comparison of sensing areas and reference areas.
Other sensing systems incorporate calibrating solutions that run in parallel with the evaluation of the target sample [8].
Beyond certain level of control, ubiquitous platforms demand dedicated drivers or more invasive hardware access [5] to perform a measurement, which makes them less appealing, since this restricts the coexistence with the original device and limits the universe of compatible platforms. Thus, it is important to characterize these techniques for a realistic level of control rather than for highly dedicated configurations. For this reason, we developed software for acquisition that operates in automatic exposure mode, rather than the more analytic manual exposure condition or in Bayer mode (nonprocessed images acquired directly from the image sensor readout). This choice makes the software more cross-platform compatible, essentially suitable with any mobile phone capable of Java ME and JSR-234, and simplifies the user interaction, since is not necessary to configure the camera exposure.
A key aspect of CSPT fingerprinting, which is the use of controlled illumination, is not problematic, since by design displays are made to deliver arbitrary contents. Thus, screens can be used to produce 2D micropositioned and pulsed light sources [24], or to illuminate with varied spectral radiances, such as in the present case (Figure 1(b)).
As mentioned above, it is critical to determine if CSPT fingerprinting can operate in the conditions proposed here, since these are those truly compatible with co-existence on mobile phones.
In order to study the effect of the distance between the sample and the phone, its alignment and the effect of ambient illumination, the phone and sample layout are held at controlled distances.
Figure 2(b) shows two images of a 15 × 15 mm2 area (same as sample layout) under IC and DC alignments using snapshots and video acquisition. In order to corroborate the alignment and determine the viewing angle of our setup, we image in this case the 15 × 15 mm2 area with a mirror in the background.
It is possible to observe the reduced field of view of the video acquisition (dashed squares in Figure 2(b)) compared with the snapshot mode. Characterizing these conditions is central to find the best configuration for the CSPT measurement on mobile phones.
Figure 3(a) collects the intensity profiles along the line aa’, indicated in the inset, for seven different z distances from 55 to 175 mm. DC and IC alignments are indicated in red and green, respectively, and the measurements are taken within a light shield, which restricts the illumination to the phone’s screen.
(a)
(b)
(c)
(d)
The length of these profiles (Figure 3(a)) grows for decreasing z, since the sample area occupies more of the image at a closer distance to the camera. It can also be observed that in IC the maximum intensity do not occur for the shortest z, but, however, at 115 mm. This is the result of the asymmetric distribution of screen and detector respect to the camera (which DC partially compensates), and due to the automatic exposure used for the image acquisition. In order to center the sample area within the captured image, in the IC alignment, the sample area must be displaced away from the illuminating region, and thus, not only the maximum intensity decreases but also the intensity distribution deteriorates, which shows a 11% variation for z 55 mm. In the DC case, the sample area is always properly illuminated, but at shorter distances part of the sample area exits the viewing field and a larger portion of poorly illuminated background enters the composition, which naturally affects the metering used by the automatic exposure.
Differences in overall intensities in IC and DC, are consequences of sample area centering, since in IC it dominated the metering used for automatic exposure, while in DC the sample area is not vertically centered in the image, in order to stay aligned with the illumination, and of the black background used in these experiment, which induce to boost the exposure.
Figure 3(b) corresponds to intensity profiles along bb’. In this direction the worsening illumination in IC, for decreasing , is more evident and reaches up to 30% variation for mm. The DC profiles are more stable and robust to different , while the intensity levels are the same as in Figure 3(a), since the metering for the automatic exposure is made over the area of the image.
A related aspect in the performance of the detection, which becomes embedded in the present characterization, is the contribution of chromatic aberrations and vignetting effects that are expected to be more serious at the lens-image edges. The analyzed measuring conditions reflected in the profiles in Figure 3 account for the vignetting effect, whereas chromatic aberrations can be controlled by averaging within ROI which avoid sample edges.
Besides alignment, a main concomitant factor for practical uses is the influence of ambient illumination. Regular CSPT measurements take place within a light shield, that restrict all illumination to that provided by the screen, however, for a system such as that proposed here, which lacks a sample holder or light shield, the alternative is to measure in a darkened room, and may not offer the same level of isolation.
Figures 3(c) and 3(d) are the same type of characterization previously discussed but with the incorporation of ambient illumination (600 lm/m2, provided by 2 Philips TLD 36W/33, about 10 : 1 compared with the ~300 nits of the screen). It can be observed that ambient illumination partially correct the variation of IC intensity profiles, especially in the bb’ direction (from 30% to 5% variation), and also equalize the intensity levels for different . The overall intensity levels remain comparable to those without ambient light.
In summary, the DC shows a more homogeneous profile and a more robust behavior with ambient light, although for z < 75 mm the intensity levels are very close to saturation (255 counts), and the effect on the discrimination ability is yet to be determined.
The final boundary condition we need to incorporate in this analysis, and which in part has defined the range of the studied z distances, is the size of the sample area and the maximum number of different subregions that can be identified and spectrally fingerprinted within such area.
We work with a 15 × 15 mm2 total sample area, which fits, with some margin, within most phone displays. Larger sample areas can certainly receive proper illumination using larger displays or increasing the distance form the screen, but that also limits the universe of potential CSPT platforms and it moves beyond standard mobile phones.
For a given total area and measuring distance, the minimum number of pixels that produce a meaningful spectral fingerprint of each sub-region limits the number of sub-regions. Thus, for the defined sample area to be evaluated at mm, which provides an acceptable compromise between illumination and detection quality, corresponds to 36 regions, each of them represented by 25 pixels in the acquired image.
The sample layout used to determine the spectral fingerprints in these conditions is detailed in Figure 2(a) and it covers the hues of the visible spectrum for saturated, unsaturated and reduced lightness tones, as described in the experimental section.
Figure 4(a) shows the sample layout for a particular illuminating color, while the overlying white squares indicate the location of the selected ROIs. A Matlab program collects, for each pixel within each ROI, the intensity variations under different illuminations. The total information captured in the measurement is presented in the figure on the right, where the video stream is rearranged as a collection of color pictures for each ROI under different illuminations. Since the video acquisition rate (15 fps) is larger than the color changing (illumination) rate, several frames are captured for each illuminating color. Variations in the captured intensity for each constant illuminating color are due to the response time of the automatic exposure control, and in this way these dynamic range variations are incorporated in the signature of each ROI.
(a)
(b) IC.
(c) IC + ambient light.
(d) DC.
(e) DC + ambient light.
Since the image in Figure 4(a) is a color image, the information is naturally unfolded in the red, green, and blue camera bands, so three intensity signatures are produced for each pixel. The CSPT fingerprints are then the concatenation of these red, green and blue intensity signatures (IR(i), IG(i), IB(i)) of each pixel where is the spectral response of the detector, , and are the spectral windows of the red, green, and blue camera channels and is the index of the frame, which is correlated with each spectral radiance emitted by the screen for a particular illuminating color specified by the triplet of weights (, , ) and for a given screen gamma ().
In order to compare the fingerprints, we use principal components (PC) analysis [23], a multivariate analysis technique that enables to transform each of these fingerprints into scores in a two dimensional PCs space that explains more than 90% of the original information. The separation between these scores and the way they cluster together is indicative of differences and similitudes captured by the fingerprints.
Figure 4(b) shows the score plots for the IC conditions without ambient illumination. All pixels involved in each ROI are classified (instead of using the average value of each ROI), in order to observe the dispersion of fingerprints that correspond to a same sub-region, which provides an idea of the accuracy of the classification. Dispersed clusters have more chances to overlap a neighbor cluster, deteriorating the ability to distinguish different categories. Scores corresponding to a same ROI are plotted with the same color (that corresponding to the color of the region in the sample layout) and surrounded by black circles. For a blue ROI the 25 corresponding scores are shown magnified to illustrate the dispersion within a cluster.
Figure 4(b) shows a good separation between different colors, which naturally deteriorates toward less saturated tones (in the limit of minimum saturation are all white), and similarly toward reduced lightness (black in the extreme condition). Moderately unsaturated tones, respond worse than moderately darkened tones, as can be observed by the closer distances between clusters, with even one overlap in the indicated clusters (1 in Figure 4(b)) whereas there are not overlaps in the moderately darkened ROIs.
For strong unsaturation there is a smaller spatial variability than for strong darkening as can be explained by the equalizing effect of high intensity values, close to maximum intensity, compared to the poorer signal-to-noise ratios at low intensities.
Figure 4(c) shows the PC plot for the IC alignment under additional ambient illumination. Different pure colors are still well separated, but changes in saturation affect the ability to separate categories that were previously well identified (2 in Figure 4(c)). The same trend of better performance for darker tones than unsaturated colors is observed for the condition with ambient light. Again, white regions are less dispersed than black ones.
In the case of DC without ambient light, different colors are well separated (Figure 4(d)), and there is a larger weight of the PC2 coordinate (28%) compared with IC (18% and 12%). Pure tones are further apart also in the PC1, while for the IC case there is a more clear identification of hue (in PC2) perpendicular to intensity (in PC1).
There are both spreads in dark and unsaturated regions, but these scores are nevertheless well separated from the rest of the regions.
Finally, the separation between different regions severely deteriorates in the case of DC with ambient light (Figure 4(e)). The reason is intensity saturation due to overexposure, and the consequence is the lost of discrimination except for extremes of the range, which is mainly an identification of intensity.Overall, the results show that CSPT fingerprinting experiments can be carried out on a standard mobile phone in the most demanding conditions for the measurement, which actually are those that simplify the operation (automatic exposure, without sample holder, front side camera in video mode, etc.) and consequently maximize the number of compatible platforms. As mentioned before, not all phone brands fully support camera control in Java, although Java compatibility is expanding and becoming ubiquitous. Regarding physical variations between mobile phones, one can argue that it will affect CSPT determinations in the same way as variation in conventional CSPT platforms. However, it is conceivable to expect that the measurement of embedded references used in classical CSPT to counteract this aspect [11] will also work for mobile phones, since both systems share the same principles. In concrete, this work shows that at least it is possible to perform CSPT fingerprinting with one type of phone, which constitutes a first step towards the development of the concept.
The characterizations performed in this work, show that the DC alignment is superior if the measurement can be realized in a dark room, which is perhaps not a severe limitation, especially if measurement times can be made as short as a couple of seconds. In this work, we did not care about a short acquisition time, but for an illuminating sequence of 7 colors displayed at 5 fps (standard mobile phones can display video at 15 fps), with a video acquisition at 15 fps (67 ms between frames) the total measuring time could be as short as 1.4 s.
With the presence of ambient light the advantages of DC alignment vanish. The reason is mainly intensity saturation due to the lack of control of camera exposure. A more advanced control of the camera exposure is perhaps the most tempting alternative, however, as discussed previously that would also limit compatibility across different platforms. Thus, it is perhaps a restriction in the ambient light and a modulation of the screen illumination (e.g. color distribution over the screen) the most convenient alternative. At the present stage, if the measurement cannot be performed in dankness the DC alignment at mm offers a viable alternative and an acceptable performance.
4. Conclusions
It has been demonstrated that an average mobile phone can serve as a CSPT platform for spectral fingerprinting, while operating within the most generic Java resources.
The positioning of a small-sample area respect to the screen and mobile phone camera has been studied, showing different viable alternatives for measurements in darkness and with additional ambient illumination.
Up to 0.16 regions/mm2 can be evaluated in the studied conditions, for a total sample area of 15 × 15 mm2, a dimension compatible with evaluation on most standard mobile phones.
Spectral fingerprinting with conventional CSPT platforms have been used for diverse chemical sensing applications including disposable gas-sensing arrays [2], odor sensing for food monitoring [25], the evaluation of ELISA tests for diagnostics [26], and the evaluation of commercial urine tests [11], among others. The spectral fingerprinting capabilities demonstrated here is an essential first step for the migration of these applications to mobile phones.