Abstract
With the development of iris biometrics, the pupil recognition method is widely used in the fields of identity recognition and so on. Most traditional iris recognition algorithms use an adaptive threshold to the eye image binarization processing; although this can at the maximum preserve the original characteristics of the image itself, the retained features in this method included a lot of noise points, including Gaussian noise and so on, in a series of image preprocessing. The subsequent iris recognition and pupil positioning are still prone to the loss of positioning accuracy, which cannot separate the pupil completely from the background area of the image, increasing the difficulty of connecting the pupil to domain localization, thus affecting the pupil positioning accuracy. Therefore, in order to meet the requirement of high precision pupil recognition accuracy in low-quality eye images, this paper takes the iris dataset provided by the China Research Institute of Automation as an example to improve the traditional pupil location method. A pupil localization method based on parameter optimization is proposed, which includes the setting of a three-step threshold, called as TST.
1. Introduction
With the innovation and iteration of iris biometrics, iris recognition technology has gradually replaced the traditional identification method. Traditional identification solves the problem of identity identification with regard to the possession of objects through the possession of these external objects to prove the person’s identity. However, for keys, ID cards, bank cards, and other external objects, they can be easily stolen by others who can use these items to obtain benefits. Biometric identification technology [1] is based on automatic identification technology of personal unique physiological or behavioral characteristics. The biological characteristic is a more reliable and convenient identification method than traditional identification methods. It will not be lost or forgotten; relative to the straightforward replication of the characteristics of fingerprint recognition, the iris and pupil are not easily retained after being used. Therefore, the iris and pupil features of humans are not easily copied and used by others. At the same time, due to the innate differences between human individuals [2] and the differences in living environment between different individuals, the physiological structures of different individuals have obvious differences over time. In addition, the human iris and pupil are highly stable in biological morphology [3], so they can be used for individual identification [4]. Reliable iris recognition depends on accurate pupil location, so pupil recognition is of great practical significance. However, there are also some difficulties in pupil localization. For example, in the process of obtaining the pupil image dataset, changing the lighting conditions, eyelashes and pupil images, glasses, and contact lenses will produce a large number of different forms of reflection, as well as interference when the pupil is off-center or on the edge of the image, and blurred pupil due to blinking. These disturbances make pupil localization under nonideal conditions a practical and challenging challenge.
Traditional pupil localization methods are mainly divided into threshold and Hough transform methods [5]. The basic idea of the pupil localization algorithm based on the threshold is to calculate binarization parameters according to sliding window, then compare the gray threshold with each pixel value in the gray image and classify each pixel value into the appropriate category. The primary threshold selection methods include the adaptive threshold method [6] and the global threshold method, among which the adaptive threshold method adopts a sliding window. By sliding the sliding window in the image, the corresponding threshold of each region is obtained, and the image binarization is carried out. This method can retain the original texture features in the image. However, it also retains details other than the pupil in the image, which significantly increases the difficulty of extracting the contour. Therefore, the pupil localization accuracy is low in nonideal cases. The global threshold method mainly compares the pixel value in the whole image by setting a fixed threshold value. It sets the pixel value more significant than the fixed threshold value as 255 and the pixel value less than the fixed threshold value as 0. Standard representative algorithms include the OTSU (OTSU method) and iterative methods. However, selecting a global threshold in the global threshold method is challenging to some extent. When the threshold is large, the whole eye region in the eye image may be classified as the target region, but the pupil and background cannot be separated accurately. When the threshold value is small, the pupil area may be incomplete so that the pupil cannot be accurately located.
To overcome the above problems, this paper proposed a robust pupil localization via a novel parameter optimization strategy called TST, which has already been described in [7]. And in order to describe TST in more detail, this paper added more details and designed more experiments for verification.
The main work of this paper is as follows: (1)A new threshold adaptive pupil localization method is proposed, which shifts the focus of pupil localization to the acquisition of binarization parameters, thus simplifying the subsequent screening of pupil contour point sets and ultimately improving the accuracy of pupil localization(2)In order to solve the small gray value difference between the pupil and the iris, which prevent us from using methods such as the histogram to obtain more accurate pupil edge gray information, a three-step threshold method is proposed, by positioning the pupil edge gray-level information within and outside the edge of the gray-level information, with the method of weighting parameters, for the edge of the pupil parameter information. In this way, the interference of external background information is removed. In contrast, the complete pupil information is retained, which ultimately improves the robustness of pupil location(3)A method for pupil contour recognition is proposed, which introduces screening criteria such as pupil area ratio, length-to-width ratio of the whole image, and Euclidean distance between the pupil edge and the center coordinate to simplify the pupil contour point set identification difficulty and increase the accuracy of pupil recognition
The rest of this paper is organized as follows: In Section 2, this paper will outline the algorithms of pupil recognition. Section 3 introduces the basic principles and steps of TST. Section 4 presents experiments on three datasets and uses the results to verify our algorithm. Section 5 discusses the performance of TST algorithm. Finally, Section 6 is the conclusion of this article.
2. Related Work
Pupil recognition is very similar to iris recognition. Daugman [4] first proposed a method for locating the inner edge of the iris based on differential operators. Then, Wildes [2] proposed to decompose the eye image into four regions of different degrees and calculate the similarity to achieve iris recognition. On the basis of the above two methods, a variety of similar pupil localization methods have been extended. Zhang et al. [8] improved the calculus operator proposed by Daugman and proposed to use polar coordinate transformation to represent the pupil area by using a matrix in the iris region in the original image while adding Laplacian noise to the iris and pupil region in the image. Then, a deidentification algorithm is used to identify the biometrics of the iris and pupil, achieving a similar pupil recognition effect. However, this method focuses on pupil privacy protection and biometric recognition of the iris and pupil and cannot accurately locate the pupil and iris region. By analyzing calculus operators, Minakova and Petrov [9] proposed an operator based on the Bresenham algorithm to integrate partial calculation methods to improve efficiency.
Meanwhile, the Bresenham algorithm was modified to calculate operators in a single arc. This method improved the speed and accuracy of pupil positioning. However, for the eye image with no apparent difference between the target and background, the segmentation parameters cannot be calculated accurately, resulting in a large error in pupil location. In addition, Chen et al. [10] and Setiawan et al. [11] improved the traditional Hough transform circle detection method by introducing the Canny edge detection operator and proposed a pupil localization method based on the circular Hough transform. This method can achieve relatively accurate localization of artificially blocked pupils in high-quality images. And Shang et al. [12] also built a pupil localization system based on this method. However, in low-quality eye images, due to the Hough transform circle detection method, there may be multiple round areas similar to pupils in the image, increasing pupil positioning error. In order to solve the problem that pupil localization is prone to failure in the case of off-axis occlusion, Dewi et al. [13] proposed an ellipse-fitting and a fine-adjustment algorithm for robust pupil localization in off-axis conditions. The accuracy of this method was 0.83 when the -value was 41.5. To improve the efficiency of pupil positioning, Li et al. [14] and Jan et al. [15] proposed to adopt the grayscale integral projection method for pupil localization. However, these methods improve the speed and reduce the accuracy.
In addition, in the practical application of pupil localization, the acquisition of the eye image is not smooth. The obtained eye image also has a lot of noise, such as pupil and iris occlusion due to strong light. These low-quality eye images will directly lead to the decrease of the pupil localization accuracy of the above methods. For this reason, Fuhl et al. [16] proposed a histogram-based pupil localization algorithm, which improved the robustness of pupil localization by combining edge filtering with the angle integral projection function. Fusek and Dobeš [17] proposed to use a self-organizing migrating algorithm (SOMA) to determine the correct shape and position of the pupil model by considering the physiological characteristics of the eyes. This method improves the speed of pupil recognition while ensuring high accuracy. Jamaludin et al. [18] proposed a deblurring method based on the Wiener filter to improve the quality of iris pattern and achieve pupil positioning. Later, literature [19] proposed using the geodesic distance to locate the pupil. This method achieves pupil location by calculating the geodesic distance of the four corners of the image, which has good stability in complex images.
In recent years, with the integration of deep learning and image segmentation, relevant scholars have also tried to use a neural network to achieve pupil location. The precision of pupil and iris segmentation can be improved by the high-precision calculation of computers. Jalilian and Uhl [20] proposed a fully connected coding network for iris segmentation. Yang et al. [21] and Choi et al. [22] proposed a network model combining fully connected convolutional networks and cavity convolution to segment the iris and achieved good experimental results. Choi et al. [23] proposed adopting a heterogeneous CNN model for pupil localization, which also achieved high recognition accuracy in specified datasets. In order to improve the accuracy of iris recognition again, Lee et al. [24] proposed to use an adversarial network for data enhancement, thus achieving more accurate pupil recognition.
However, the pupil localization method based on a convolutional neural network needs to learn specific data features. It is not easy to achieve target localization for the target features that have never been learned. In this paper, the pupil location method based on the global threshold method is improved, and a robust localization via a novel parameter optimization strategy is proposed, which includes a three-step parameter optimization, referred to as TST. The algorithm is used to accurately extract the target area and achieve complete separation of the pupil from the background, as shown in Figure 1. In biometrics, the pupil and iris are clearly demarcated, as shown in Figure 2. It can be seen from Figure 2 that the pupil region represented by the red ellipse is significantly different from the iris region represented by the yellow ellipse. There is also a significant difference between the skin area outside the orange ellipse and the iris area.
Therefore, we design a binary parameter search algorithm with pupil edge gray value jump. The initial search coordinates are determined by setting the region of interest. The gray parameters of the outer and inner edges of the pupil were obtained to ensure the integrity of pupil image information. Then, linear interpolation was used to narrow the difference between inner and outer edges, and the binarization parameters suitable for pupil separation were obtained.
3. Methodology
This section improves the traditional threshold pupil localization algorithm and proposes a method of pupil localization via a novel parameter optimization to locate the pupil in eye images. After preprocessing the image, we first proposed a binarization parameter acquisition method based on a three-step threshold to obtain accurate pupil boundary segmentation parameters. The method mainly includes an -nearest neighbor domain search, binarization parameter optimization, and boundary value. Secondly, we designed a screening method suitable for the pupil contour point set to increase the accuracy of pupil location. Finally, we combine the binary parameter acquisition method based on a three-step threshold with the pupil contour point set screening method to achieve accurate pupil location.
3.1. Image Preprocessing
Since there may be RGB images in the images we process, we adopt a weighted average method to grayscale the images, and the formula is as follows:
where is the final pixel value; , , and are the pixel values of the corresponding channels; is the coordinates of row and column in the image; and , , and are the weight parameters. After graying the image, there is still a lot of abrupt point noise in the dataset. In order to reduce the influence of such noise on pupil positioning, the algorithm adopts Gaussian filtering to denoise the image, and the formula is as follows:
where is the standard deviation and is the pixel value.
3.2. ROI Region Definition
In biological morphology, the ratio is between 4/1 and 3/1 for the iris to the pupil. Meanwhile, due to the acquisition of the dataset including some face regions such as eyelids and eyebrows, we define the initial ROI region as 1/4 of the image center, as shown in Algorithm 1.
| ||||||||||||||||
where is the center point of the original image; is the high range of ROI; is the wide range of ROI; and are the coordinates of the four vertices of ROI as shown in Figure 3.
3.3. -Nearest Neighbor Domain Search
Unlike the gray-level histogram method and iteration method, TST searches for a way to determine the final binarization parameters; in a more sophisticated image, simply using a histogram or iterative method to get the threshold, the pupil cannot be separated from the external area, which makes it hard for the pupil and eyelash, affecting the follow-up positioning accuracy of the pupils. The near- neighborhood search can accurately determine the gray parameters of the inner edge of the pupil to separate the pupil from the image background. This section will introduce the near- neighborhood search method in detail, as shown in Algorithm 2. The steps are as follows: (1)Take the minimum coordinate point of the gray value in the ROI area as the initial search coordinate, set the search step, and start the search with the initial search coordinate, as shown in Figure 4.
| ||||||||||||||||||||||||||||||||||||||||||
In the figure, is the initial search coordinate, is the search step size, and the default value is 20. (2)In the rough positioning of the pupil edge, there are some sharp points in the image. When the search values in the eight directions of this point are equal, the rated step size will be automatically increased, and the calculation formula is as follows:
where is the extra step size, is the number of exceptions that occurred, and and are the search values returned. (3)In the search process, when all search directions jump simultaneously, the minimum value among them is taken as the new starting point, and its calculation formula is as follows:
where is the new starting coordinate. , , , , , , , and are the dimensionless pixel values to the right of the starting search point. is the lower-left pixel value of the starting search point (dimensionless). and are the gray values at the lower right (dimensionless). The is the generating function of the horizontal and vertical coordinates; (4)Set the critical condition of the gray value. The formula is as follows:
After all the were obtained, the minimum search value was taken as the internal pupil edge separation parameter, and the formula is as follows:
where is the returned search value, is the gray value of the next step, is the current gray value of the search, is the critical condition and the default value is 1.5, and is the minimum binarization parameter value of pupil separation parameter and it is dimensionless. is the minimum value function; is the maximum value function.
3.4. Optimization of Binarization Parameters
The -nearest neighbor domain search algorithm obtained the pupil edge’s approximate inner boundary gray value. However, due to the high exposure or multiple light source points in the acquisition process of some eye images, there may be reflected light spots inside the pupil as the gray value of the pupil differs significantly from that of the reflected light spot. As a result, the binarization parameters obtained by near- neighborhood search do not contain the complete pupil range. Therefore, it is necessary to optimize the search of binarization parameters based on the -nearest neighbor domain search to obtain the pupil outer edge parameters. The basic idea is to realize the extreme marginalization of binary parameters by reducing the search step size and changing the judging boundary conditions. The idea is shown in the algorithm, and the main steps are as follows: (1)Through the last section of the nearly neighborhood search method, we get the initial coordinate as starting points in eight directions of gray value jumps of the coordinates of the point, because the eight coordinates of the gray value of the step before leaping the gray value, so they represent the pupil in the edge location of the binary parameter values; as a parameter for image binarization, part of the image of the pupil will likely be lost. In order to ensure the integrity of the pupil, we reset these eight coordinates as the initial coordinate points(2)In the-nearest neighbor domain search method, based on the initial coordinates of the starting point, eight of them point the direction to search, and in the process of binary parameter optimization, because the pupil within the gray value range has been confirmed, the search algorithm is no longer needed for eight directions at the same time; it is only needed according to the coordinate point in the nearlyneighborhood search process which is the returned direction search. Its basic formula is
where is the gray value after the binarization parameters are refined on the basis of the inner pupil edge parameters, is the gray value corresponding to the current coordinate in the binarization optimization search algorithm, is the gray value corresponding to the last coordinate, and is the new jump criterion with an initial value of 2. In the process of optimizing the search by the above binary parameters, the initial value of the search step is set as 1 in order to prevent the gray value of the pupil and iris in the eye image from increasing in sequence due to the dark light, thus affecting the judgment of the jump limit. (3)A relatively delicate pupil edge parameter is obtained after the binarization parameter optimization of the gray value of the inner pupil edge obtained by the near- neighborhood search method. The minimum value of the returned optimization parameters in eight directions is taken as the final value of binarization parameter optimization, and the formula is as follows:
where is the binarization parameters of the eight directions after the parameters are defined by the method of binarization parameter optimization based on near- neighborhood search. is the gray value of the outer pupil edge selected after binarization parameter optimization.
| ||||||||||||||||||||||||||||||||||||||||
3.5. Improved Linear Interpolation Method
The inner margin and outer margin of the pupil and the method of acquisition obtained a rough location of the inner and outer pupil edges. However, when there is no obvious jump boundary between the gray values of the pupil and iris, the rough parameters of the outer edge of the pupil obtained by binarization parameter optimization will be pretty inaccurate. In this case, binarization of the image will lead to the integration of the pupil with the external background, which will make it challenging to locate the contour-point set of the connected domain of the pupil in the follow-up, thus making it unable to achieve accurate positioning of the pupil, as shown in Figure 5.
In order to eliminate the influence of no apparent difference between the outside edge of the pupil and the gray background value and solve the problem of the pupil and the outside background being mixed, we put forward the boundary value method to reduce the error caused by binarization parameter optimization. The main idea is linear interpolation. By reducing the weight of the parameters on the outer edge of the pupil and increasing the weight of the parameters on the inner edge of the pupil, the grayscale difference between the inner and outer edges can be well reduced to obtain the actual grayscale parameters of the pupil, as shown in Algorithm 4. Its calculation steps are as follows: (1)Set the upper limit range of binarization parameters and determine whether to use weight parameters to reduce parameter errors by judging whether the gray value parameters of the outer edge of the pupil optimized by binarization parameters are in the upper limit range(2)The binarization parameters of the image are determined by the set constraints
where is the parameter for binarization of the eye image determined by the three-step threshold method, is the gray value parameter of the inner edge of pupil, is the gray value parameter of the outer edge of pupil, and is a constraint on the maximum number of arguments in the bound value. The initial value is 80.
| ||||||||||||||||||||
3.6. Connected Domain Filtering
After binarization of the image with the parameters obtained by the three-step threshold method, no matter the original multilight source points in the image, or the occlusions such as eyelashes and eye sockets, there are apparent segmentation boundaries with the pupil, as shown in Figure 6.
In order to accurately identify the pupil-connected domain, we first introduce the area proportion constraint of the outer tangent rectangle of the pupil-connected domain, whose calculation formula is as follows:
where is the area of the connected domain tangent rectangle, is the rightmost abscissa value of the tangent rectangle, is the leftmost abscissa value of the tangent rectangle, is the bottommost ordinate value of the tangent rectangle, and is the uppermost ordinate value of the tangent rectangle. Secondly, the average Euclidean distance constraint between the contour point set and the center coordinate is introduced, and its calculation formula is
where is the average Euclidean distance of the ith contour point set, is the number of pixels, is the pixel coordinates of the contour points, and is the center coordinates of the eye image. Finally, the empirical value of the number range of pupil contour points is added to calculate the pupil-connected domain satisfying the conditions, and the formula is as follows:
where is the contour point set of the pupil and is the number of pixels in the contour point set.
3.7. Least Square Ellipse Fitting
In morphology, the morphological characteristics of the pupil are similar to those of the ellipse, and some pupils may be close to the circle. Therefore, we adopt the ellipse-fitting method of the least square method to fit the pupil and achieve the positioning of the pupil. The general formula is as follows:
According to the general equation of an ellipse, solving the equation requires at least five measuring points on the ellipse contour. By randomly selecting five-coordinate points from the contour point set in the pupil-connected domain and putting them into the equation for solving, the final ellipse-fitting equation can be obtained to achieve pupil location.
4. Experiments
4.1. Datasets and Metrics
4.1.1. Datasets
The CASIA-IrisV1 dataset is the earliest version of the dataset provided by the Chinese Institute of Automation [25]. The dataset contained 756 high-quality eye images, which were enhanced to distinguish the pupil from its background area. Meanwhile, in the process of data collection of the CASIA-IrisV1 version, the collected personnel are strictly required not to wear glasses and keep their eyes as wide as possible, so that the pupil area will not be blocked by eyelashes, hair, and other organs, which indirectly simplifies the pupil identification difficulty of the CASIA-IrisV1 dataset. In order to verify the high accuracy of TST in high-quality eye images and prevent the algorithm from unilaterally adapting to pupil localization in complex images, but with the accuracy decreasing in high-quality images, this paper first adopts the CASIA-IrisV1 dataset for verification.
The CASIA-Irisv4 dataset is also a dataset provided by the Chinese Institute of Automation. The dataset contains iris images from more than 1800 real objects and 1000 virtual objects. All iris images are 8-bit gray files collected or synthesized under near-infrared illumination. At the same time, the CASIA-IrisV4 dataset has a diversity of eye images. For example, in Interval, a circular NIR LED array was designed to allow the iris camera to capture very clear iris images. In Lamp, elastic deformation of the iris texture was induced by light reaction, and the pupil expanded and contracted correspondingly under different light conditions. In addition, the CASIA-IrisV4 dataset also includes glasses-wearing, composite images, and iris images with specular reflection due to glasses-wearing. Therefore, this dataset can well verify the accuracy and robustness of the algorithm in complex iris image localization. In order to verify the validity of TST, the CASIA-IrisV4 dataset was also used.
The CASIA-Iris-Aging dataset is a dataset provided by the Chinese Institute of Automation. In the CASIA-IrisV1 and CASIA-IrisV4 datasets, eye images with multiple light sources or high exposure rarely exist. In order to verify the practical effect of TST in such eye images, this dataset is introduced in this paper. In this dataset, images with pupils obscured by eyelashes or hair and images with multiple light points or high exposure are included.
4.1.2. Metrics
In order to obtain the specific pixel-level error, the improved mean error is used as the evaluation index of pupil positioning accuracy. Its calculation formula is as follows:
where is the artificially marked boundary value of pupil outer tangent rectangle; is the pupil tangent rectangular boundary obtained by TST; represent the upper, lower, left and right boundaries, respectively; and is the error value.
4.2. Experimental Results
In order to verify that TST has a better pupil localization effect, this paper made comparisons with the classical Hough [15] pupil localization method, Gangwar [19] pupil localization method, Fuhl [20] pupil localization method, and traditional threshold method. It can be seen from the results that TST has high pupil localization accuracy in high-quality datasets and maintains good robustness in complex eye images. However, Gangwar and Fuhl failed to separate the pupil from the image background, resulting in a decrease in the pupil localization accuracy of their algorithms in complex images, as shown in Figure 7.
As shown in Table 1, TST maintains high accuracy in both the high-quality CASIA-IrisV1 dataset and the complex image dataset of CASIA Iris Subject Ageing. Even in the specially processed CASIA-IrisV4 dataset, 86.8% accuracy was achieved.
The above results show that TST can better separate the pupil from the background area by fitting the gray parameters of the pupil edge, which simplifies the subsequent screening of the pupil-connected domain and improves the accuracy and robustness of pupil location.
5. Discussion
TST showed good pupil positioning accuracy and robustness in the above dataset images for both the high-quality CASIA-IrisV1 dataset and the challenging CASIA-IrisV4 dataset. The reason is that TST is different from other traditional pupil localization methods, which focus more on the selection of image binarization parameters. Through the three-step threshold method included in TST, the gray parameters of the inner edge of the pupil are obtained to eliminate the interference of most background targets. Then, by expanding the parameters of the inner edge of the pupil again, the integrity of the target pupil is guaranteed, and the information of the pupil edge will not be lost due to the selection of too small binarization parameters. Finally, through improved linear interpolation method, the pupil parameters and outer edge-to-edge values make the final binarization parameters suitable for the separation of the pupil to ensure the integrity of the pupil’s case, as much as possible from the interference of background factors, and simplify the pupil-connected domain-filtering conditions. However, TST-adaptive adopts the adaptive threshold method to binarize the image, which retains most interference factors in the image and cannot achieve complete pupil separation, thus increasing the difficulty of pupil location and reducing the accuracy of pupil location.
In the pupil-connected domain in the process of screening, based on the idea of IOU losses of the convolutional neural network computation, we proposed the pupil of the connected domain circumscribed rectangular area ratio, aspect ratio, and contour point sets and the average Euclidean distance of the image center joining the pupil filter-connected domain, combining the pupil contour point number. Thus, the difficulty of filtering the pupil-connected domain is reduced. The accuracy and robustness of pupil location are increased. In Figure 8, TST in the 1-pixel error range reached the peak pupil location accuracy. The reason is that in the binary parameter selection problem, we adopt the three steps of the threshold value method to determine the threshold parameter, which hugely fits the pupil edge information. Therefore, in the process of pupil orientation, the positioning error is smaller.
(a)
(b)
(c)
6. Conclusion and Future
This paper proposed a method of pupil localization via a novel parameter optimization, called TST. TST includes the three-step threshold method. Compared with other pupil location algorithms, the three-step threshold method pays more attention to the early image processing, thus simplifying the difficulty of subsequent pupil location. In addition, in order to solve the problem that the pupil localization accuracy decreases when the pupil is close to the background gray value, this paper designed a unique filtering algorithm of the pupil-connected domain. The experimental results show that TST has a better pupil localization effect and is suitable for pupil localization in complex images.
In the future, TST may help to improve the overall performance of eye movement tracking due to its good performance. We will improve the reliability of eye movement tracking by combining the reflected light spot of the pupil with the pupil center coordinate of TST positioning.
Data Availability
To test our method, we used the CASIA-IrisV1 dataset, the CASIA-IrisV4 dataset, and 2032 eye images from the CASIA Iris Subject Ageing dataset collected by the Institute of Automation, Chinese Academy of Sciences, and the CASIA iris image database, in http://biometrics.idealtest.org/activeuser.do?id=31855, Chinese Academy of Sciences Institute of Automation.
Conflicts of Interest
The authors declare that they have no conflicts of interest.