Abstract
The research reported in this paper focuses on the modeling of Local Binary Patterns (LBPs) and presents an a priori model where LBPs are considered as combinations of permutations. The aim is to increase the understanding of the mechanisms related to the formation of uniform LBPs. Uniform patterns are known to exhibit high discriminative capability; however, so far the reasons for this have not been fully explored. We report an observation that although the overall a priori probability of uniform LBPs is high, it is mostly due to the high probability of only certain classes of patterns, while the a priori probability of other patterns is very low. In order to examine this behavior, the relationship between the runs up and down test for randomness of permutations and the uniform LBPs was studied. Quantitative experiments were then carried out to show that the relative effect of uniform patterns to the LBP histogram is strengthened with deterministic data, in comparison with the i.i.d. model. This was verified by using an a priori model as well as through experiments with natural image data. It was further illustrated that specific uniform LBP codes can also provide responses to salient shapes, that is, to monotonically changing intensity functions and edges within the image microstructure.
1. Introduction
The Local Binary Pattern (LBP) methodology [1] was first proposed as a texture descriptor, but it has later been applied to various other fields of computer vision: for example, face recognition, facial expression recognition, modeling motion and actions, as well as medical image analysis. Numerous modifications and improvements have been suggested to the original LBP methodology for various applications, while the LBPs have also been proposed for signal processing tasks beyond image processing (e.g., [2]). A detailed list of various applications and papers related to the LBP methodology is available in CMV Oulu pages [3].
Before the introduction of Local Binary Patterns, co-occurrence statistics descriptors based on binary features and -tuples [4], as well as the texture unit and texture spectrum (TUTS) method [5], have been studied. -tuples have been studied in, for example, [4, 6] for texture retrieval. It was discovered that the distribution of individual -tuples could not reach the classification accuracy of quantized binary features such as BTCS [4].
The possibility of using only uniform and rotation invariant binary patterns distinguishes the Local Binary Pattern methodology from its predecessors, because it enables a more compact image representation. It has been widely accepted that uniform LBPs, which contain at most two circular 0-1 or 1-0 transitions, are highly applicable and thus have been frequently used in various applications—not only in texture analysis. While many modifications to the original LBP have been proposed, most image analysis applications still take advantage of a combination of LBP and uniform patterns, despite other modifications in sampling, such as applying Gabor filtering as a preprocessing step [7]. However, it has been unclear how these particular uniform patterns contribute to increasing the discriminative capabilities of the LBPs. It was shown in [8] that uniform patterns are a priori very frequent even with random data. The observations from the existing research raise naturally the question “Why are uniform patterns so discriminative?”. It was also shown in [8] that the percentage of uniform patterns further increases with natural image data compared to a priori model. In this paper, some of the mechanisms related to this increase in occurrence probability will be addressed.
LBPs are represented in this paper as compositions of individual -tuples, that is, permutations. We denote the set of all possible permutations as the “permutation space.” The permutation model represents a middle ground in terms of the complexity between the image intensity representation and the binary pattern feature representation. The total number of permutations in the permutation space is smaller than the number of instances in the intensity space but higher than the amount of instances in the binary pattern space. The permutation based approach is applied here both through an a priori model and through experiments with natural images to examine the particularly discriminative quality of uniform LBPs. The aim of this study is also to better understand the relationship between uniform LBPs and the properties of deterministic nonrandom image data.
This paper is composed of eight sections. Section 2 contains an introduction to LBP methodology as well as background and related work. Section 3 defines the permutation space used and a priori probability model for the uniform patterns. In Section 4, a modification to the original permutation space is defined for modeling purposes, while in Section 5 the previously defined concepts are used to analyze the uniformity of Local Binary Patterns. In Section 6 qualitative and quantitative experiments with a priori model and natural image data are performed. Sections 7 and 8 provide further discussion and conclusions.
2. Background and Related Work
2.1. Derivation of Local Binary Patterns
A Local Binary Pattern is derived for a specific pixel neighborhood radius by comparing the intensities of discrete circular sample points to the intensity of the center pixel (clockwise or counterclockwise), starting from a certain angle. The comparison determines whether the corresponding location in the Local Binary Pattern of length is 1 or 0. A value 1 is assigned if the center pixel intensity is smaller than the sample pixel intensity and 0 otherwise. Sample number is the most commonly used, with circle radius ; however, also other values for the radius and sample numbers can be used. If a sample point is located between pixels, the intensity value used for the comparison can be determined by bilinear interpolation (see Figure 1). Using this sampling procedure, sweeping over the whole image is denoted by [9, 10].
After the LBP extraction, each pixel in an image is replaced by a binary pattern, except at the borders of the image where all of the neighbour values do not exist. The feature vector of an image then consists of a histogram of the pixel LBPs. The initial length of the histogram is since each possible LBP is assigned a separate bin. If there are regions in an image (e.g., a normalized face image could be divided into blocks for enhancing the spatial accuracy of the histograms [9]), the histograms can be combined into a single histogram with a length of .
2.2. Uniform LBPs
Uniform Local Binary Patterns are patterns with at most two circular 0-1 and 1-0 transitions. For example, patterns 00111000, 11111111, 00000000, and 11011111 are uniform, and patterns 01010000, 01001110, or 10101100 are not uniform. Selecting only uniform patterns contributes to both reducing the length of the feature vector (LBP histogram) and improving the performance of classifiers using the LBP features (see [1, 9–11]). Uniform LBPs can also be applied to obtain rotation invariance [10]. In [12–14] global rotation invariance for the LBPs was achieved by applying a Discrete Fourier Transform to the uniform bins of the LBP histograms. In [13], the rotation invariance of LBP variants was also analyzed.
There are several methods for performing LBP histogram comparison. These include histogram intersection, log-likelihood statistics, and Chi-square statistics [9]. It is also possible to use multiple LBPs simultaneously, also with different radiuses, to describe a certain image location. The natural disadvantage of this is the further increase in the length of the feature vector.
2.3. Related Research
The use of uniform Local Binary Patterns was proposed in [15] as a way to reduce the high dimension of the original LBP feature vector. The use of uniform patterns can be seen as a filter type feature selection method [16], since it is related directly to the image data. Also, a beam search method was proposed in [15] using feedback from a classifier (for texture images), extending to a wrapper type feature selection [16]. Later, numerous other filter and wrapper type methods have been proposed for LBP feature selection, including Fisher separation criterion (FSC) based learning, Boosting (AdaBoost), LDA, and PCA; see [1] for a complete description. Face recognition has typically been used as a benchmark application. In [11] a machine learning approach was chosen to study which individual LBP bins were most discriminative in facial expression recognition. A boosting classifier was used, and 91.9% of the most discriminative patterns turned out to be uniform. However, it has been unclear how these particular uniform patterns contribute to increasing the discriminative capabilities of the LBP methodology.
In this paper, the space of all possible -tuples is constructed, and its relation to individual uniform and nonuniform LBP codes and to the class of all uniform LBP codes is modeled. Hence, we propose forming the LBPs as a result of an intermediate nonlinear rank ordering operation in order to facilitate the understanding of the Local Binary Patterns. Rank ordering and the census transform were introduced in [17] as nonparametric descriptors. Since then, many variants of descriptors based on ordinal intensity representation have been proposed. Recently, the local intensity order pattern (LIOP) descriptor showed excellent performance [18] in keypoint matching. It was further developed in [19]. In LIOP the numbers of occurrences of -tuples among local patches are selected as histogram bins in a rotation invariant manner. The methods proposed in this paper can also support the design of new descriptors and promote understanding of the existing descriptors based on -tuple processing, for example, [18, 19].
LBPs have also been described as vector quantized responses to linear filters [20]. This allows the analysis of the properties of the LBP operator and the modeling of its relationship to other filter bank based descriptors. The focus of this paper is also to provide an alternative approach to the filter bank based LBP decomposition [20] and to suggest an alternative approach to the previous work in [8] for analyzing the formation of uniform patterns.
In [8] the a priori distribution of uniform LBPs was studied, and it was observed that their a priori probability is rather high also with independent identically distributed (i.i.d) data. This indicates that these patterns do not necessarily relate only to image structures such as small edges, corners, and line-ends, as was previously thought, but also to the LBP sampling process itself. The percentage of uniform patterns has been also shown to further increase from the estimated a priori probability [8] in applications using natural image data. The exact distribution of LBPs was studied by calculating the volume of multidimensional polytopes in [8]. It has also been shown that the minimum between the total number of zeros and ones in a LBP can be used to uniquely characterize the occurrence probability of the LBPs with i.i.d. data [21]. However, in [21] the a priori probabilities were not linked to the occurrence frequencies of uniform patterns. In both studies [8, 21], a link between information theory and a priori occurrence probabilities of the LBPs was speculated.
In [8] the LBPs were modeled using a space partitioning approach, where the pixel intensities (see Definition 1) were mapped into LBP binary pattern space. In practice this means that, for example, for the operator the dimension of the intensity space is , for 8-bit pixels, and a certain location of an image (consisting of a set of intensities) would represent an individual point in this space. This particular intensity set could then be further mapped into the LBP space.
Definition 1. The intensity space I used in derivation of Local Binary Patterns consists of sets of instances in space ordered circularly around the center in addition to the center point . Its dimension is , where represents the range of the intensities.
The a priori probabilities of LBPs in the case of continuously distributed i.i.d variables (as intensities) were considered already in [21]. The a priori probability of individual LBPs (for i.i.d data distribution without interpolation) is completely determined by its descriptor (see Definition 2) according to (1), following the binomial distribution [8, 21].
Definition 2. The descriptor [8, 21, 22] for a Local Binary Pattern is calculated as the minimum between the cardinalities of the sets consisting of 1 bits and 0 bits.
For example, the descriptor for pattern 00110011 is four, for pattern 00000000 zero, and for pattern 01010100 three.
Theorem 3. The probability of an LBP with M contour samples () to occur with continuously distributed i.i.d. data [8, 21], without considering interpolation, given descriptor is
3. Constructing the Permutation Space
3.1. Definition of the Permutation Space
We propose adding a “mid-space” between the intensity space and the LBP space, the permutation space illustrated in Figure 2. The concepts of root permutations and child LBPs in Figure 2 will be explained in the later sections. This provides an alternative approach to [8] in modeling the uniform patterns and allows modeling some of the fundamental differences between -tuple and LBP based approaches for low level image representation [1, 18, 19]. An LBP is here modeled to consist of multiple instances of unit permutations located among the permutation space. The a priori probability of each unit permutation is equal with i.i.d. data, which is readily well known within nonparametric statistics [23].
Definition 4. Let the intensity space I be defined as sets consisting of intensities around the center in addition to the center point within a local LBP neighborhood. A unit permutation is defined as an individual permutation around the center in addition to the center point rank formed by rank ordering the intensity samples as so that the smallest intensity is assigned a rank of 1. In the case of tied intensities, the ranks of the tied intensities are taken from an i.i.d distribution. The length of the unit permutation is then (), where the number of contour samples in the corresponding LBP is equal to .
The permutation space P contains all possible unit permutations for the circular neighborhood. It can be derived from the intensity space so that the number of instances is reduced (since, in practice ). The number of instances in the permutation space is . The binary pattern space b consists of instances of sets composed of bits excluding the center, for being the number of contour samples among the LBP, respectively. The dimension of the binary pattern space is . The binary pattern space b can be produced directly from the high-dimensional intensity space as in [8]. It can also be derived indirectly through the permutation space consisting of reduced number of instances. Each individual LBP can also be projected back to the permutation space from the binary pattern space into a collection of unit permutations (see Figure 2).
In this paper, when considering the a priori probabilities, interpolation is not used. However, its effect to the final LBP distribution is evaluated in the experimental section using natural image data. Interpolation in the context of studying the uniform patterns has been considered more in depth in [8]. In general, the number of uniform patterns has shown to grow with interpolation [8], due to increased dependency and correlation between the neighboring sample points as they are averaged using bilinear weighting from their four neighbors.
3.2. Local Binary Pattern Operator for Permutations and Reverse Mapping
The mapping operator between the intensity space and the LBP space and between the permutation space and the LBP space is defined in the following. The mapping operator can be used for both permutation space and intensity space to derive an LBP. In the case of the permutation space, instead of intensities, the magnitudes of the ranks are considered. The mapping from an instance of the LBP space (a particular LBP code) into the permutation space () is defined indirectly as forming the set of all unit permutations which result in this particular LBP according to the operator.
Definition 5. (a) The LBP mapping operator is defined between the instances of spaces , or as 1 for instances which have a magnitude greater or equal than the center and 0 for instances which have magnitude smaller than the center. In the case of mapping and ties between the center intensity and contour intensities , the rank of the center is assigned the minimum within the combined set of the tied ranks . This preserves the uniformity of the LBP also when the intermediate permutation space is used. The resulting LBP code is a concatenation of the individual bits.
(b) A mapping from intensity space to binary pattern space is defined by applying the LBP mapping operator for intensity set resulting in binary pattern .
(c) A mapping from an instance of intensity space to an instance of permutation space is defined as , where an operator extracts the rank ordering of intensity samples among the intensity set so that the smallest element will be assigned to value 1.
(d) A mapping from the permutation space into the binary pattern space is defined by applying the LBP mapping operator to the set of ranks resulting in a binary pattern among the binary pattern space.
(e) A reverse mapping from an instance of binary pattern space into permutation space is defined indirectly as forming all the elements according to the criteria results in a match (from all elements of ).
As an example, consider an arbitrary Local Binary Pattern, for example, pattern 00110011. It is a result of applying the operator to a unit permutation, where the rank of the center pixel (ordinal value) is always 5 (rank of smallest being 1). An example of a permutation which could produce this particular LBP could be for center pixel and for the other pixels.
This is not the only possible unit permutation for this particular LBP. The degree of freedom related to the unit permutations for a given LBP is determined by the following elements: the rank of the center pixel, the number of bits above the center pixel, that is, number of 1s in LBP code, and number of bits below the center, that is, number of 0's. The a priori probability of occurrence for an individual unit permutation is then constant , with i.i.d. data, where is the length of the unit permutation and is the number of contour samples in an LBP. Hence, the intensity space is divided into larger fractions of equal unit probability, still allowing the derivation of Local Binary Patterns.
In the case of LBP 00110011, the number of unit permutations invoked by the restriction “four locations above the center” is , and the number of unit permutations invoked by the restriction “four locations below the center” is also . As a consequence, the total occurrence probability of the LBP under consideration becomes , from which is assigned for each of the unit permutations. As another example, consider the LBP with , 01000, containing, for example, the unit permutation {5} center (since four zeros are below it), contour, which results in a total cumulative probability of for all the unit permutations.
3.3. Modeling a Priori Distribution of Uniform Patterns Only with i.i.d. Data
Next we consider the total occurrence probability of all uniform patterns with i.i.d. data with respect to all LBPs. The number of uniform patterns with respect to descriptor is described completely in (2) and (3) with respect to and descriptor . For (2) (even ) varies between and , and for (3) (odd ) varies between and .
For example, consider an arbitrary LBP with . The case of consists of uniform patterns 0000 and 1111. When , the uniform patterns are 0001, 0010, 0100, 1000, and the inversions of these patterns. In these cases all patterns are uniform. Descriptor value (i.e. ) leads to the uniform patterns of 1100, 0110, 0011, and 1001. The other nonuniform patterns for the highest possible value for are 0101 and 1010.
With LBPs having the total occurrence probability of the set of all uniform patterns, for i.i.d. data distribution and even is,
The first term of (4) includes the a priori probability of all-zero and all-one uniform LBPs, the second term is the sum of probabilities of patterns where varies between and for even , and the third term includes the highest descriptor uniform patterns.
According to the permutation concept proposed here, the number of all unit permutations for uniform patterns becomes then *, while the total number of all permutations for modeling LBPs with sample number becomes . In Figure 3 the total occurrence probability of uniform patterns is plotted with respect to the LBP sample number using the (4) (for odd the summation in (4) was adjusted according to (3)).
4. Modified Permutation Space
To analyze the formation of LBPs and uniform patterns in particular, we propose a modification to the permutation space by removing the center rank of a unit permutation. This requires the definitions of an intermediate unit permutation, a root permutation, and an intermediate root permutation.
A certain LBP can be composed of multiple unit permutations, but if an instance of the intensity space set is mapped to the LBP binary pattern space, a single unique intermediate unit permutation can be assigned to the permutation space and it can be uniquely used for determining the resulting LBP (neglecting ties).
Definition 6. An intermediate unit permutation is defined as a unit permutation as the permutation . The directions of the arrows describe the order in which the rank ordering () and mapping () are performed.
Depending on the context, a root permutation may be derived from an intermediate unit permutation, which is simply extracted by rank ordering the intensity set without the center, or in more abstract case it may refer to a normal unit permutation from which the rank of the center pixel is removed (see Definition 7 and Table 1).
Definition 7. A root permutation is a unit permutation produced by removing the rank of the center. The rank of the center pixel is first removed from the unit permutation, and the rank ordering () is applied again for the magnitudes of the remaining ranks. An intermediate root permutation refers simply to an -tuple formed by rank ordering the LBP contour sample intensities (without considering the center). The total length of the root permutation or the intermediate root permutation is then , that is, equal to .
Two different LBPs or unit permutations can produce the same root permutation. For example, LBPs with , 0000 and 1111 could be composed of instances of permutations {5} center, {1, 2, 3, 4} contour, and {1} center, {2, 3, 4, 5} contour, respectively, the root permutation for both being {1, 2, 3, 4}. Given an arbitrary LBP, only if the full intensity set or the intermediate unit permutation is known, a certain root permutation can be used to describe that particular LBP. However, it will not be unique, since the information from the center intensity value or rank is missing.
Given a root permutation or an intermediate root permutation, multiple child LBPs can be generated if a new center pixel (called bias) is assigned. A root permutation is allowed to generate child LBPs by setting up a new bias level (instead of the center pixel). In practice, any bias value between or above the magnitudes of the ranks could be used, but for clarity we use threshold values in the middle of the integer ranks (0.5, 1.5, 2.5, 3.5, etc.). In Table 1, three root permutations are shown, and the child LBPs which can be generated by these root permutations are described by changing the center bias. One of the child LBPs will always represent the original LBP.
Definition 8. A child LBP is an LBP generated from the root permutation or from the intermediate root permutation by comparing successively the bias level to the magnitude of the root permutation ranks (one by one) and assigning bit 1 if the magnitude of the bias level is below the magnitude of the rank permutation and 0 otherwise (in the corresponding location). That is, the LBP mapping operator is applied to the root permutation by using the set bias as a center.
In Section 6, when performing the experiments, we will use an approach where all possible root permutations (corresponding to contour samples of the LBPs) are pre-generated and the bias is then adjusted to each of its possible locations for each root permutation.
4.1. Monotonicity of the Root Permutation
Consider fully ordered (with respect to some circular shift) root permutations of length containing only one long increasing or decreasing run, in addition to the transitions from the smallest to the largest or from largest to the smallest. We describe next that for these root permutations, all of the child LBPs (generated by changing the center pixel bias) are uniform (See Table 1).
For the lowest and the highest bias value, the resulting child LBPs derived from the root permutation are all uniform since they consist of only all-zero and all-one LBPs. For the second lowest bias and the second highest bias, the resulting patterns are uniform since they include only one instance of one or zero (the descriptor being one). The remaining bias values (between to ) for are considered next.
The first observation is that each fully ordered permutation can be circularly shifted left or right times in order to make it a monotonically increasing (or decreasing) permutation starting with the lowest (highest) element. It can also be shifted back so that the generated child LBP is not altered. Let us focus only on the length (longest) up (or down) run while also omitting the circularity. Let {B} be the set of the root permutation instances below the bias value and {A} the set of the root permutation instances above the bias value. It is evident that for the set {B}, if the ordering of the permutations is monotonically increasing, then they are fully ordered in increasing order also in {A}. The same applies for the decreasing permutations.
When changing the bias by one (increasing or decreasing), the instance where the bit transition occurs, generated by the bias, also shifts by one to the right or to the left. While acknowledging the initial conditions which hold for descriptor values 0 and 1 (as described before), it is evident that only one bit right next to the transition point can change and it will change to the direction which preserves the uniformity. See, for example, Table 1 Case C. The ordered root permutations are likely to be common with natural image data as pointed out in [4]. Also, statistically their likeliness to occur with random data should be lower than with deterministic data, which will be considered in the next section.
5. On the Relation between Runs Test for Permutations and Uniform Patterns
The simplest tests of randomness for two-valued data (e.g., coin-toss data) are based on estimating the total number of instances of each value or to count the number of successive instances of each [23]. For example, the sequence (T T T T T T T T) is not likely to be generated randomly, while the sequence (T F T T F F T T) would be more likely to result from a random process. If there are too many or too few instances of each value, the generating process is not likely to be random (i.i.d.).
The runs up and down test can be applied for numeric data, such as intensities or ranks, to examine the number of monotonically increasing (decreasing) sequences (runs). According to the runs test, monotonicity is the strongest indication of nonrandomness. If the length of the runs is high (few runs within the data), the data is not likely to be generated by a random process. In other words, the hypothesis of randomness is rejected.
Given a root permutation, if the changes from the smallest towards the largest element, or vice versa, occur always next to the set of elements which have previously been changed, all of the child LBPs will be uniform. We denote these root permutations as complete uniform root permutations (see Table 1 case C as an example). Fully ordered root permutations described in the previous section form a subset of complete uniform root permutations. For instance, permutations {6, 4, 1, 2, 3, 5}, {1, 2, 3, 4}, and {1, 2, 3, 4, 8, 7, 6, 5} are complete uniform root permutations (see also Figure 4).
Definition 9. Complete uniform root permutations are root permutations for which all of the child LBPs are uniform. The number of complete uniform root permutations is given by , (), where is the number of contour samples in LBP and the length of a root permutation.
The total number of complete uniform root permutations can be examined through the following example: consider the bias changing from its lowest level towards the highest (Figure 4). For the lowest level, only one uniform LBP can be found (of all ones, e.g., 11111 in the case of ). For the second lowest bias level, all the child LBPs are also uniform, since only one bit is changed in comparison with the previous bias, and the other bits are 1s. For the following bias levels, uniform patterns will be generated if and only if the successive change among the child LBPs is always next to previous changed bits (either circularly to the left or to the right). For example, for an LBP of 11000111 (indexes 1, …, 8) the next change to zero could occur only on index location 2 (to the left) or location 6 (to the right) in a circular manner.
The changing bit will also indicate the successive value of the root permutation formed among the path of the permutation tree (see Figure 4). The number of leaf nodes for this tree is then equal to the total number of complete uniform root permutations, which is , (). It consists of patterns for the second lowest bias and for each of these patterns, a perfect binary tree of height .
Definition 10. Runs up and down test result for the root permutations is defined as the total number of successive increasing or decreasing circular runs within a certain root permutation.
It was observed that all of the complete uniform root permutations with matched to category two patterns (-tuples) according to the circular runs test. However, in this paper the proof of the equivalence of the runs level 2 patterns and the complete uniform root permutations is omitted for . We emphasize that the number of runs according to the runs up and down test for permutations is not the same as the uniformity level of the pattern. The runs test for permutations is a more flexible and general test of randomness, and it can only be applied if the rank order statistics of the successive samples in a pattern are known, since a fixed threshold (bias) is not used as in LBP. The derivation of the rank order statistics is not necessary for extracting the LBPs, since the thresholding according to the center pixel's intensity value determines the pattern uniquely. However, the LBP can still be uniquely determined from the rank permutations. Hence, changing the bias level for generating the child LBPs from root permutations can be seen as a unifying approach between the LBPs and permutation space, where the contribution of the center pixel is adjusted by using the bias.
6. Experiments
6.1. Qualitative Tests on -Tuples and LBPs
In this subsection, the distribution of the -tuples (intermediate root permutations) is studied with natural image data. The objective is to characterize which individual -tuples are most common with natural image data and to make implications on their role in the formation of uniform patterns. Also, the spatial response of the -tuples to different image structures is studied with different kind of images. The runs level 2 intermediate root permutations described in the previous section will be shown to be among the most common -tuples with natural image data. This observation can promote the understanding of the uniform patterns and their high occurrence probability with natural images in particular.
In Figures 6, 7, 8, and 9 the most common intermediate root permutations are shown for different test images of Figure 5. The total number of occurrences of each permutation is also shown. For each instance of a certain permutation in the corresponding test image, a neighborhood of pixels was extracted and all of the local intensity blocks for the given permutation were combined, that is, added together. The intensity scale was then normalized based on the minimum and maximum values within the sum of the permutation blocks. In order to better distinguish between the true monotonic runs level 2 -tuples in nonflat image areas, only patterns which did not contain ties were extracted.
When comparing the responses in Figures 6 and 9, it can be observed that with small neighborhood radius ( with ) the -tuples appear to correspond to edges in various orientations. It can also be observed that the most common intermediate root permutations are typically of runs level 2. Hence, the theoretical analysis in the previous sections is also supported by occurrence statistics of the natural images. It should be noted that the property described in the previous sections, stating that the runs level 2 root permutations will always produce a uniform pattern independently of the bias chosen, holds for the intermediate root permutations as well. Independently of the center chosen, these intermediate root permutations will always produce a uniform LBP (to the direction ). The orientations of the detected edges follow the direction of the most common gradients among the test images (see e.g., Figure 9 and the corresponding test image 4). In Figure 7, the local neighborhood is extended to of 8 and radius of 8 using test image 2. It can be observed that now the monotonically changing image structures specific to the sky and to the water dominate the average intensity blocks. In Figure 8, of 6 and radius of 6 are used. It can be observed that the intensity structures corresponding to the -tuples become smoother compared to the lower radius. In this case the -tuples capture larger scale changes.
Tests with repeated textures were also performed. The images shown in Figure 10, from the Outex [24] dataset, were used. In Figure 11, the response of the most common -tuples within (8, 2) neighborhood with interpolation is shown using the Outex images. The most common permutations in Figure 11 correspond to the structures present in wood_012 texture sample. It consists of a gradually changing monotonic texture pattern. According to this experiment it would seem that especially monotonic changes contribute to the formation of runs level 2 patterns.
Figure 12 shows the average intensity blocks of the LBPs in (6, 2) neighborhood related to the most common intermediate root permutation of test image 4, that is, permutation {6, 4, 2, 1, 3, 5}. The shown LBPs correspond to situation where the bias is changed gradually from its minimum value to the maximum value. It can be noted that as the descriptor of the LBPs grows, the spatial support for the edges becomes stronger. With small , the detected structure becomes limited to the proximity of the center pixel. If the permutation tree representation of Figure 4 is used (in this case ), the corresponding path for this runs level 2 permutation is , , , in a circular manner. With similar tests using nonuniform patterns, the response becomes partly limited to the area of the pattern itself, without significant edge support (see Figure 13).
6.2. Tests with a Priori Model
In this subsection, the a priori distribution of LBPs is further studied. The objective is to better understand which factors contribute to the formation of uniform patterns in particular. The proposed approach can also give new perspectives on the previous studies in [8, 21]. Based on the examples in Section 4 (e.g., Table 1), we hypothesize that root permutations which include monotonically ordered subsets would produce more uniform patterns than permutations of arbitrary order. This is analyzed first.
The total number of unit permutations (and final LBPs) was , corresponding to the number of instances in the constructed permutation space. The following experimental procedure was then applied.(1)A table containing all possible intermediate root permutations was constructed.(2)Using these permutations as root permutations, the center bias was changed for each permutation times in order to generate all the child LBPs for the permutation space corresponding to all unit permutations in given LBP neighborhood .
In Figure 14, the length of the longest monotonic run among the root permutations is plotted, along with the total share of permutations resulting in uniform child LBPs. For example, in the case of , if the length of the longest monotonic run among the root permutations is 3, roughly 40% of the resulting LBPs are uniform, and if the length of the longest monotonic run is 7, roughly 70% of the resulting LBPs are uniform. These results would seem to support the given hypothesis.
(a)
(b)
(c)
Next, we studied the correspondence between the runs test and the relative share of uniform Local Binary Patterns. In Figure 15, the setup was the same as previously used. The percentage of permutations resulting in a uniform child LBP is plotted, with respect to the total number of LBPs within the current category. The result of the circular up and down runs test for randomness is indexed on the -axis. It can be observed that as the result of the runs up and down test for permutations decreases, the share of uniform patterns increases. It can also be observed that for the runs test result 2, all the LBPs are uniform.
(a)
(b)
(c)
In Figure 16 the data corresponding to the previous experimental setup is plotted with and also with respect to descriptor . It can be observed that, as decreases, the share of permutations resulting in uniform LBP increases. Also, as the result of the circular up and down runs test for randomness decreases, the relative share of uniform permutations (frequency of permutations resulting in uniform patterns) increases. This implies that uniform patterns become more frequent with a stronger hypothesis of nonrandomness according to the runs test. According to Figure 16, for the descriptor values 0 and 1, all of the root permutations result in uniform patterns, since all patterns (of all-zero LBP, all-one LBP, all-zero LBP including a single 1, and all-one LBP including a single 0) are then uniform. When the result of runs up and down test is 2, the patterns remain uniform despite the increase in .
We also examined the relationship between the runs test and individual LBPs by considering their root permutations. It is clear that, for example, LBP 01010101 must contain at least 8 runs. However, LBP 10000 could contain various numbers of runs according to runs up and down test for randomness, since now the leftmost LBP bit (or the bit change next to it) alone restricts the degree of freedom of the underlying root permutations. The remaining long sequence of all zeros allow a large number of combinations considered as possible root permutations.
In the following, a reverse mapping of Definition 5(e) is used. For LBPs with , 000101, 101110 the number of all circular runs among the root permutations varied between 4 and 6, with a mean of 4.667 and variance 0.908. For uniform patterns 011100 and 111000 the runs test result varied between 2 and 6 with a mean of 3.33 and variance 1.829. The rotation of the patterns did not seem to affect the result, as was expected. For uniform pattern 00000000 the number of runs varied between 2 and 8 with a mean of 5.333 and variance 1.4223 (in this case, the total number of root permutations was ). For uniform pattern 01100000 the result of the runs test varied between 2 and 6 with a mean of 4.667 and variance 1.245. The results from the test indicate that nonuniform patterns, in average, contain more runs than the uniform patterns, as can be expected.
In Figure 17, the number of uniform patterns (permutations resulting into uniform patterns) is shown, with respect to the length of the longest monotonic run and descriptor , while using the same approach as in the previous figures of this subsection. It can be observed that maximum run lengths around 3, 4, and 5 provide most of the contribution to uniform Local Binary Patterns with LBP neighborhood of in the case of the a priori model. As expected, as the descriptor grows, the number of permutations resulting in uniform LBPs also decreases due to reduced number of binomial combinations. In general, the number of particular unit permutations related to high descriptor LBPs is smaller (see (1)), emphasizing the relative effect of these particular unit permutations on the final LBP histogram.
6.3. Experiments with Natural Image Data
The experiments of the previous subsection with the a priori model are next considered in the case of natural image data, by using the test images of Figure 5. In the following, the effect of ties and the effect of runs level 2 permutations are considered separately. This allows studying the role of these two in the formation of uniform patterns in particular. The grayscale range of all test images is 8 bits. Figures 18 and 19 correspond to the experiments with the a priori model shown in Figures 16 and 17, respectively. In Figure 18, the percentage of uniform patterns with respect to runs up and down test and descriptor is shown. An increased percentage of uniform patterns with larger values, in comparison with the a priori model, can be observed. These patterns correspond to monotonic edge structures in the images. In Figure 19 the number of uniform patterns is increased, especially for larger descriptor values and longer run lengths. While being a priori is extremely rare, the high patterns are actually among the most common ones with natural image data when only uniform patterns are considered. The results of Figures 17 and 19 also indicate that natural image data increases the monotonicity of the root permutations for the uniform patterns. As a consequence, the number of uniform patterns is increased.
The effect of ties is considered next. In Figure 20, the percentages of local neighborhoods containing at least one tie are shown with respect to neighborhood radius and the possible usage of interpolation using all of the test images of Figure 5. For uniform patterns, the percentage share of patterns containing at least one tie is also shown. It can be observed that ties are not, on average, more common among uniform patterns than with the other patterns. Similar results were obtained with the Outex textures of Figure 10. This implies that although ties could cause some minor increase in certain LBP bins (e.g., all-one bin or all-zero bin) within flat image areas, as could be predicted from the a priori model used in [8], it seems that they cannot explain most of the increase related to the occurrence frequency of uniform patterns, when using natural images. A possible explanation to this could be that ties tend to occur within flat image areas where the relative amount of i.i.d. noise is more significant.
In Figure 21, the percentage share of runs level 2 patterns is shown for each of the test images of Figure 5. A minor increase in the number of these patterns can be observed for test images 2 and 3. This could be related to the large monotonic areas of the sky and the water present in these images. The a priori estimate of the number of runs level 2 permutations (corresponding to , see Definition 9) is also shown in Figure 21. Natural images seem to increase the share of the runs level 2 permutations significantly, compared to the a priori estimate. The percentage share of runs level 2 permutations among uniform patterns is further shown for all of the test images in Figure 21 (as an overall average for test images 1–4). It can be observed that also with natural image data, the occurrence frequency of these permutations increases further among uniform patterns.
When studying Figure 21, it can be observed in the case of that interpolation increases the share of runs level 2 permutations in each of the test images. The share of uniform patterns naturally increases also when using interpolation. It can be observed that while the number of runs level 2 patterns is significantly larger with the test images than with the a priori model, increasing the neighborhood radius decreases the number of runs level 2 patterns (see e.g., the results in (8, 1) and (8, 2) neighborhoods with interpolation). This could be explained by reduced correlation among the pixel intensities within the local neighborhood, as the radius increases.
6.4. Quantitative Tests
We also performed initial experiments with the FERET facial recognition database [25] (FAFB, FAFC, DUP1, and DUP2) sets and Outex TC 0012 [24] rotation invariant texture categorization set to test whether the intermediate root permutations of runs level 2 would perform better in classification than traditional LBP with uniform patterns. In all of the following tests, if not explicitly noted, ties were coded in an increasing order of the rank magnitudes among the -tuples, so that a neighborhood containing only tied intensities was assigned to the increasing permutation {1, 2, 3, …, M}.
In Figure 22, the recognition rates for the FERET sets are shown. The length of the feature vector in was 720 (i.e., factorial of 6). It can be observed, that using only runs level 2 -tuples (total of 96 out of 720), the recognition rate was not significantly lower than with the full permutation histogram. Increasing the permutation neighborhood to (8, 2) with interpolation and using runs level 2 patterns only (total of 512 patterns out of 40320) an average recognition accuracy close to the reported LBP(8, 2) accuracy [9] was obtained. However, it appears that for larger (say 8 or more) and small radiuses the number of runs level 2 permutations drops even with natural image data. This can be related to the effect of noise, the high dimension of the permutation histogram, and the substantially low a priori probability of the runs level 2 patterns. The a priori probability of runs level 2 patterns in neighborhood (without considering interpolation) is as low as 1.27%, emphasizing the significant role of these permutation bins in the overall permutation histogram. To increase the number of runs level 2 patterns, we used a averaging filter as a preprocessing step with of 8.
The effect of ties to the performance of -tuples can also be estimated from the results of Figure 22 with . It can be observed that neglecting ties reduces the recognition performance for radius of 2, but with larger radiuses the recognition accuracy is not significantly altered. The length of the individual block histogram in [9] for the LBPs was 59, which is smaller than for the permutations.
In tests on rotation invariant texture classification (see the results of Table 2) with the Outex TC 0012 set [24], a rotation invariant mapping of intermediate root permutations (-tuples) was used. For each available permutation pattern, possible rotations were assigned, and these were then combined in a rotation invariant manner [10]. In the case of runs level 2 patterns only, a similar procedure was also performed. Using (6, 2) neighborhood the total number of runs level 2 patterns was 96, and for each rotation invariant bin, 6 rotated bins were assigned. The final length of the feature vector was then 16 bins. All rotation invariant -tuples (i.e., not only runs level 2) in the (6, 2) neighborhood resulted in a feature vector length of 120 bins. In the (8, 2) neighborhood with interpolation using only runs level 2 patterns, by always assigning 8 patterns into the same rotation invariant category the final length of the feature vector became 64 bins for the -tuples. The highest recognition rate for the -tuples was obtained with these parameters using the runs level 2 patterns only. Also the feature vector length in the (8, 2) neighborhood was shorter than with the (6, 2), which would seem to indicate that the runs 2 -tuples are among the most salient ones, if the -tuples alone are considered. However, traditional rotation invariant LBP in a (8, 1) neighborhood with interpolation resulted in a better recognition rate (64.6% accuracy) in [14] with the same test set. Also, the length of the feature vector for the LBPs was shorter (10 bins). Log-likelihood distance metric with 1-nearest neighbor classification was used in these experiments.
The previous results raise naturally the question, why are LBPs more discriminative than -tuples, despite the usage of a similar feature vector length reduction method (i.e. the usage of runs level 2 patterns) than the uniform patterns. According to the previous qualitative and quantitative analysis (see also Figure 22), it appears that the performance of the -tuples is decreased in comparison with LBPs for two reasons. First, their performance is decreased due to large number of ties among small local radiuses. However, this is changed if a larger radius is used. Second, a small change in the intensity order (i.e., on the permutation) for larger (e.g., more than 6) changes the bin placement of the histograms and increases susceptibility to noise, which is not the case in LBP. Above-mentioned issues should be considered carefully when designing descriptors based on -tuple processing. As a rule of thumb for selecting the number of samples in an -tuple, the radius of the local neighborhood should be at least equal to the number of circular sample points (see Figure 22). In the recently proposed LOCP descriptor [26] a binary representation of successive circular pairs in a local neighborhood was used. This approach avoids the negative effect of ties by changing the following circular bit only if the intensity is changed, that is, the bin placement of the permutation histogram does not change due to possible minor intensity order differences. As a consequence, robustness to noise could be achieved in [26] by only considering the neighboring circular pairs when deriving the binary pattern.
7. Discussion
In this work, the observation that Local Binary Patterns can be modeled as compositions of rank permutations was used to study some of the mechanisms related to the formation of uniform patterns. As previously observed, uniform patterns are a priori very frequent (with i.i.d. data), but this seems to be mostly explained by how they are located according to the descriptor (see (1)–(4)). For most values there are exactly instances of uniform LBP rotations and inversions, while the number of all LBPs increases rapidly with descriptor . As a consequence the relative contribution of uniform LBPs is larger at low values. Therefore, according to a priori model, uniform LBPs contain a larger share of unit permutations compared to other LBPs and are very frequent.
According to our investigation, the contribution of uniform patterns seems to get even stronger with a stronger evidence of nonrandomness among the underlying data. This issue was quantitatively analyzed in this paper using the runs up and down test for permutations. Monotonicity among the root permutations would be a strong indicator of nonrandomness. In tests with natural images, the majority of the most common circular permutations belonged to category 2 according to the runs test, corresponding to monotonically changing intensity structures. Using natural image data also increased the length of monotonic runs among the permutations, compared to the a priori model.
With real-world images occurrence percentages of uniform patterns of 70–90% and beyond are typical [10]. For example, with the percentage of uniform patterns in textures was in the range of 57.6–79.6% in [10], while the a priori probability that we estimated in this paper was less than 30% (see Figure 3) for i.i.d. data without interpolation. A considerable portion of this increase can be explained by the bilinear subsampling (interpolation when deriving the LBP code) as described in [8], but we also propose that a portion of this increase could be explained by the capability of the uniform patterns to respond to deterministic properties within the image microstructure.
The relatively high occurrence probability of uniform patterns a priori, and even higher occurrence probability with natural images, could be compared also with the relative occurrence probabilities of individual -tuples [4], since we showed that LBPs could be seen as compositions of -tuples. Monotonic -tuples dominate the occurrence statistics of natural images [4], which tends to increase the share of uniform patterns. This behavior was modeled quantitatively in this paper (see Figure 14).
Permutations with runs test result 2 can be used to capture monotonic edges and monotonic spatial image features, as was shown in Section 6. By increasing the radius of the local neighborhood, also the extent of the detected change could be increased. The formation of these patterns was examined in Figure 4. The understanding of this behavior could facilitate the development of new image descriptors inspired by the uniformity principle of Local Binary Patterns. Initial tests on the performance of runs level 2 permutation histograms were performed in Section 6, and it appeared that their performance did not exceed the original LBP. However, since the formation of the runs level 2 permutations can also be modeled as a tree structure, a graph based matching approach for the permutations could also be possible.
The use of only certain permutations inspired by the uniform pattern principle would seem to enhance the properties of -tuples (see the quantitative experiments in Section 6). However, for larger the feature extraction cost is higher for the -tuples and the required feature vector becomes longer. Therefore, in these cases the traditional LBP with uniformity is preferable. Future work includes examining the possibilities to enhance the performance of -tuple based descriptors, for example, [18, 19] based on the principles proposed. We showed that the uniform pattern principle can be at least partly extended to -tuples as well. This could provide a variety of alternatives for increasing the performance of -tuple based descriptors. For example, the pooling scheme in [18, 19] could be adjusted, not only to take into account rotation but also to select -tuples according to their runs test score. This could also allow increasing the number of neighborhood samples among pooled -tuples without increasing the descriptor length significantly.
8. Conclusions
We proposed the modeling of LBPs through nonlinear intermediate mapping into permutations. The permutation set was further modified in order to gain a more flexible LBP model by removing the effect of the center pixel on the actual permutations and by modulating the effect of the center pixel by introducing an adjustable bias term. The approach proposed in this paper was intended to provide further understanding of the LBPs and of the uniform LBPs in particular. The notion of root permutations was introduced in order to model the formation process of uniform patterns. Monotonicity among the root permutations was shown to be in an important role in the increased share of uniform patterns with natural images. The possible relationship between the runs up and down test for randomness for permutations and the selection of uniform patterns for LBP histograms was also considered. According to our investigation, the response of uniform patterns is enhanced when the result of the runs test is low, that is, indicating nonrandomness and correlation.
The a priori occurrence probability of uniform LBPs is high. This has been previously observed to be a result from the sampling process itself as well as from the use of bilinear interpolation. In addition to this, we provided quantitative analysis on how the Local Binary Pattern methodology, together with selecting only the uniform patterns, can be seen as a process which further enhances the response of such deterministic underlying image intensity structures, which are not likely to be formed by statistically random distribution or phenomena. Being also shape primitives, the uniform Local Binary Patterns naturally embody response to various microshapes.
Acknowledgment
The research was funded by the Academy of Finland Project no. 254430.