Theorems on Positive Data: On the Uniqueness of NMF

Abstract

We investigate the conditions for which nonnegative matrix factorization (NMF) is unique and introduce several theorems which can determine whether the decomposition is in fact unique or not. The theorems are illustrated by several examples showing the use of the theorems and their limitations. We have shown that corruption of a unique NMF matrix by additive noise leads to a noisy estimation of the noise-free unique solution. Finally, we use a stochastic view of NMF to analyze which characterization of the underlying model will result in an NMF with small estimation errors.

1. Introduction

Large quantities of positive data occur in research areas such as music analysis, text analysis, image analysis, and probability theory. Before deductive science is applied to large quantities of data, it is often appropriate to reduce data by preprocessing, for example, by matrix rank reduction or by feature extraction. Principal component analysis is an example of such preprocessing. When the original data is nonnegative, it is often desirable to preserve this property in the preprocessing. For example, elements in a power spectrogram, probabilities, and pixel intensities should still be nonnegative after the processing to be meaningful. This has led to the construction of algorithms for rank reduction of matrices and feature extraction generating nonnegative output. Many of the algorithms are related to the nonnegative matrix factorization (NMF) algorithm proposed by Lee and Seung [1, 2]. NMF algorithms factorize a nonnegative matrix 𝐕∈ℝ₊^𝑛×𝑚 or 𝐑∈ℝ₊^𝑛×𝑚 into two nonnegative matrices 𝐖∈ℝ₊^𝑛×𝑟 and 𝐇∈ℝ₊^𝑟×𝑚: 𝐕≈𝐑=𝐖𝐇. (1)There are no closed-form solutions to the problem of finding 𝐖 and 𝐇 given a 𝐕, but Lee and Seung [1, 2] proposed two computationally efficient algorithms for minimizing the difference between 𝐕 and 𝐖𝐇 for two different error functions. Later, numerous other algorithms have been proposed (see [3]).

An interesting question is whether the NMF of a particular matrix is unique. The importance of this question depends on the particular application of NMF. There can be two different viewpoints when using a model like NMF—either one can believe that the model describes nature and that the variables 𝐖 and 𝐇 have a physical meaning or one can believe that the model can capture the part of interest even though there is not a one-to-one mapping between the parameters and the model, and the physical system. When using NMF, one can wonder whether 𝐕 is a disturbed version of some underlying 𝐖𝐇 or whether the data is constructed by another model or, in other words, a ground truth 𝐖 and 𝐇 does exist. These questions are important in evaluating whether or not it is a problem that there is another NMF solution, 𝐖^′𝐇^′, to the same data, that is, 𝐕≈𝐑=𝐖𝐇=𝐖^′𝐇^′. (2) If NMF is used even though the data is not assumed to be generated by (1), it may not be a problem that there are several other solutions. On the other hand, if one assumes that a ground truth exists, it may be a problem if the model is not detectable, that is, if it is not possible to find 𝐖 and 𝐇 from the data matrix 𝐕.

The first articles on the subject was two correspondences between Berman and Thomas. In [4] Berman asked for what amounts to a simple characterization of the class of nonnegative matrices 𝐑 for which an NMF exists. As we shall see, the answer by Thomas [5] can be transferred into an NMF uniqueness theorem.

The first article investigating the uniqueness of NMF is Donoho and Stodden [6]. They use convex duality to conclude that in some situations, where the column vectors of 𝐖 “describe parts,” and for that reason are nonoverlapping and thereby orthogonal, the NMF solution is unique.

Simultaneously with the development of NMF, Plumbley [7] worked with nonnegative independent component analysis where one of the problems is to estimate a rotation matrix 𝐐 from observations on the form 𝐐𝐬, where 𝐬 is a nonnegative vector. In this setup, Plumbley investigates a property for a nonnegative independent and identically distributed (i.i.d.) vector 𝐬 such that 𝐐 can be estimated. He shows that if the elements in 𝐬 are grounded and a sufficiently large set of observations is used, then 𝐐 can be estimated. The uniqueness constraint in [7] is a statistical condition of 𝐬.

The result in [7] is highly relevant to the NMF uniqueness due to the fact that in most cases new NMF solutions will have the forms 𝐖𝐐 and 𝐐⁻¹𝐇 as described in Section 3. By using Plumbley's result twice, a restricted uniqueness theorem for NMF can be constructed.

In this paper, we investigate the circumstances under which NMF of an observed nonnegative matrix is unique. We present novel necessary and sufficient conditions for the uniqueness. Several examples illustrating these conditions and their interpretations are given. Additionally, we show that NMF is robust to additive noise. More specifically, we show that it is possible to obtain accurate estimates of 𝐖 and 𝐇 from noisy data when the generating NMF is unique. Lastly, we consider the generating NMF as a stochastic process and show that particular classes of such processes almost surely result in unique NMFs.

This paper is structured as follows. Section 2 introduces the notation, some definitions, and basic results. A precise definition and two characterizations of a unique NMF are given in Section 3. The minimum constraints of 𝐖 and 𝐇 for a unique NMF are investigated in Section 4. Conditions and examples of a unique NMF are given in Section 5. In Section 6, it is shown that in situations where noise is added to a data matrix with a unique NMF, it is possible to bound the error of the estimates of 𝐖 and 𝐇. A probabilistic view on the uniqueness is considered in Section 7. The implication of the theorems is discussed in Section 8, and Section 9 concludes the paper.

2. Fundamentals

We will here introduce convex duality that will be the framework of the paper, but first we shall define the notation to be used. Nonnegative real numbers are denoted as ℝ₊,‖⋅‖_𝐹 denotes the Frobenius norm, and span(⋅) is the space spanned by the set of vectors. Each type of variables has its own font. For instance, a scalar is denoted 𝑥, a column vector is denoted 𝐱, a row vector is denoted by 𝐱, a matrix is denoted by 𝐗, a set is denoted by 𝒳, and a random variable is denoted by 𝒳. Moreover, 𝐱^𝑗_𝑖 is the 𝑖th index of the vector 𝐱^𝑗. When a condition for a set is used to describe a matrix, it is referring to the set of column vectors in the matrix. The NMF is symmetric in 𝐖^𝑇 and 𝐇, so the theorems for one matrix may also be used for the other.

In the paper, we make a geometric interpretation of the NMF similar to that used in both [5, 6]. For that, we need the following definitions.

Definition 1. The positive span is given by span⁺(𝐛¹,…,𝐛^𝑑)={𝐯=∑_𝑖𝐛^𝑖𝐚_𝑖∣𝐚∈ℝ^𝑑₊}.

In some literature, the positive span is called the conical hull.

Definition 2. A set 𝒜 is called a simplicial cone if there is a set ℬ such that 𝒜=span⁺(ℬ). The order of a simplicial cone 𝒜 is the minimum number of elements in ℬ.

Definition 3. The dual to a set 𝒜, denoted 𝒜^∗, is given by 𝒜^∗={𝐯∣𝐯^𝑇𝐚≥0forall𝐚∈𝒜}.

The following lemma is easy to prove and will be used subsequently. For a more general introduction to convex duality, see [8].

Lemma 1. (a) If 𝒳=𝑠𝑝𝑎𝑛⁺(𝐛¹…,𝐛^𝑑), then 𝐲∈𝒳^∗ if and only if 𝐲^𝑇𝐛^𝑛≥0 for all 𝑛.
(b) If 𝒳=𝑠𝑝𝑎𝑛⁺(𝐁^𝑇) and 𝐁^𝑇=[𝐛¹,…,𝐛^𝑑] is invertible, then 𝒳^∗=𝑠𝑝𝑎𝑛⁺(𝐁⁻¹).
(c) If 𝒴⊆𝒳, then 𝒳^∗⊆𝒴^∗.
(d) If 𝒴 and 𝒳 are closed simplicial cones and 𝒴⊂𝒳, then 𝒳^∗⊂𝒴^∗.

3. Dual Space and the NMF

In this section, our definition of unique NMF and some general conditions for unique NMF are given. As a starting point, let us assume that both 𝐖 and 𝐇 have full rank, that is, 𝑟=rank(𝐑).

Let 𝐖^′ and 𝐇^′ be any matrices that fulfil, 𝐑=𝐖𝐇=𝐖^′𝐇^′. Then, span(𝐖)=span(𝐑)=span(𝐖^′). The column vectors of 𝐖 and 𝐖^′ are therefore both bases for the same space and as a result there exists a basis shift matrix 𝐐∈ℝ^𝑟×𝑟 such that 𝐖^′=𝐖𝐐. It follows that 𝐇^′=𝐐⁻¹𝐇. Therefore, all NMF solutions where 𝑟=rank(𝐑), are of the form 𝐑=𝐖𝐐𝐐⁻¹𝐇. In these situations, the ambiguity of the NMF is the 𝐐 matrix. Note that if 𝑟>rank(𝐑) the above arguments are not valid because rank(𝐖) can differ from rank(𝐖^′) and thereby span(𝐖)≠span(𝐖^′).

Example 1. The following is an example of an 𝑅₊^4×4 matrix of rank 3, where there are two NMF solutions but no 𝐐 matrix to connect the solutions 1100101001010011=𝐑=𝐑^}_𝐖𝐈^}_𝐇=𝐈^}_𝐖′𝐑^}_𝐇′. (3) We mention in passing that Thomas [5] uses this matrix to illustrate a related problem. This completes the example.

Lemma 2 (Minc [9, Lemma 2 1.1]). The inverse of a nonnegative matrix is nonnegative if and only if it is a scaled permutation.

Lemma 2 shows that all NMF solutions on the forms 𝐖𝐐 and 𝐐⁻¹𝐇, where 𝐐 is a scaled permutation, are valid, and thereby that NMF only can be unique up to a permutation and scaling. This leads to the following definition of unique NMF in this paper.

Definition 4. A matrix 𝐑 has a unique NMF if the ambiguity is a permutation and a scaling of the columns in 𝐖 and rows in 𝐇.

The scaling and permutation ambiguity in the uniqueness definition is a well-known ambiguity that occurs in many blind source separation problems. With this definition of unique NMF, it is possible to make the following two characterizations of the unique NMF.Theorem 1. If 𝑟=rank(𝐑), an NMF is unique if and only if the positive orthant is the only 𝑟-order simplicial cone 𝒬 such that 𝑠𝑝𝑎𝑛⁺(𝐖^𝑇)⊆𝒬⊆𝑠𝑝𝑎𝑛⁺(𝐇)^∗.Proof. The proof follows the analysis of the 𝐐 matrix above in combination with Lemma 1(b). The theorem can also be proved by following the steps of the proof in [5].Theorem 2 (see [6]). The NMF is unique if and only if there is only one 𝑟-order simplicial cone 𝒬 such that 𝑠𝑝𝑎𝑛⁺(𝐑)⊆𝒬⊆𝒫, where 𝒫 is the positive orthant.Proof. The proof follows directly from the definitions.The first characterization is inspirited by [5] and the second characterization is implicit introduced in [6]. Note that the two characterizations of the unique NMF analyze the problem from two different viewpoints. Theorem 1 takes a known 𝐖 and 𝐇 pair as starting point and looks at the solution from the “inside,” that is, the 𝑟-dimensional space of row vectors in 𝐖 and column vectors in 𝐇. Theorem 2 looks at the problem from the “outside,” that is, the 𝑛-dimensional column space of 𝐑.

4. Matrix Conditions

If 𝐑=𝐖𝐇 is unique, then both 𝐖 and 𝐇 have to be unique, respectively, that is, there is only one NMF of 𝐖 and one of 𝐇, namely, 𝐖=𝐖𝐈 and 𝐇=𝐈𝐇. In this section, a necessary condition for 𝐖 and 𝐇 is given and a sufficient condition is shown.

The following definition will be shown to be a necessary condition for both the set of row vectors in 𝐖 and column vectors in 𝐇.

Definition 5. A set 𝒮 of vectors in ℝ^𝑑₊ is called boundary close if for all 𝑗≠𝑖 and 𝑘>0 there is an element 𝐬∈𝒮 such that 𝐬_𝑗<𝑘𝐬_𝑖. (4)

In the case of closed sets, the boundary close condition is that 𝐬_𝑗=0 and 𝐬_𝑖≠0. In this section, the sets will be finite (and therefore closed), but in Section 7 the general definition above is needed.

Theorem 3. The set of row vectors in 𝐖 has to be boundary close for the corresponding NMF to be unique.

Proof. If the set of row vectors in 𝐖 are not boundary close, there exist indexes 𝑗≠𝑖 and 𝑘>0 such that the 𝑗th element is always more than 𝑘 times larger than the 𝑖th element in the row vectors in 𝐖. Let 𝒬=span⁺(𝐪¹,…,𝐪^𝑟), where 𝐪^𝑛=𝐞^𝑖+𝑘𝐞^𝑗𝐞if𝑛=𝑖,_𝑛otherwise, (5) and 𝐞^𝑛 denotes the 𝑛th standard basis vector. This set fulfils the condition span⁺(𝑊^𝑇)⊆𝒬⊂𝒫 and we therefore, using Theorem 1, conclude that the NMF cannot be unique.

That not only the row vectors of 𝐖 with small elements determine the uniqueness can be seen from the following example.

Example 2. The following is an example where 𝐖 is not unique but 𝐖=[^𝐖₃₁₁] is.
Let 𝐖=011101110. (6) Here 𝐖 is boundary close but not unique since 𝐖=𝐖𝐈=𝐈𝐖. The uniqueness of 𝐖=[^𝐖₃₁₁] can be verified by plotting the matrix as shown in Figure 1, and observe that the conditions of Theorem 1 are fulfilled. This completes the example.

Figure 1 

A three-dimensional space is scaled such that the vectors are in the hyper plane: {𝐩∣[111]𝐩=1}. By the mapping to the hyper plane, a plane in ℝ3 is mapped to a line and a simplicial cone is mapped to an area. In the figure, it can be observed that the dashed triangle (desired solution) is the only triangle (third-order simplicial cone) that contains the shaded area (positive span of 𝐖) while being within the solid border (the dual of 𝐇). The NMF can be concluded to be unique by Theorem 1.

In three dimensions, as in Example 2, it is easy to investigate whether a boundary close 𝐖 is unique—if 𝐖=𝐖^′𝐇^′, then 𝐇^′ can only have two types of structure: either the trivial (desired) solution where 𝐇^′=𝐈 or a solution where only the diagonal of 𝐇^′ is zero. In higher dimensions, the number of combinations of nontrivial solutions increases and it becomes more complicated to investigate all possible nontrivial structures. For example, if 𝐖 is the matrix from Example 2, then the matrix ^0𝑥𝐖^002𝑑𝑐=𝟎𝐖𝟎𝐖 (7) is boundary close and can be decomposed in several ways, for example, ^0𝑥𝐖^002𝑑𝑐=𝟎𝐈𝟎𝐖=𝐖𝟎𝟎𝐈𝟎𝐖𝟎𝟎𝐈𝐈𝟎𝐖=𝐈𝟎𝟎𝐈𝟎𝐖𝟎𝐖. (8) Instead of seeking necessary and sufficient conditions for a unique 𝐖, a sufficient condition not much stronger than the necessary is given. In this sufficient condition, we only focus on the row vectors of 𝐖 with a zero (or very small) element.Definition 6. A set of vectors 𝒮 in ℝ^𝑑₊ is called strongly boundary close if it is boundary close, and there exists a 𝑧>0 and a numbering of the elements in the vectors such that for all 𝑘>0 and 𝑛∈{1,…,𝑑−1} there are 𝑑−𝑛 vectors from 𝒮,{𝐬¹,…,𝐬^𝑑−𝑛} that fulfil the following:
(1)𝐬^𝑗_𝑛<𝑘∑_𝑖>𝑛𝐬^𝑗_𝑖 for all 𝑗; and(2)𝜅₂([𝐛¹,…,𝐛^𝑑−𝑛])≤𝑧, where 𝜅₂(⋅) is the “condition number” of the matrix defined as the ratio between the largest and smallest singular values [10, page 81], 𝐛^𝑗=𝐏_𝑛𝐬^𝑗 and 𝐏_𝑛∈ℝ^{𝑑−𝑛×𝑑} is a projection matrix that picks the 𝑑−𝑛 last element of a vector in ℝ^𝑑.Theorem 4. If 𝑠𝑝𝑎𝑛⁺(𝐖^𝑇) is strongly boundary close, then 𝐖 is unique.The proof is quite technical and is therefore given in the Appendix. The most important thing to notice is that the necessary condition in Theorem 3 and the sufficient conditions in Theorem 4 are very similar. The first item in the strongly boundary close definition states that there have to be several vectors with small value. The second item ensures that the vectors with small value are linear independent in the last elements.

5. Uniqueness of R

In this section, a condition for unique 𝐕 is analyzed. First, Example 3 is used to investigate when a strongly boundary close 𝐖 and 𝐇 pair is unique. The section ends with a constraint for 𝐖 and 𝐇 that results in a unique NMF.

Example 3. This is an investigation of uniqueness of 𝐑 when 𝐖 and 𝐇 are given as 𝐇=,𝛼11𝛼001𝛼00𝛼100𝛼11𝛼𝐖=𝐇^𝑇, (9) where 0<𝛼<1. Both 𝐖 and 𝐇 are strongly boundary close and the 𝑧 parameter can be calculated as 𝑧=𝜅₂([𝐛¹,…,𝐛^𝑑−𝑛])=𝜅₂(𝛼11𝛼)=1+𝛼.1−𝛼 (10) The equation above shows that small 𝛼 will result in a 𝑧 close to one and an 𝛼 close to one results in a large 𝑧. In Figure 2, the matrix 𝐑=𝐖𝐇 is plotted for 𝛼∈{0.1,0.3,0.5,0.7}. The dashed line is the desired solution and is repeated in all figures. It is seen that the shaded area span⁺(𝐖^𝑇) is decreasing when 𝛼 increases, and that the solid border span⁺(𝐇)^∗ increases when 𝛼 increases. For all 𝛼-values, both the shaded area and the solid border intersect with the dashed triangle. Therefore, it is not possible to get another solution by simply increasing/decreasing the desired solution. The figure shows that the NMF is unique for 𝛼∈{0.1,0.3} and not unique for 𝛼∈{0.5,0.7} where the alternative solution is shown by a dotted line. That the NMF is not unique for 𝛼∈{0.5,0.7} can also be verified by selecting the 𝐐 to be the symmetric orthonormal matrix 𝐐=𝐐^𝑇=𝐐⁻¹=13−1222−1222−1, (11) and see that both 𝐖𝐐 and 𝐐⁻¹𝐇 are nonnegative. If 𝛼=0.3, then the matrix 𝐑 is given by 1𝐑=100109603093010060109100309303010010960309930601091003030930100109601003093060109. (12) This shows that 𝐑 needs no zeros for the NMF to be unique. This completes the example.

(a) 𝛼=0.1

(b) 𝛼=0.3

(c) 𝛼=0.5

(d) 𝛼=0.7

Figure 2 

The figure shows data constructed as in Example 3 and plotted in the same manner as in Figure 1, that is, the dashed triangle is the desired solution, the solid line is the border of the dual of 𝐇, and the shaded area is the positive span of 𝐖. It can be seen that the NMF is unique when 𝛼 equals 0.1 or 0.3 but not when 𝛼 equals 0.5 or 0.7. In the cases where the NMF is not unique, an alternative solution is shown with a dotted line.

In the example above, 𝐖 equals 𝐇^𝑇 and thereby fulfils the same constraints. In many applications, the meaning of 𝐖 and 𝐇 differs, for example, in music analysis where the column vectors of 𝐖 are spectra of notes and 𝐇 is a note activity matrix [11].

Next, it is investigated how to make an asymmetric uniqueness constraint.

Definition 7. A set of vectors in ℝ^𝑑 is called sufficiently spread if for all 𝑗 and 𝑘>0, there is an element 𝐬∈𝒮 such that 𝐬_𝑗>𝑘_𝑖≠𝑗𝐬_𝑖. (13)

Note that in the definition for sufficiently spread set the 𝑗th element is larger than the sum in contrast to the strongly boundary close definition where the 𝑗th element is smaller than the sum.

Lemma 3. The dual space of a sufficiently spread set is the positive orthant.

Proof. A sufficiently spread set is nonnegative and the positive orthant is therefore part of the dual set for any sufficiently spread set. Let 𝐛 be a vector with a negative element in the 𝑗th element and select ∑𝑘=^𝑖≠𝑗|𝐛_𝑖|−𝐛_𝑗. (14) In any sufficiently spread set, an 𝐬 exists, such that 𝐬_𝑗>𝑘∑_𝑖≠𝑗𝐬_𝑖 and therefore 𝐬^𝑇𝐛=𝐬_𝑗𝐛_𝑗+_𝑖≠𝑗𝐬_𝑖𝐛_𝑖≤𝐬_𝑗𝐛_𝑗+(_𝑖≠𝑗𝐬_𝑖)(_𝑖≠𝑗|𝐛_𝑖|)=−𝐛_𝑗(−𝐬_𝑗+𝑘_𝑖≠𝑗𝐬_𝑖)<0. (15) The 𝐛 is therefore not in the dual to any sufficiently spread set.

In the case of finite sets, the sufficiently spread condition is the same as the requirement for a scaled version of all the standard basis vectors to be part of the sufficiently spread set. It is easy to verify that a sufficiently spread set also is strongly boundary close and that the 𝑧 parameter is one.

Theorem 5. If a pair [𝐖^𝑇,𝐇] is sufficiently spread and strongly boundary close, then the NMF of 𝐑=𝐖𝐇 is unique.

Proof. Lemma 3 states that the dual set of a sufficiently spread set is the positive orthant, span⁺(𝐇)^∗=𝒫=span⁺(𝐈)^∗. (16) Theorem 4 states that 𝐖𝐈 is unique and by using (16) and Theorem 1 we conclude that 𝐑=𝐖𝐇 is unique.

Theorem 5 is a stronger version of the results of Donoho and Stodden [6, Theorem 1]. Theorem 1 in [6] also assumes that 𝐇 is sufficiently spread, but the condition for 𝐖^𝑇 is stronger than the strongly boundary close assumption.

6. Perturbation Analysis

In the previous sections, we have analyzed situations with a unique solution. In this section, it is shown that in some situations the nonuniqueness can be seen as estimation noise on 𝐖 and 𝐇. The error function that describes how close an estimated [𝐖^′,𝐇]^′ pair is to the true [𝐖,𝐇] pair is 𝐽_(𝐖,𝐇)(𝐖^′,𝐇^′)=min_𝐏,𝐃(‖𝐖−𝐖^′(𝐃𝐏)‖_𝐹+‖𝐇−(𝐃𝐏)⁻¹𝐇^′‖_𝐹), (17) where 𝐏 is a permutation matrix and 𝐃 is a diagonal matrix.

Theorem 6. Let 𝐑=𝐖𝐇 be a unique NMF. Given some 𝜖>0, there exists a 𝛿>0 such that any nonnegative 𝐕=𝐑+𝐍, where ‖𝐍‖_𝐹<𝛿 fulfils 𝐽_(𝐖,𝐇)(𝐖^′,𝐇^′)<𝜖, (18) where [𝐖^′,𝐇^′]=𝑎𝑟𝑔𝑚𝑖𝑛_{𝐖′∈ℝ+𝑛×𝑟,𝐇′∈ℝ+𝑟×𝑚}‖𝐕−𝐖^′𝐇^′‖_𝐹. (19)

The proof is given in the appendix. The theorem states that if the observation is corrupted by additive noise, then it will result in noisy estimation of 𝐖 and 𝐇. Moreover, Theorem 6 shows that if the noise is small, it will result in small estimation errors. In this section, the Frobenius norm is used in (17) and (19) to make Theorem 6 concrete. Theorem 6 is also valid with the same proof if any continuous metric is used instead of the Frobenius norm in those equations. Example 4. This example investigates the connection between the additive noise in 𝐕 and the estimation error on 𝐖 and 𝐇. The column vectors in 𝐖 are basis pictures of a man, a dog, and the sun as shown in Figures 3(a), 3(b), and 3(c). In Figure 3(d), the sum of the three basis pictures is shown. The matrix 𝐇 is the set of all combinations of the pictures, that is, 𝐇=100011101010110011101. (20) Theorem 5 can be used to conclude that the NMF of 𝐑=𝐖𝐇 is unique because both 𝐖^𝑇 and 𝐇 are sufficiently spread and thereby also strongly boundary close.

(a)

(b)

(c)

(d)

Figure 3 

The three basis pictures: (a) a dog, (b) a man, and (c) the sun from Example 4, individually and summed in (d).

In the example, two different noise matrices, 𝐍_𝑁 and 𝐍_𝑀, are used. The 𝐍_𝑁 matrix models noisy observation and has elements that are random uniform i.i.d. The 𝐍_𝑀 matrix contains elements that are minus one in the positions where 𝐑 has elements that are two and zero elsewhere, that is, 𝐍_𝑀 is minus one in the positions where the dog and the man are overlapping. In this case, the error matrix 𝐍_𝑀 simulates a model mismatch that occurs in the following two types of real-world data. If the data set is composed of pictures, the basis pictures will be overlapping and a pixel in 𝐕 will consist of one basis picture and not a mixture of the overlapping pictures. If the data is a set of amplitude spectra, the true model is an addition of complex values and not an addition of the amplitudes.
The estimation error of the factorization 𝐽_(𝐖,𝐇)(𝐖^′,𝐇^′) is plotted in Figure 4 when the norm of the error matrix is 𝜇, that is, 𝐕=𝐖𝐇+(𝐍/‖𝐍‖_𝐅)𝜇. An estimate of the [𝐖^′,𝐇^′] pair is calculated by using the iterative algorithm for Frobenius norm minimized by Lee and Seung [2]. The algorithm is run for 500 iterations and is started from 100 different positions. The decomposition that minimizes ‖𝐕−𝐖^′𝐇^′‖_𝐹 is chosen, and 𝐽_(𝐖,𝐇)(𝐖^′,𝐇^′) is calculated numerically. Figure 4 shows that when the added error is small, it is possible to estimate the underlying parameters. When the norm of added noise matrix increases, the behavior of the two noise matrices, 𝐍_𝑁 and 𝐍_𝑀, differ. For 𝐍_𝑁, the error of the estimate increases slowly with the norm of the added matrix while the estimation error for 𝐍_𝑀 increases dramatically when the norm is larger than 2.5. In the simulation, we have made the following observation that can explain the difference in the performance of the two types of noise. When 𝐍_𝑁 is used, the basis pictures remain noisy versions of the man, the dog, and the sun. When 𝐍_𝑀 is used and the norm is larger than 2.5, the basis pictures are the man excluding the overlap, the dog excluding the overlap, and the overlap of man and dog. Another way to describe the difference is that the rank of 𝐍_𝑀 is one and the disturbance is in one dimension, where 𝐍_𝑁 is full rank and the disturbance is in many dimensions. This completes the example.

Figure 4 

The graph shows the connection between the norm of the additive error ‖𝐍‖𝐹 and the estimation error of the underlying model 𝐽(𝐖,𝐇)(𝐖′,𝐇′). The two noise matrices from Example 4, 𝐍𝑁 and 𝐍𝑀, are plotted. In this example, the curves are aligned for small errors, and for larger errors, the model error 𝐍𝑀 results in much larger estimation errors.

Corollary 1. Let 𝐑=𝐖𝐇 be a unique NMF and 𝐕=𝐖^{0𝑥002𝑑𝑐}𝐇^{0𝑥002𝑑𝑐}, where ^0𝑥𝐖^002𝑑𝑐=𝐖+𝐍_𝑊 and ^0𝑥0𝐇^02𝑑𝑐=𝐇+𝐍_𝐻. Given 𝐑 and 𝜖>0 there exists a 𝛿>0 such that if the largest absolute value of both 𝐍_𝑊 and 𝐍_𝐻 is smaller than 𝛿, then 𝐽_{(0𝑥002𝑑𝑐𝐖,0𝑥002𝑑𝑐𝐇)}(𝐖^′,𝐇^′)<𝜖, (21) where 𝐖^′,𝐇^′ are any NMF of 𝐕.Proof. This follows directly from Theorem 6. The corollary can be used in situations where there are small elements in 𝐖 and 𝐇 but no (or not enough) zero elements—as in the following example.

Example 5. Let 𝐑=𝐖𝐇, where 𝐖,𝐇 is generated as in Example 3. Let all elements in both 𝐍_𝑊 and 𝐍_𝐻 be equal to 𝜂. In Figure 5, 𝐕 is plotted when 𝛼=0.3 and 𝜂={0.01,0.05,0.10,0.15}. In this example, neither the shaded area nor the solid border intersect with the desired solution. Therefore, it is possible to get other solutions by simply increasing/decreasing the desired solution. For 𝜂={0.01,0.05}, the corners of the solutions are close to the corners of the desired solution. When 𝜂=0.1, the corners can be placed mostly on the solid border and still form a triangle that contains the shaded area. When 𝜂=0.15, the corners can be anywhere on the solid border. This completes the example.

(a) 𝜂=0.01

(b) 𝜂=0.05

(c) 𝜂=0.10

(d) 𝜂=0.15

Figure 5 

Data constructed as in Example 5 and plotted in the same manner as in Figure 1, that is, the dashed triangle is the desired solution, the solid line is the border of the dual of 𝐇, and the shaded area is the positive span of 𝐖. In all the plots, 𝛼 equals 0.3 and 𝜂 equals 0.01, 0.05, 0.1, and 0.15.

7. Probability and Uniqueness

In this section, the row vectors of 𝐖 and the column of 𝐇 are seen as results of two random variables. Characteristics of the sample space (the possible outcome) of a random variable that lead to unique NMF will be investigated.

Theorem 7. Let the row vectors of 𝐖 be generated by the random variable 𝒳_𝑊 and let the column vectors of 𝐇 be generated by a random variable 𝒳_𝐻. If the sample space of 𝒳_𝑊 is strongly boundary close and the sample space of 𝒳_𝐻 is sufficiently spread, then for all 𝜖>0 and 𝑘<1, there exist 𝑁_𝜖 and 𝑀_𝜖 such that 𝑝(min_𝐃,𝐏(‖𝐃𝐏𝐐−𝐈‖_𝐹)<𝜖)>𝑘, (22) where 𝐐 is any matrix such that 𝐖𝐐 and 𝐐⁻¹𝐇 are nonnegative and the data size 𝐑∈ℝ₊^𝑛×𝑚 is such that 𝑛>𝑁_𝜖 and 𝑚>𝑀_𝜖.

Proof. If the data is scaled, 𝐃₁𝐑𝐃₂, it does not change the nonuniqueness of the solutions when measured by the 𝐐 matrix. The proof is therefore done on the normalized versions of 𝐖 and 𝐇. Let 𝒴_𝑊 and 𝒴_𝐻 be the normalized version of 𝒳_𝑊 and 𝒳_𝐻. There exist finite sets 𝐖 and 𝐇 of vectors in the closure of 𝒴_𝑊 and 𝒴_𝐻 that are strongly boundary close and sufficiently spread. By Theorem 5, it is known that 𝐕=𝐖𝐇 is unique. By increasing the number of vectors sampled from 𝒴_𝑊 and 𝒴_𝐻, for any 𝜖^′>0, there will be two subsets of the vectors, 𝐖^′ and 𝐇^′, that with a probability larger that any 𝑘<1 will fulfil 𝜖^′>‖𝐖−𝐖^′‖_𝐹+‖𝐇−𝐇^′‖_𝐹. (23) It is possible to use Corollary 1 on this subset. The fact that limiting min_𝐃,𝐏(‖𝐃𝐏𝐐−𝐈‖_𝐹) is equivalent to limiting (21) when the vectors are normalized concludes the proof.

Example 6. Let all the elements in 𝐇 be exponential i.i.d. and therefore generated with a sufficiently spread sample space. Additionally, let each row in 𝐖 be exponential i.i.d. plus a random vector with the sample space {(⁰¹₁),(¹⁰₁),(¹¹₀)} and thereby strongly boundary close. In Figure 6, the above variables are shown for the following four matrix sizes 𝐑∈{ℝ^10×10,ℝ^40×40,ℝ^100×100,ℝ^500×500}. This completes the example.

(a) 𝑅[10×10]

(b) 𝑅[40×40]

(c) 𝑅[100×100]

(d) 𝑅[500×500]

Figure 6 

The figure shows data constructed as in Example 6 plotted in the same manner as the previous figure with the exception that each row vector of 𝐖 is plotted instead of the positive span of the vectors. The size of 𝐑 is shown under each plot.

8. Discussion

The approach in this paper is to investigate when nonnegativity leads to uniqueness in connection with NMF, 𝐕≈𝐑=𝐖𝐇. Nonnegativity is the only assumption for the theorems, and the theorems therefore cannot be used as argument for an NMF to be nonunique if there is additional information about 𝐖 or 𝐇. An example with stronger uniqueness results is the sparse NMF algorithm of Hoyer [12] built on the assumption that the row vectors in 𝐇 have known ratios between the 𝐿₁ norm and the 𝐿₂ norm. Theis et al. [13] have investigated uniqueness in this situation and shown strong uniqueness results. Another example is data matrices with an added constant on each row. For this situation, the affine NMF algorithm [14] can make NMF unique even though the setup violates Theorem 3 in this paper.

As shown in Figure 4, the type of noise greatly influences on the error curves. In applications where noise is introduced because the additive model does not hold as, for example, when 𝐕 is pictures or spectra, it is possible to influence the noise by making a nonlinear function on the elements of 𝐕. Such a nonlinear function is introduced in [15] and experiments show that it improves the results. A theoretical framework for finding good nonlinear functions will be interesting to investigate.

The sufficiently spread condition defined in Section 5 has an important role for unique NMF due to Lemma 3. The sufficiently spread assumption is seen indirectly in related areas where it also leads to unique solutions, for example, in [7] where the groundedness assumption leads to variables with a sufficiently spread sample space. If the matrix 𝐇 is sufficiently spread, then the columns in 𝐖 will occur (almost) alone as columns in 𝐕. Deville [16] uses the “occur alone” assumption, and thereby sufficiently spread assumption, to make blind source separation possible.

9. Conclusion

We have investigated the uniqueness of NMF from three different viewpoints as follows:

(i)uniqueness in noise free situations;(ii)the estimation error of the underlying model when a matrix with unique NMF is added with noise; and(iii)the random processes that lead to matrices where the underlying model can be estimated with small errors. By doing this, we have shown that it is possible to make many novel and useful characterizations that can be used as theoretical underpinning for using the numerous NMF algorithms. Several open issues can be found in all the three viewpoints that, if addressed, will give a better understanding of nonnegative matrix factorization.

Appendix

Proof of Theorem 4. The theorem state that 𝐖=𝐖𝐈 is a unique NMF. To proof this, it is shown that the condition for Theorem 1 is fulfilled. The positive orthant is self-dual (𝐈=𝐈⁻¹) and thereby 𝒬⊆𝒫, where 𝒬 is an 𝑟-order simplicial cone that contains span⁺(𝐖^𝑇). Let the set of row vectors in 𝐖 be denoted by 𝒲. An 𝑟-order simplicial cone, like 𝒬, is a closed set and it therefore needs to contain the closure of 𝒲 denoted by 𝒲. The two items in Definition 6 of strongly boundary close can be reformulated for 𝒲 that contains the border:
(1)𝐬^𝑗_𝑛=0 for all 𝑗,(2)the vectors [𝐛¹,…,𝐛^𝑑−𝑛] are linearly independent. The rest of the proof follows by induction. If 𝑟=2, then 𝒲=𝒫 and is therefore unique. Let therefore 𝑟>2. Then 𝑟−1 linearly independent vectors in 𝒲 have zero as the first element, and 𝑟−1 of the basis vectors therefore need to have zero in the first element. In other words, there is only one basis vector with a nonzero first element. Let us call this vector 𝐛¹. For all 𝑗>1 there is a vector in 𝒲 which is nonnegative in the first element and zero in the 𝑗th element, so all the elements in 𝐛¹ except the first have to be zero. The proof is completed by seeing that if the first element is removed from the vectors in 𝒲, it is still strongly boundary close and the problem is therefore the 𝑟−1 dimensional problem.

Proof of Theorem 6. Let 𝒢 be the open set of all 𝐖^′,𝐇^′ pairs that are close to 𝐖 and 𝐇, 𝒢={[𝐖^′,𝐇^′]∣𝐽_(𝐖,𝐇)(𝐖^′,𝐇^′)<𝜖}. (A.1) Let 𝒢 be the set of all nonnegative 𝐖^⌣,𝐇^⌣ pairs that are not in 𝒢 and where max(𝐖^⌣,𝐇^⌣)≤1+max(𝐑). The uniqueness of 𝐑 ensures that ‖𝐑−𝐖^⌣𝐇^⌣‖_𝐹>0, (A.2) for all [𝐖^⌣,𝐇^⌣]∈𝒢. The fact that the Frobenius norm is continuous, 𝒢 is a closed bounded set, and the statement above is positive ensures that min_{[⌣𝐖,⌣𝐇]∈𝒢}(‖𝐑−𝐖^⌣𝐇^⌣‖_𝐹)=𝛿^′>0, (A.3) since a continuous function attains its limits on a closed bounded set [17, Theorem 4.28]. The 𝐖^⌣,𝐇^⌣ pairs that are not in 𝒢 and where max(𝐖^⌣,𝐇^⌣)>1+max(𝐑) can either be transformed by a diagonal matrix into a matrix pair from 𝒢, [𝐖^⌣𝐃,𝐃^⌣−1𝐇]∈𝒢, having the same product (𝐖^⌣𝐇^⌣) or it can be transformed into a pair where both 𝐖^⌣ and 𝐇^⌣ have large elements, that is, max(𝐖^⌣𝐇^⌣)>1+max(𝐑)²=1+max(𝐑), (A.4) and thereby ‖𝐑−𝐖^⌣𝐇^⌣‖_𝐹>1.
Select 𝛿 to be 𝛿=min(1,𝛿^′)/2. The error of the desired solution 𝐑=𝐖𝐇 can be bounded by ‖𝐕−𝐑‖_𝐹=‖𝐍‖_𝐹<𝛿. Let 𝐕^⌣ be any matrix constructed by a nonnegative matrix pair not from 𝒢. Because of the way 𝛿 is selected, ‖𝐕^⌣−𝐑‖_𝐹≥2𝛿. By the triangle inequality, we get ‖𝐕^⌣−𝐕‖_𝐹+‖𝐕−𝐑‖_𝐹≥‖𝐕^⌣−𝐑‖_𝐹‖𝐕^⌣−𝐕‖_𝐹≥‖𝐕^⌣−𝐑‖_𝐹−‖𝐕−𝐑‖_𝐹>2𝛿−𝛿=𝛿>‖𝐕−𝐑‖_𝐹. (A.5) All solutions that are not in 𝒢 therefore have a larger error than 𝐖𝐇 and will not be the minimizer of the error.

Acknowledgments

This research was supported by the Intelligent Sound project, Danish Technical Research Council Grant no. 26-02-0092. The work of M. G. Christensen is supported by the Parametric Audio Processing project, Danish Research Council for Technology, and Production Sciences Grant no. 274-06-0521. Part of this work was previously presented at a conference [18].