Online Coregularization for Multiview Semisupervised Learning

Abstract

We propose a novel online coregularization framework for multiview semisupervised learning based on the notion of duality in constrained optimization. Using the weak duality theorem, we reduce the online coregularization to the task of increasing the dual function. We demonstrate that the existing online coregularization algorithms in previous work can be viewed as an approximation of our dual ascending process using gradient ascent. New algorithms are derived based on the idea of ascending the dual function more aggressively. For practical purpose, we also propose two sparse approximation approaches for kernel representation to reduce the computational complexity. Experiments show that our derived online coregularization algorithms achieve risk and accuracy comparable to offline algorithms while consuming less time and memory. Specially, our online coregularization algorithms are able to deal with concept drift and maintain a much smaller error rate. This paper paves a way to the design and analysis of online coregularization algorithms.

1. Introduction

Semi-supervised learning (S²L) is a relatively new subfield of machine learning which has become a popular research topic throughout the last two decades [1–6]. Different from standard supervised learning (SL), the S²L paradigm learns from both labeled and unlabeled examples. In this paper, we investigate the online semi-supervised learning (OS²L) problems with multiple views which have four features: (1) data is abundant, but the resources to label them are limited; (2) data arrives in a stream and cannot store them all; (3) the target functions in each view agree on labels of most examples (compatibility assumption); (4) the views are independent given the labels (independence assumption).

OS²L algorithms take place in a sequence of consecutive rounds. On each round, the learner is given a training example and is required to predict the label if the example is unlabeled. To label the examples, the learner uses a prediction mechanism which builds a mapping from the set of examples to the set of labels. The quality of an OS²L algorithm is measured by the cumulative loss it makes along its run. The challenge of OS²L is that we do not observe the true label for unlabeled examples to evaluate the performance of prediction mechanism. Thus, if we want to update the prediction mechanism, we have to rely on indirect forms of feedback.

Lots of OS²L algorithms have been proposed in recent years (see a survey in [7, 8]). A popular idea is defining an instantaneous risk function and decreasing its value in an online manner to avoid optimizing the primal semi-supervised problem directly [9–11]. References [12–14] also treat the OS²L problem as online semi-supervised clustering in that there are some must-links pairs (in the same cluster) and cannot-links pairs (cannot in the same cluster), but the effects of these methods are often influenced by “bridge points” (see a survey in [15]).

Coregularization [2, 16] is a method of improving the generalization accuracy of SVMs [17] by using unlabeled data in different views. Multiple hypotheses are trained in coregularization framework and are required to make similar predictions on any given unlabeled example. Moreover, theoretical investigations demonstrate that the coregularization approach reduces the Rademacher complexity by an amount that depends on the “distance” between the views [18, 19]. Unfortunately, basic offline coregularization algorithms are still unable to deal with long-playing large-scale OS²L problems directly because of the constraint of time and memory.

In this paper, we introduce a novel online coregularization framework for the design and analysis of new OS²L algorithms. Since decreasing the primal coregularization objective function is impossible before obtaining all the training examples, we propose a Fenchel conjugate transform to increase the dual problem incrementally. The existing online coregularization algorithms in previous work can be viewed as an approximation of the dual ascending process based on gradient ascent. New online coregularization algorithms are derived based on the idea of ascending the dual function more aggressively. We also discuss the applicability of our framework to the settings where the target hypothesis is not fixed but drifts with the sequence of examples.

To the best of our knowledge, the closest prior work is proposed by de Ruijter and Tsivtsivadze [10]. Their method defines an instantaneous regularized risk function using part of examples to avoid optimizing the primal coregularization problem directly. The learning process is based on convex programming with stochastic gradient descent in kernel space. The update scheme of this work can also be derived from our online coregularization framework.

The rest of the paper will be organized as follows. In Section 2 we begin with a primal view of multiview semi-supervised learning problem based on coregularization. In Section 3 our new framework for designing and analyzing online coregularization algorithms is introduced. Next, in Section 4, we demonstrate that the existing online coregularization algorithms can be derived from our framework using gradient ascent. New online coregularization algorithms are derived based on aggressive dual ascending procedures in Section 5. Experiments and analyses are in Section 6. In Section 7, conclusions and possible extensions of our work are given.

2. Basic Problem Setting

Our notation and problem setting are formally introduced in this section. The italic lower case letters refer to scalars (e.g., 𝛼 and 𝑤), and the bold letters refer to vectors (e.g., 𝝎 and 𝝀). (𝐱_𝑡,𝑦_𝑡,𝜎_𝑡) denotes the 𝑡th training example, where 𝐱_𝑡=(𝐱_𝑡⁽¹⁾,𝐱_𝑡⁽²⁾,…,𝐱_𝑡^(𝑠)) is seen in 𝑠 views with 𝐱_𝑡^(𝑖)∈𝑋^(𝑖) (𝑖∈{1,2,…,𝑠}), 𝑦_𝑡 is its label, and 𝜎_𝑡 is a flag to determine whether the label can be seen. If 𝜎=1, the example is labeled; and if 𝜎=0, the example is unlabeled. The hinge function is denoted by [𝑎]₊=max{𝑎,0}. ⟨𝝎,𝐱⟩ denotes the inner product between vectors 𝝎 and 𝐱.

In most cases, the hypotheses used for prediction come from a parameterized set of hypotheses, ℋ={ℎ_𝝎∶𝝎∈𝑉}, where 𝑉 is a subset of a vector space. For simplicity and concreteness, we focus on semi-supervised binary classification problems in two views in this paper. Consider an input sequence (𝐱₁^(1,2),𝑦₁,𝜎₁),(𝐱₂^(1,2),𝑦₂,𝜎₂),…,(𝐱_𝑇^(1,2),𝑦_𝑇,𝜎_𝑇), where 𝐱_𝑡^(1,2)=(𝐱_𝑡⁽¹⁾,𝐱_𝑡⁽²⁾), 𝑦_𝑡∈{−1,+1}, and 𝜎_𝑡∈{0,1} (𝑡∈{1,2,…,𝑇}). The goal is to learn the function pair 𝑓=(𝑓⁽¹⁾,𝑓⁽²⁾), where 𝑓⁽¹⁾=⟨𝝎⁽¹⁾,𝐱⁽¹⁾⟩ and 𝑓⁽²⁾=⟨𝝎⁽²⁾,𝐱⁽²⁾⟩ are linear classifiers in two views, respectively. To learn max-margin decision boundaries, the multiview S²L problem based on coregularization [16] can be written as minimizing: 𝐽𝝎⁽¹⁾,𝝎⁽²⁾=12𝛾₁𝝎^(1)𝟐+12𝛾₂𝝎^(2)𝟐+^𝑇_𝑡=1𝜎_𝑡1−𝑦_𝑡𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾₊+𝜇1−𝑦_𝑡𝝎⁽²⁾,𝐱_𝑡⁽²⁾₊+^𝑇_𝑡=11−𝜎_𝑡𝛾_𝑐𝑑𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾,𝝎⁽²⁾,𝐱_𝑡⁽²⁾, (1) where 𝛾₁ and 𝛾₂ are trade-off parameters for complexity functions in the two views, 𝜇 is a real-valued parameter to balance data fitting, 𝛾_𝑐 is the coupling parameter that regularizes the pair towards compatibility using unlabeled data, and 𝑑(⋅,⋅) is the distance function which measures the difference between the predictions for the same example and composes the multiview coregularizer.

In previous approaches based on coregularization [16, 19], the distance function 𝑑(⋅,⋅) is often defined as a square function: 𝑑𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾,𝝎⁽²⁾,𝐱_𝑡⁽²⁾=𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾,𝐱_𝑡⁽²⁾². (2) The distance function is defined as an absolute function (using 𝑙₁ norm) in this paper (this idea is also adopted by Szedmak and Shawe-Taylor [18] and Sun et al. [20]): 𝑑𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾,𝝎⁽²⁾,𝐱_𝑡⁽²⁾=𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾,𝐱_𝑡⁽²⁾. (3) Furthermore, (3) is composed of two hinge functions (see Figure 1 for an illustration) 𝑑=𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾,𝐱_𝑡⁽²⁾₊+𝝎⁽²⁾,𝐱_𝑡⁽²⁾−𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾₊. (4) In the next section, we will show that the online coregularization problem can be discussed in the dual form of (1) more easily and directly while using the absolute distance function.

(a)

(b)

Figure 1 

Two different distance functions and hinge functions. The top plane shows the two different distance function. The bottom plane shows that the absolute function |𝐱| can be decomposed into two hinge functions [𝐱]+ and [−𝐱]+.

Denote the instantaneous loss on round 𝑡 as 𝑔_𝑡𝝎⁽¹⁾,𝝎⁽²⁾=𝜎_𝑡1−𝑦_𝑡𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾₊+𝜇1−𝑦_𝑡𝝎⁽²⁾,𝐱_𝑡⁽²⁾₊+1−𝜎_𝑡𝛾_𝑐𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾,𝐱_𝑡⁽²⁾, (5) where 𝑡∈{1,2,…,𝑇}. We thus get a simple version of (1) using (5): 𝐽𝝎⁽¹⁾,𝝎⁽²⁾=12𝛾₁𝝎^(1)𝟐+12𝛾₂𝝎^(2)𝟐+^𝑇_𝑡=1𝑔_𝑡𝝎⁽¹⁾,𝝎⁽²⁾. (6)

The minimization problem of (6) in an online manner is what we consider in the rest of this paper.

3. Online Coregularization by Ascending the Dual Function

In this section, we propose a unified online coregularization framework for multi-view semi-supervised binary classification problems. Our presentation reveals how the multiview S²L problem based on coregularization in Section 2 can be optimized in an online manner.

Before describing our framework, let us recall the definition of Fenchel conjugate that we use as a main analysis tool in this paper (see the appendix for more details). The Fenchel conjugate of a function 𝑓∶dom𝑓→ℝ is defined as 𝑓^∗(𝝀)=sup⟨𝝀,𝝎⟩−𝑓(𝝎)∶𝝎∈dom𝑓. (7)

As shown in (7), the Fenchel conjugate is defined only for single variable function in former convex analysis. We extend the definition of Fenchel conjugate to multivariables functions for solving online multiview S²L problem in this paper.

The Fenchel conjugate of a multivariables function 𝑓(𝝎⁽¹⁾,𝝎⁽²⁾,…,𝝎^(𝑠)) (𝝎⁽¹⁾∈𝑋⁽¹⁾,𝝎⁽²⁾∈𝑋⁽²⁾,…,𝝎^(𝑠)∈𝑋^(𝑠)) is defined as 𝑓^∗𝝀⁽¹⁾,𝝀⁽²⁾,…,𝝀^(𝑠)=sup^𝑠_𝑖=1𝝀^(𝑖),𝝎^(𝑖)−𝑓𝝎⁽¹⁾,𝝎⁽²⁾,…,𝝎^(𝑠)∶𝝎⁽¹⁾∈𝑋⁽¹⁾,𝝎⁽²⁾∈𝑋⁽²⁾,…,𝝎^(𝑠)∈𝑋^(𝑠). (8) Specially, the Fenchel conjugate of hinge functions is a key to transfer coregularization from offline to online in this paper.

Proposition 1. Let 𝑓(𝝎⁽¹⁾,𝝎⁽²⁾)=[𝑏−(⟨𝝎⁽¹⁾,𝐱⁽¹⁾⟩−⟨𝝎⁽²⁾,𝐱⁽²⁾⟩)]₊, where 𝝎⁽¹⁾∈𝑋⁽¹⁾, 𝝎⁽²⁾∈𝑋⁽²⁾, and 𝑏∈ℝ. The Fenchel conjugate of 𝑓(𝝎⁽¹⁾,𝝎⁽²⁾) is 𝑓^∗𝝀⁽¹⁾,𝝀⁽²⁾=−𝛼𝑏𝑖𝑓𝝀⁽¹⁾=−𝛼𝐱⁽¹⁾,𝝀⁽²⁾=𝛼𝐱⁽²⁾,𝑤ℎ𝑒𝑟𝑒𝛼∈[0,1]∞𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. (9)

Proof. We first rewrite the 𝑓(𝝎⁽¹⁾,𝝎⁽²⁾) as follows: 𝑓𝝎⁽¹⁾,𝝎⁽²⁾=𝑏−𝝎⁽¹⁾,𝐱⁽¹⁾−𝝎⁽²⁾,𝐱⁽²⁾₊=max_[]𝛼∈0,1𝛼𝑏−𝝎⁽¹⁾,𝐱⁽¹⁾−𝝎⁽²⁾,𝐱⁽²⁾. (10) Based on the definition of Fenchel conjugate for multivariables functions, we can obtain that 𝑓^∗𝝀⁽¹⁾,𝝀⁽²⁾=max_{𝝎(1),𝝎(2)}𝝀⁽¹⁾,𝝎⁽¹⁾+𝝀⁽²⁾,𝝎⁽²⁾−𝑓𝝎⁽¹⁾,𝝎⁽²⁾=max_{𝝎(1),𝝎(2)}𝝀⁽¹⁾,𝝎⁽¹⁾+𝝀⁽²⁾,𝝎⁽²⁾−max_[]𝛼∈0,1𝛼𝑏−𝝎⁽¹⁾,𝐱⁽¹⁾−𝝎⁽²⁾,𝐱⁽²⁾=max_{𝝎(1),𝝎(2)}min_[]𝛼∈0,1𝝀⁽¹⁾,𝝎⁽¹⁾+𝝀⁽²⁾,𝝎⁽²⁾−𝛼𝑏−𝝎⁽¹⁾,𝐱⁽¹⁾−𝝎⁽²⁾,𝐱⁽²⁾=min_[]𝛼∈0,1max_{𝝎(1),𝝎(2)}−𝛼𝑏+𝝀⁽¹⁾+𝛼𝐱⁽¹⁾,𝝎⁽¹⁾+𝝀⁽²⁾−𝛼𝐱⁽²⁾,𝝎⁽²⁾=min_[]𝛼∈0,1−𝛼𝑏+max_𝝎(1)𝝀⁽¹⁾+𝛼𝐱⁽¹⁾,𝝎⁽¹⁾+max_𝝎(2)𝝀⁽²⁾−𝛼𝐱⁽²⁾,𝝎⁽²⁾. (11)
Since the previous third equality follows from the strong max-min property, it can be transferred into a min-max problem. If 𝝀⁽¹⁾+𝛼𝐱⁽¹⁾≠0, max_𝝎(1)⟨𝝀⁽¹⁾+𝛼𝐱⁽¹⁾,𝝎⁽¹⁾⟩ is ∞, and if 𝝀⁽²⁾−𝛼𝐱⁽²⁾≠0, max_𝝎(2)⟨𝝀⁽²⁾−𝛼𝐱⁽²⁾,𝝎⁽²⁾⟩ is ∞; otherwise, if 𝝀⁽¹⁾+𝛼𝐱⁽¹⁾=0 and 𝝀⁽²⁾−𝛼𝐱⁽²⁾=0, we have 𝑓^∗(𝝀⁽¹⁾,𝝀⁽²⁾)=−𝛼𝑏.

Back to the primal problem, we want to get a sequence of boundary 𝝎₀^(1,2),𝝎₁^(1,2),…,𝝎_𝑇^(1,2)(𝝎_𝑡^(1,2)=(𝝎_𝑡⁽¹⁾,𝝎_𝑡⁽²⁾), 𝑡∈{0,1,…,𝑇}) which makes 𝐽(𝝎₀^(1,2))≥𝐽(𝝎₁^(1,2))≥⋯≥𝐽(𝝎_𝑇^(1,2)). Unfortunately, decreasing the objective function 𝐽(𝝎⁽¹⁾,𝝎⁽²⁾) directly is impossible, while the training examples arrive in a steam. In order to avoid this contradiction, we propose a Fenchel conjugate transform for multi-view S²L problems based on coregularization.

An equivalent problem of (6) is min_{𝝎0(1,2),𝝎1(1,2),…,𝝎𝑇(1,2)}12𝛾₁𝝎₀^(1)𝟐+12𝛾₂𝝎₀^(2)𝟐+^𝑇_𝑡=1𝑔_𝑡𝝎_𝑡^(1,2),s.t.∀𝑖∈1,2,…,𝑇,𝝎_𝑖⁽¹⁾=𝝎₀⁽¹⁾,𝝎_𝑖⁽²⁾=𝝎₀⁽²⁾. (12)

Using the Lagrange dual function, we can rewrite (12) by introducing a vector group (𝝀₁^(1,2),𝝀₂^(1,2),…,𝝀_𝑇^(1,2)): max_{𝝀1(1,2),𝝀2(1,2),…,𝝀𝑇(1,2)}min_{𝝎0(1,2),𝝎1(1,2),…,𝝎𝑇(1,2)}12𝛾₁𝝎₀^(1)𝟐+12𝛾₂𝝎₀^(2)𝟐+^𝑇_𝑡=1𝑔_𝑡𝝎_𝑡^(1,2)+^𝑇_𝑡=1𝝀₁⁽¹⁾,𝝎₀⁽¹⁾−𝝎_𝑡⁽¹⁾+^𝑇_𝑡=1𝝀₁⁽²⁾,𝝎₀⁽²⁾−𝝎_𝑡⁽²⁾. (13) Consider the dual function 𝐷𝝀₁^(1,2),𝝀₂^(1,2),…,𝝀_𝑇^(1,2)=min_{𝝎0(1,2),𝝎1(1,2),…,𝝎𝑇(1,2)}12𝛾₁𝝎₀^(1)𝟐+12𝛾₂𝝎₀^(2)𝟐+^𝑇_𝑡=1𝑔_𝑡𝝎_𝑡^(1,2)+^𝑇_𝑡=1𝝀_𝑡⁽¹⁾,𝝎₀⁽¹⁾−𝝎_𝑡⁽¹⁾+^𝑇_𝑡=1𝝀_𝑡⁽²⁾,𝝎₀⁽²⁾−𝝎_𝑡⁽²⁾=−max_𝝎0(1)^𝑇_𝑡=1−𝝀_𝑡⁽¹⁾,𝝎₀⁽¹⁾−12𝛾₁𝝎₀^(1)𝟐−max_𝝎0(2)^𝑇_𝑡=1−𝝀_𝑡⁽²⁾,𝝎₀⁽²⁾−12𝛾₂𝝎₀^(2)𝟐−^𝑇_𝑡=1max_{𝝀𝑡(1,2)}𝝀_𝑡⁽¹⁾,𝝎_𝑡⁽¹⁾+𝝀_𝑡⁽²⁾,𝝎_𝑡⁽²⁾−𝑔_𝑡𝝎_𝑡^(1,2)1=−2𝛾₁−^𝑇_𝑡=1𝝀_𝑡⁽¹⁾²−12𝛾₂−^𝑇_𝑡=1𝝀_𝑡⁽²⁾²−^𝑇_𝑡=1𝑔^∗_𝑡𝝀_𝑡^(1,2), (14) where 𝑔^∗_𝑡(𝝀_𝑡^(1,2)) is the Fenchel conjugate of 𝑔_𝑡(𝝎_𝑡^(1,2)). The primal problem can be described as maximizing the dual function as in the following min_𝝎(1,2)𝐽𝝎^(1,2)=max_{𝝀1(1,2),𝝀2(1,2),…,𝝀𝑇(1,2)}𝐷𝝀₁^(1,2),𝝀₂^(1,2),…,𝝀_𝑇^(1,2). (15)

Based on our definition of Fenchel conjugate for multivariables functions, the Fenchel conjugate of 𝑔_𝑡(𝝎_𝑡^(1,2)) can be rewritten as (based on Proposition 1 and Lemma A.2 in the appendix) 𝑔^∗_𝑡𝝀_𝑡^(1,2)=−𝜎_𝑡𝛼_𝑡1+𝜇𝛼_𝑡2,if𝝀_𝑡⁽¹⁾=−𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾−1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽¹⁾,𝝀_𝑡⁽²⁾=−𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾+1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽²⁾,where𝛼_𝑡1,𝛼_𝑡2∈[0,1],𝛼_𝑡0∈[−1,1];∞,otherwise. (16) Since our goal is to maximize the dual function, we can restrict to the first case in (16). 𝑔^∗_𝑡(𝝀_𝑡^(1,2)) has 3 associated coefficient variables which are 𝛼_𝑡0,𝛼_𝑡1, and 𝛼_𝑡2.

Based on the previous analysis, the dual function can be rewritten using a new coefficient vectors group 𝜶₁,𝜶₂,…,𝜶_𝑇, where 𝜶_𝑡=[𝛼_𝑡0,𝛼_𝑡1,𝛼_𝑡2](𝑡∈{1,2,…,𝑇}). Consider the following: 𝐷𝜶₁,𝜶₂,…,𝜶_𝑇1=−2𝛾₁^𝑇_𝑡=1𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾+1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽¹⁾²−12𝛾₂^𝑇_𝑡=1𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾−1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽²⁾²+^𝑇_𝑡=1𝜎_𝑡𝛼_𝑡1+𝜇𝛼_𝑡2. (17)

As shown in (17), our task has been transferred to a constrained quadratic programming (QP) optimization problem. Every input training example brings a vector 𝜶 into the dual function. 𝜶₁,𝜶₂,…,𝜶_𝑇 are independent, so we can update the vectors group (𝜶₁,𝜶₂,…,𝜶_𝑇) on each learning round to ascend the dual problem incrementally. Obviously, unobserved examples would make no influence on the value of dual function in (17) by setting their associated coefficient variables to zero.

Denote (𝜶_𝑖)_𝑡 the coefficient vector 𝜶_𝑖 on round 𝑡 (𝑖∈{1,2,…,𝑇}). The update process of coefficient vectors group (𝜶₁,𝜶₂,…,𝜶_𝑇) on round 𝑡 should satisfy the following conditions:(i)if 𝑡+1≤𝑖≤𝑇, (𝛼_𝑖0)_𝑡,(𝛼_𝑖1)_𝑡,(𝛼_𝑖2)_𝑡=0;(ii)𝐷((𝜶₁)_𝑡,(𝜶₂)_𝑡,…,(𝜶_𝑇)_𝑡)≥𝐷((𝜶₁)_𝑡−1,(𝜶₂)_𝑡−1,…,(𝜶_𝑇)_𝑡−1).

The first one means that the unobserved examples do not make influence on the value of dual function, and the second means that the value of dual function never decreases during the online coregularization process. Therefore, the dual function on round 𝑡 can also be written as 𝐷𝜶_1𝑡,𝜶_2𝑡,…,𝜶_𝑇𝑡1=−2𝛾₁^𝑡_𝑖=1𝜎_𝑖𝑦_𝑖𝛼_𝑖1𝑡𝐱_𝑖⁽¹⁾+1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐𝐱_𝑖⁽¹⁾²−12𝛾₂^𝑡_𝑖=1𝜎_𝑖𝑦_𝑖𝜇𝛼_𝑖2𝑡𝐱_𝑖⁽²⁾−1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐𝐱_𝑖⁽²⁾²+^𝑡_𝑖=1𝜎_𝑖𝛼_𝑖1𝑡+𝜇𝛼_𝑖2𝑡. (18)

Based on Lemmas A.1 and A.3 in the appendix, we can obtain that each coefficient vectors group (𝜶₁,𝜶₂,…,𝜶_𝑇) has an associated boundary vector group (𝝎⁽¹⁾,𝝎⁽²⁾). On round 𝑡, the associated boundaries of ((𝜶₁)_𝑡,(𝜶₂)_𝑡,…,(𝜶_𝑇)_𝑡) are 𝝎_𝑡⁽¹⁾=1𝛾₁^𝑡_𝑖=1𝜎_𝑖𝑦_𝑖𝛼_𝑖1𝑡𝐱_𝑖⁽¹⁾+1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐𝐱_𝑖⁽¹⁾,𝝎_𝑡⁽²⁾=1𝛾₂^𝑡_𝑖=1𝜎_𝑖𝜇𝑦_𝑖𝛼_𝑖2𝑡𝐱_𝑖⁽²⁾−1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐𝐱_𝑖⁽²⁾. (19)

To make a summary, we propose a template online coregularization algorithm by dual ascending procedure in Algorithm 1.

INPUT: four positive scalars: 𝛾1, 𝛾2, 𝛾𝑐 and 𝜇. INITIALIZE: a coefficient vectors group 𝜶10,𝜶20,…,𝜶𝑇0 and its associated      decision boundary vectors 𝝎0(1), 𝝎0(2). PROCESS: For 𝑡=1,2,…,𝑇       Receive an example (𝐱𝑡(1,2),𝑦𝑡,𝜎𝑡),       Choose a new coefficient vectors group 𝜶1𝑡,𝜶2𝑡,…,𝜶𝑇𝑡 that satisfies          𝐷𝜶1𝑡,𝜶2𝑡,…,𝜶𝑇𝑡≥𝐷𝜶1𝑡−1,𝜶2𝑡−1,…,𝜶𝑇𝑡−1,       Return two new associated decision boundary vectors 𝝎𝑡(1), 𝝎𝑡(2)       using (19).

Algorithm 1 

A template online co-regularization algorithm for multiview semi-supervised binary classification problems. This template algorithm aims for increasing the dual function on each learning round.

Essentially, our online coregularization framework aims to break the large QP in the primal objective function into a series of dual ascending procedures on each learning round. Therefore, we can ascend the dual function in an online manner.

4. Analysis of Previous Work Based on Gradient Ascent in the Dual

In the previous section, a template algorithm framework for online coregularization is proposed based on the idea of ascending the dual function. In Algorithm 1, we can obtain that algorithms that derive from our framework may vary in one of two ways. First, different algorithms may update different dual variables on each learning round. The second way in which different algorithms may vary is how to update the chosen variables to ascend the dual function.

Some online coregularization algorithms [9, 10] have been suggested in recent years. These approaches have a similar idea of “defining an instantaneous coregularized risk to avoid optimizing the primal coregularization problem directly.” In these works, there are two popular instantaneous coregularized risk functions 𝐽_𝑡(𝝎⁽¹⁾,𝝎⁽²⁾) which are defined as 𝐽_𝑡𝝎⁽¹⁾,𝝎⁽²⁾=𝜎_𝑡1−𝑦_𝑡𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾₊+𝜇1−𝑦_𝑡𝝎⁽²⁾,𝐱_𝑡⁽²⁾₊+1−𝜎_𝑡𝛾_𝑐𝑑𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾,𝝎⁽²⁾,𝐱_𝑡⁽²⁾,𝐽_𝑡𝝎⁽¹⁾,𝝎⁽²⁾=12𝛾₁𝝎^(1)𝟐+12𝛾₂𝝎^(2)𝟐+𝜎_𝑡1−𝑦_𝑡𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾₊+𝜇1−𝑦_𝑡𝝎⁽²⁾,𝐱_𝑡⁽²⁾₊+1−𝜎_𝑡𝛾_𝑐𝑑𝝎⁽¹⁾,𝐱_𝑡⁽¹⁾,𝝎⁽²⁾,𝐱_𝑡⁽²⁾. (20) The online coregularization process in these works is based on convex programming with gradient descent on instantaneous coregularized risk function in kernel space. The step size 𝜌_𝑡 is often defined to decay at a certain rate [11], for example, 𝜌_𝑡=1/√𝑡. The update process in these approaches can be summarized as 𝝎_𝑡^(1,2)=𝝎^(1,2)_𝑡−1−𝜌_𝑡𝜕𝐽_𝑡𝝎^(1,2)𝜕𝝎^(1,2)_{𝝎(1,2)𝑡−1}. (21)

In the following, we demonstrate that these algorithms can be derived from our online coregularization framework. Since the dual coefficient vectors 𝜶₁,𝜶₂,…,𝜶_𝑇 are independent, the dual function can be ascended by updating only the associated coefficient vector 𝜶_𝑡 of the new arrived training example (𝐱_𝑡^(1,2),𝑦_𝑡,𝜎_𝑡) on round 𝑡 that means 𝜶_𝑖𝑡=𝜶_𝑖𝑡−1,if𝑖≠𝑡. (22) And the task on round 𝑡 can be rewritten as ascending 𝐷_𝑡𝜶_𝑡1=−2𝛾₁𝛾₁𝝎⁽¹⁾_𝑡−1+𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾+1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽¹⁾²−12𝛾₂𝛾₂𝝎⁽²⁾_𝑡−1+𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾−1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽²⁾²+^𝑡_𝑖=1𝜎_𝑖𝛼_𝑖1𝑡+𝜇𝛼_𝑖2𝑡. (23)

Using a gradient ascent (GA) step on 𝜶_𝑡, the update process on round 𝑡 can be written as 𝛼_𝑡0𝑡=𝜌_𝑡𝜕𝐷_𝑡𝜶_𝑡𝜕𝛼_{𝑡0𝜶𝑡=𝟎}=−𝜌_𝑡1−𝜎_𝑡𝛾_𝑐𝝎⁽¹⁾_𝑡−1,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾_𝑡−1,𝐱_𝑡⁽²⁾,𝛼_𝑡1𝑡=𝜌_𝑡𝜕𝐷_𝑡𝜶_𝑡𝜕𝛼_{𝑡1𝜶𝑡=𝟎}=𝜌_𝑡𝜎_𝑡1−𝑦_𝑡𝝎⁽¹⁾_𝑡−1,𝐱_𝑡⁽¹⁾₊,𝛼_𝑡2𝑡=𝜌_𝑡𝜕𝐷_𝑡𝜶_𝑡𝜕𝛼_{𝑡2𝜶𝑡=𝟎}=𝜌_𝑡𝜎_𝑡𝜇1−𝑦_𝑡𝝎⁽²⁾_𝑡−1,𝐱_𝑡⁽²⁾₊, (24) where 𝜌_𝑡≥0 is a step size.

In fact, the dual coefficient vectors in 𝝎^(1,2)_𝑡−1 can also be updated in (23). Since 𝝎^(1,2)_𝑡−1 has 𝑡−1 dual coefficient vectors, it is impossible to update them, respectively. We introduce a new variable 𝜂_𝑡 into (23), 𝐷_𝑡𝜶_𝑡,𝜂_𝑡1=−2𝛾₁𝛾₁1−𝜂_𝑡𝝎⁽¹⁾_𝑡−1+𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾+1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽¹⁾²−12𝛾₂𝛾₂1−𝜂_𝑡𝝎⁽²⁾_𝑡−1+𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾−1−𝜎_𝑡𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽²⁾²+^𝑡−1_𝑖=1𝜎_𝑖1−𝜂_𝑡𝛼_𝑖1𝑡+𝜇1−𝜂_𝑡𝛼_𝑖2𝑡+𝜎_𝑖𝛼_𝑡1𝑡+𝜇𝛼_𝑡2𝑡. (25)

From (25), we can get that a gradient ascent update on 𝜂_𝑡 actually means to multiply all the dual coefficient vectors in 𝝎^(1,2)_𝑡−1 by 1−𝜂_𝑡. Since every dual coefficient variable in 𝝎^(1,2)_𝑡−1 is constrained, we also constrain 𝜂_𝑡∈[0,1]. The initial value of 𝜂_𝑡 is zero. While using a gradient ascent on 𝜂_𝑡, we obtain that 𝜂_𝑡=𝜌_𝑡𝛾₁𝝎⁽¹⁾_𝑡−1,𝝎⁽¹⁾_𝑡−1+𝛾₂𝝎⁽²⁾_𝑡−1,𝝎⁽²⁾_𝑡−1−^𝑡−1_𝑖=1𝜎_𝑖𝛼_𝑖1𝑡−^𝑡−1_𝑖=1𝜎_𝑖𝜇𝛼_𝑖2𝑡+. (26)

Based on the previous analysis, the gradient ascent process of boundary vector group (𝝎⁽¹⁾,𝝎⁽²⁾) can be written as 𝝎_𝑡⁽¹⁾=1−𝜂_𝑡𝝎⁽¹⁾_𝑡−1+1𝛾₁𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝑡𝐱_𝑡⁽¹⁾+1−𝜎_𝑡𝛼_𝑡0𝑡𝛾_𝑐𝐱_𝑡⁽¹⁾,𝝎_𝑡⁽²⁾=1−𝜂_𝑡𝝎⁽²⁾_𝑡−1+1𝛾₂𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝑡𝐱_𝑡⁽²⁾−1−𝜎_𝑡𝛼_𝑡0𝑡𝛾_𝑐𝐱_𝑡⁽²⁾. (27)

As far as we know, all the existing online coregularization algorithms in previous work can be viewed as an approximation of our dual ascending process using gradient ascent in (27).

5. Deriving New Algorithms Based on Aggressive Dual Ascending (ADA) Procedures

In the previous section, we show that the online coregularization algorithms in previous work can be derived from our framework. These algorithms lead to a conservative increase of the value of the dual function since they only modify a single dual vector using gradient ascent on each learning round. In fact, more aggressive online coregularization algorithms can also be derived from our framework.

In this section we describe broader and, in practice, more powerful online coregularization algorithms which increase the dual function more aggressively on each learning round. The motivation for the new algorithms is as follows. Intuitively, update schemes that yield larger increases of the dual function are likely to reach the minimal value of primal objective function faster. Thus, they are in practice likely to suffer a smaller number of mistakes.

5.1. Updating Single Dual Coefficient Vector

The update scheme we described in Section 4 for increasing the dual function modifies the associated coefficient vector of the new arrived training example which is based on gradient ascent, and all the variables in the vector share a same step size. This simple algorithm can be enhanced by solving the following optimization problem on each learning round max_{𝛼𝑡1,𝛼𝑡2∈[]0,1,𝛼𝑡0∈[]−1,1}𝐷𝜶₁,𝜶₂,…,𝜶_𝑇s.t.∀𝑖≠𝑡,𝜶_𝑖=𝜶_𝑖𝑡−1. (28)

According to the type of the new arrived example, (28) can be solved in different ways. If the new arrived example is labeled, we have 𝜎_𝑡=1, and the task on round 𝑡 can be rewritten as max_{𝛼𝑡1,𝛼𝑡2∈[]0,1}−12𝛾₁𝛾₁𝝎⁽¹⁾_𝑡−1+𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾²−12𝛾₂𝛾₂𝝎⁽²⁾_𝑡−1+𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾²+𝛼_𝑡1+𝜇𝛼_𝑡2=max_{𝛼𝑡1∈[]0,1}−12𝛾₁𝛾₁𝝎⁽¹⁾_𝑡−1+𝑦_𝑡𝛼_𝑡1𝐱_𝑡⁽¹⁾²+𝛼_𝑡1+max_{𝛼𝑡2∈[]0,1}−12𝛾₂𝛾₂𝝎⁽²⁾_𝑡−1+𝜇𝑦_𝑡𝛼_𝑡2𝐱_𝑡⁽²⁾²+𝜇𝛼_𝑡2. (29) Since 𝛼_𝑡1,𝛼_𝑡2∈[0,1], we can obtain that 𝛼_𝑡1𝑡𝛾=min¹1−𝑦_𝑡𝝎⁽¹⁾_𝑡−1,𝐱_𝑡⁽¹⁾₊𝐱_𝑡⁽¹⁾,𝐱_𝑡⁽¹⁾,𝛼,1_𝑡2𝑡𝛾=min²1−𝑦_𝑡𝝎⁽²⁾_𝑡−1,𝐱_𝑡⁽²⁾₊𝜇𝐱_𝑡⁽²⁾,𝐱_𝑡⁽²⁾,1. (30) Otherwise, if the new arrived example is unlabeled, we have 𝜎_𝑡=0, and (28) can be rewritten as max_{𝛼𝑡0∈[]−1,1}−12𝛾₁𝛾₁𝝎⁽¹⁾_𝑡−1+𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽¹⁾²−12𝛾₂𝛾₂𝝎⁽²⁾_𝑡−1−𝛼_𝑡0𝛾_𝑐𝐱_𝑡⁽²⁾². (31) Since 𝛼_𝑡0∈[−1,1], we can obtain that 𝛼_𝑡0𝑡=min−1/𝛾_𝑐𝝎⁽¹⁾_𝑡−1,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾_𝑡−1,𝐱_𝑡⁽²⁾1/𝛾₂𝐱_𝑡⁽²⁾,𝐱_𝑡⁽²⁾+1/𝛾₁𝐱_𝑡⁽¹⁾,𝐱_𝑡⁽¹⁾,𝛼,1if_𝑡0𝑡−≥0max1/𝛾_𝑐𝝎⁽¹⁾_𝑡−1,𝐱_𝑡⁽¹⁾−𝝎⁽²⁾_𝑡−1,𝐱_𝑡⁽²⁾1/𝛾₂𝐱_𝑡⁽²⁾,𝐱_𝑡⁽²⁾+1/𝛾₁𝐱_𝑡⁽¹⁾,𝐱_𝑡⁽¹⁾𝛼,−1if_𝑡0𝑡<0. (32)

Based on the previous analysis, the update process of boundary vector group (𝝎⁽¹⁾,𝝎⁽²⁾) can be written as 𝝎_𝑡⁽¹⁾=𝝎⁽¹⁾_𝑡−1+1𝛾₁×𝜎_𝑡𝑦_𝑡𝛼_𝑡1𝑡𝐱_𝑡⁽¹⁾+1−𝜎_𝑡𝛼_𝑡0𝑡𝛾_𝑐𝐱_𝑡⁽¹⁾,𝝎_𝑡⁽²⁾=𝝎⁽²⁾_𝑡−1+1𝛾₂×𝜎_𝑡𝜇𝑦_𝑡𝛼_𝑡2𝑡𝐱_𝑡⁽²⁾−1−𝜎_𝑡𝛼_𝑡0𝑡𝛾_𝑐𝐱_𝑡⁽²⁾. (33)

In contrast to the gradient approaches in Section 4, this approach ascends the dual function more aggressively. So far, our focus was on an update which modifies a single dual coefficient vector. In fact, all the associated coefficient vectors of the arrived examples can be updated during the online regularization process. We now examine another update scheme based on our online coregularization framework that suggests the modification of multiple dual coefficient vectors on each learning round.

5.2. Updating Multiple Dual Coefficient Vectors

We now describe a new update scheme which modifies multiple dual coefficient vectors on each learning round. In practice, we can get the example set {(𝐱₁^(1,2),𝑦₁,𝜎₁),(𝐱₂^(1,2),𝑦₂,𝜎₂),…,(𝐱_𝑡^(1,2),𝑦_𝑡,𝜎_𝑡)} on round 𝑡, and all the associated coefficient vectors of these examples can be updated to ascend the dual function.

Denote 𝑆_𝑡⊆[𝑡] the buffer on round 𝑡, where 𝑡∈𝑆_𝑡 and [𝑡]={1,2,…,𝑡}. If 𝑖∈𝑆_𝑡, the associated coefficient vector of the 𝑖th example can be updated on round 𝑡. We optimize the dual function over the dual coefficient vectors whose indices belong to 𝑆_𝑡. Consider the following: max_{𝜶𝑖𝑖∈𝑆𝑡}𝐷𝜶₁,𝜶₂,…,𝜶_𝑇s.t.∀𝑖∉𝑆_𝑡,𝜶_𝑖=𝜶_𝑖𝑡−1. (34) Obviously, this more general update scheme could make a larger increase on each learning round. If setting 𝑆_𝑡=[𝑡], (34) makes use of all the examples that have been observed and thus is likely to make the largest increase of the dual function. And if setting 𝑆_𝑡={𝑡}, (34) degenerates in the update scheme we have described in Section 5.1. The time complexity for solving (34) is 𝑂((2×sizeof(𝑆_𝑡))³) (worst case).

Similar to the idea in Section 4, we can also introduce a new variable 𝜂_𝑡 into (34), max_{𝜶𝑖𝑖∈𝑆𝑡,𝜂𝑡∈[]0,1}𝐷𝜶₁,𝜶₂,…,𝜶_𝑇s.t.∀𝑖∉𝑆_𝑡,𝜶_𝑖=1−𝜂_𝑡×𝜶_𝑖𝑡−1. (35)𝜂_𝑡 is more like a forgetting factor [21] to downweight the contribution of observations whose indices do not belong to 𝑆_𝑡. When tracking the changes in the data stream, it is likely that recent observations will be more indicative of its appearance than more distant ones. Incorporating a forgetting factor in the online learning algorithms is a good way to moderate the balance between old and new observations. We can also obtain that 𝜂_𝑡=0 (𝑡∈{1,2,…,𝑇}) indicates that no forgetting is to occur.

For practical purpose, we test two choices of 𝑆_𝑡 to update multiple dual coefficient vectors in this paper.(i)Buffer-𝑁. Let the buffer size be 𝜏. Buffer-𝑁 replaces the oldest example (𝐱^(1,2)_𝑡−𝜏,𝑦_𝑡−𝜏,𝜎_𝑡−𝜏) in the buffer with the new arrived example (𝐱_𝑡^(1,2),𝑦_𝑡,𝜎_𝑡) on each learning round, which means 𝑆_𝑡={𝑡−𝜏+1,𝑡−𝜏+2,…,𝑡}.(ii)Buffer-𝐿. This buffering strategy replaces the oldest unlabeled point in the buffer with the incoming point while keeping labeled points. The oldest labeled point is evicted from the buffer only when it is filled with labeled points.

Figure 2 shows the set of the associated coefficient vectors which are used to ascend the dual function on each learning round for different choices of 𝑆_𝑡. Essentially, different choices of 𝑆_𝑡 construct different QP problems on each learning round.

(a)

(b)

(c)

(d)

Figure 2 

Four choices of 𝑆𝑡 to update multiple dual coefficient vectors. The horizontal thin line on each learning round represents the whole training data sequence, while the thick boxes represent the set of examples whose dual coefficient vectors are used to ascend the dual function on that round. Essentially, different choices of 𝑆𝑡 construct different QP problems on each learning round. Obviously, if 𝑆𝑡=[𝑡], the update scheme chooses to solve the largest possible QP on each learning round; and if 𝑆𝑡={𝑡}, the update scheme chooses to solve the smallest possible QP on each learning round.

5.3. Sparse Approximations for Kernel Representation

In practice, kernel functions are always used to find a linear classifier, like SVM. Our online coregularization framework only contains the product of two points, so we can easily introduce the kernel function into our framework. If we note 𝐾 the kernel matrix such that 𝐾_𝑖𝑗=Φ𝐱_𝑖⋅Φ𝐱_𝑗, (36)𝐱_𝑖 can be replaced by Φ(𝐱_𝑖) in our framework. Therefore, we can rewrite (19) as 𝝎_𝑡⁽¹⁾=1𝛾₁^𝑡_𝑖=1𝜎_𝑖𝑦_𝑖𝛼_𝑖1𝑡+1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐Φ⁽¹⁾𝐱_𝑖⁽¹⁾=^𝑡_𝑖=1𝛽_𝑖⁽¹⁾_𝑡Φ⁽¹⁾𝐱_𝑖⁽¹⁾,𝝎_𝑡⁽²⁾=1𝛾₂^𝑡_𝑖=1𝜎_𝑖𝜇𝑦_𝑖𝛼_𝑖2𝑡−1−𝜎_𝑖𝛼_𝑖0𝑡𝛾_𝑐Φ⁽²⁾𝐱_𝑖⁽²⁾=^𝑡_𝑖=1𝛽_𝑖⁽²⁾_𝑡Φ⁽²⁾𝐱_𝑖⁽²⁾. (37) Unfortunately, our previous derived online coregularization algorithms with kernel functions have to store the example sequence up to the current round, and the stored matrix size is 𝑡×𝑡 (worst case). For practical purpose, we present two approaches to sparsify the kernel representation of boundaries on each learning round.

Absolute Threshold. To construct a sparse representation for the boundaries, absolute threshold discards the examples whose associated coefficients 𝛽^(1,2) are close to zero. Let 𝜀>0 denote the absolute threshold. When an arrived example 𝐱_𝑖^(1,2) would not be used to update the boundaries in further learning process, 𝐱_𝑖^(1,2) is discarded if |(𝛽_𝑖^(1,2))_𝑡|<𝜀. The examples whose indices are in 𝑆_𝑡 cannot be discarded on round 𝑡 since they would be used to ascend the dual function.

𝑘 Maximal Coefficients (𝑘-MC). Another way to sparsify the kernel representation is to keep the examples of which the absolute value of 𝛽^(1,2) is the first 𝑘 maximum. Similar as the absolute threshold, 𝑘-MC does not discard the examples in 𝑆_𝑡 which would be used to ascend the dual function on round 𝑡. Based on this sparse approximation, the stored matrix size on round 𝑡 reduces to (𝑘+sizeof(𝑆_𝑡))×(𝑘+sizeof(𝑆_𝑡)).

The previous two sparse approximations are both motivated by the fact that the examples which have larger coefficients tend to exert more influence on our learned boundaries.

6. Experiments

This section presents a series of experimental results to report the effectiveness of our derived online coregularization algorithms. It is known that the performance of semi-supervised learning depends on the correctness of model assumptions. Thus, our focus is on comparing different online coregularization algorithms with multiple views, rather than different semi-supervised regularization methods.

We report experimental results on two synthetic and a real word binary classification problems. The prediction function in online coregularization algorithms are adopted as the average of the prediction functions from two views 𝑓_𝑡𝐱^(1,2)1=sign2𝝎_𝑡⁽¹⁾,𝐱⁽¹⁾+𝝎_𝑡⁽²⁾,𝐱⁽²⁾. (38)

Based on the idea of “interested in the best performance and simply select the parameter values minimizing the error” [3], we select combinations of the parameter values on a finite grid in Table 1, and it is sufficient to perform algorithm comparisons.

Table 1 

A finite grid of parameter values. We find the best performance of each online co-regularization algorithm on this finite grid.

The training sequences are generated randomly from each data sets. To avoid the influence of different training sequences, all results on each dataset are the average of five such trials (this idea is inspired by [11]). The error rates have ±1 standard deviation. While using buffering strategies to update multiple dual coefficient vectors, the buffer size is fixed at 100 to avoid high computational complexity on each learning round. We implemented all the experiments using MATLAB.

6.1. Two-Moons-Two-Lines Synthetic Data Set

This synthetic data set is generated similarly to the toy example used in [16, 19] in which examples in two classes appear as two moons in one view and two oriented lines in another (see Figure 3 for an illustration). This data set contains 2000 examples, and only 5 examples for each class are labeled. A Gaussian and linear kernel are chosen for the two-moons and two-lines views respectively. In this data set, the offline coregularization algorithms (CoLapSVM) [16] achieve an error rate of 0.

(a) Two-moons view

(b) Two-lines view

Figure 3 

Distribution of two-moons-two-lines data set.

The best performance of all the online coregularization algorithms in Section 5 is presented in Table 2. We also provide some additional details during the online coregularization process.

Table 2 

Mean test error rates on the two-moons-two-lines synthetic data set. The error rates are reported for three different sparse approximations. For gradient ascent, we choose a decaying step size 𝜌_𝑡=0.1/√𝑡. The result shows that our derived online co-regularization algorithms achieve test accuracy comparable to offline co-regularization (CoLapSVM). The online co-regularization algorithms based on aggressive dual ascending procedures perform better than those based on gradient ascent.

We compare cumulative runtime curves of online coregularization algorithms with different sparse approximation approaches in Figure 4. Online coregularization algorithms with sparse representation perform better than the basic online coregularization algorithms on the growth rate. The cumulative runtime growth curves of online coregularization algorithms with sparse approximation approaches scale only linearly, while the others scale quadratically.

(a)

(b)

Figure 4 

Cumulative runtime growth curves. From left to right (a) ADA (𝑆𝑡={𝑡}); (b) ADA (Buffer-𝑁). The curves have the similar trends for different online coregularization algorithms. Online coregularization algorithms with sparse representation perform better than the basic online coregularization algorithms on the growth rate.

We also compare the number of examples in the kernel representation of boundary vectors in two views on each learning round for different sparse approximation approaches. Figure 5 shows that only part of examples have to be stored (and computed) while using sparse approximation approaches. Online coregularization algorithms without sparse approximation approaches are time consuming and memory-consuming, and it is intractable to apply them to real-world long time tasks.

(a) Two-moons view

(b) Two-lines view

Figure 5 

The number of examples in the kernel representation of boundary vectors in two views for different sparse approximation approaches. The online coregularization algorithm used in this experiment is ADA (Buffer-𝑁). If no sparse representation approaches are used, the kernel representation contains all the arrived examples on each learning round. The number of examples in the kernel representation increases slowly while using an absolute threshold, and the number is at most 𝑘 while using 𝑘-MC for online coregularization algorithms.

In Section 3, we have demonstrated the relationship between the primal objective function 𝐽(𝝎_𝑡^(1,2)) and the dual function 𝐷((𝜶₁)_𝑡,(𝜶₂)_𝑡,…,(𝜶_𝑇)_𝑡). We compare the primal objective function versus the dual function on the training sequence of two-moons-two-lines data set as 𝑡 increases in Figure 6. The result shows that the two curves approach each other along the online coregularization algorithms run. The value of dual function never decreases as 𝑡 increases; correspondingly, the curve of primal function has a downward trend and some little fluctuations. We also observe that the curve of the primal objective function which updates multiple dual coefficient vectors on each learning round shows a more smooth downward trend and has less rapid fluctuations. This experiment supports the theory that increasing the dual problem can achieve comparable risks of the primal objective function.

(a)

(b)

Figure 6 

The value curves of the primal objective function and the dual function during the incremental learning process. From left to right (a) ADA (𝑆𝑡={𝑡}) and (b) ADA (Buffer-𝑁). The two curves approach each other as 𝑡 increases.

We report the performance of 𝝎_𝑡^(1,2) on the whole two-moons-two-lines data set in Figure 7. This result shows that the boundary vector is adjusted to be a better one along the online coregularization algorithms run. Since our algorithms adjust the decision boundary vector according to the local agreement of 𝑆_𝑡 in the two views on each learning round, the curve of the error rate is not always decreasing along the online coregularization process. It is also the reason why our online coregularization algorithms can track the changes in the data sequence (more detail in Section 6.3). Similar as the experiments in Figure 6, we can observe that the error rate of 𝝎_𝑡^(1,2) which updates multiple dual coefficient vectors on each learning round shows a more smooth downward trend and has less rapid fluctuations.

(a)

(b)

Figure 7 

Error rates of 𝝎𝑡(1,2) on the whole two-moons-two-lines data set. From left to right (a) ADA (𝑆𝑡={𝑡}) and (b) ADA (Buffer-𝑁). The error rate of 𝝎𝑡(1,2) has a downward trend, but it is not always decreasing during the online coregularization process.

6.2. Web Page Categorization

We applied our derived online coregularization algorithms to the WebKB text classification task studied in Blum and Mitchell [2]; Sindhwani et al. [16]; Sun and Shawe-Taylor [19]. The task is to predict whether a web page is a course home page or not. The data set consists of 1051 web pages in two views (page and link) collected from the computer science department web sites at four U.S. universities: Cornell, University of Washington, University of Wisconsin, and University of Texas. The first view of the data is the textual content of a webpage itself, and the second view is all links pointing to the web page from other web pages. We preprocessed each view by removing stop words, punctuation, and numbers and then applied Porter’s stemming to the text [22]. This problem has an unbalanced class distribution since there are a total of 230 course home pages and 821 noncourse. In addition, words that occur in five or fewer documents were ignored. This resulted in 2332 and 87 dimensional vectors for two views, respectively. Finally, document vectors were normalized to TDIDF features (the product of term frequency and inverse document frequency) [23]. As in [19], we randomly label 3 course and 9 noncourse examples for each class. In this experiment, the linear kernel is used for both views.

In this data set, the offline coregularization algorithms (CoLapSVM) [16] achieve an error rate of 6.32%. In Table 3, we report the best performance of all the online coregularization algorithms on the web page data set.

Table 3 

Mean test error rates on the web page data set. The error rates are reported for three different sparse approximations. For gradient ascent, we choose a decaying step size 𝜌_𝑡=0.1/√𝑡. The result shows that our derived online co-regularization algorithms achieve test accuracy comparable to offline co-regularization (CoLapSVM).

6.3. Rotating Two-Moons-Two-Lines Synthetic Data Set

When the underlying distributions, both 𝑃(𝐱^(1,2)) and 𝑃(𝑦∣𝐱^(1,2)), change during the course of learning, the algorithms are expected to track the changes in the data sequence. In this subsection, we test the applicability of our framework to settings where the target hypotheses in different views are not fixed but rather drift with the sequence of examples. To demonstrate that our online coregularization algorithms can handle concept drift, we perform our experiments on rotating two-moons-two-lines data sequence. This data set contains 8000 examples, and only 1% examples for each class are labeled. Figure 8 shows that the two-moons-two-lines data set smoothly rotate 360^∘ during the sequence, and the target boundaries in two views drift with the sequence of examples. In the rotating two-moons-two-lines data set, the points will change their true labels during the sequence, and every stationary decision boundary will have an error rate of 50% approximately. A Gaussian and linear kernel are chosen for the two rotating moons and two rotating lines view, respectively.

(a) Two rotating moons view

(b) Two rotating lines view

Figure 8 

Rotating two-moons-two-lines data sequence. We spin the two-moons-two-lines data set in top left of the two views during the sequence so that two moons and two lines smoothly rotate 360∘ in every 8000 examples.

In Table 4, we report the best performance of all the online coregularization algorithms on the rotating two-moons-two-lines data sequence. In this experiment, we also discuss the effect of the buffer size to track the changes in the data sequence.

Table 4 

Mean test error rates on the rotating two-moons-two-lines synthetic data sequence. The error rates are reported for three different sparse approximations. For gradient ascent, we choose a stationary step size 𝜌_𝑡=0.1. The result shows that our derived online co-regularization algorithms are able to track the changes in the sequence and maintain a smaller error rate compared with batch learning algorithms. Specially, ADA (Buffer-𝑁) performs better than the other online co-regularization algorithms.

Obviously, when tracking the changes in the rotating two-moons-two-lines data sequence, it is likely that recent examples will be more indicative of the boundaries than more distant ones. Buffer-𝑁 updates the boundaries using the recent examples, it is also the reason why ADA (Buffer-𝑁) performs better than the other online coregularization algorithms. We report the error rates of ADA (Buffer-𝑁) with different buffer sizes on rotating two-moons-two-lines synthetic data sequence in Figure 9. This experiment illustrates that a suitable size of Buffer-𝑁 is able to adapt to the changing sequence and maintain a small error rate.

Figure 9 

Error rates of ADA (Buffer-𝑁) with different buffer sizes on the rotating two-moons-two-lines synthetic data sequence.

7. Conclusion and Further Discussion

In this paper we presented an online coregularization framework based on the notion of ascending the dual function. We demonstrated that the existing online coregularization algorithms in previous work can be viewed as an approximation of our dual ascending process using gradient ascent. New online coregularization algorithms are derived based on aggressive dual ascending procedures. For practical purpose, we proposed two sparse approximation approaches for kernel representation to reduce the computational complexity. Experiments showed that our online coregularization algorithms can adjust the boundary vector with the input sequence and have risk and error rates comparable to offline algorithms. Specially, our online coregularization algorithms can handle the settings where the target boundaries are not fixed but rather drift with the sequence of examples in different views.

There are many interesting questions remaining in the online semi-supervised learning setting. For instance, we plan to study new online learning algorithms for other semi-supervised learning models. Another direction is how to choose effective combination of the parameter values more intelligently during the online coregularization process.

Appendix

Fenchel Conjugate

The Fenchel conjugate of a function 𝑓∶𝑆→ℝ is defined as 𝑓^∗(𝝀)=sup⟨𝝀,𝝎⟩−𝑓(𝝎)∶𝝎∈𝑆. (A.1) Since 𝑓^∗ is defined as a supremum of linear functions, it is convex. Here, we describe a few lemmas of Fenchel conjugate which we use as theoretical tools in this paper. More details are in [24].

Lemma A.1. Let 𝑓 be a closed and convex function and let 𝜕𝑓(𝝎) be its differential set at 𝝎, Then, for all 𝝀∈𝜕𝑓(𝝎), we have 𝑓(𝝎)+𝑓^∗(𝝀)=⟨𝝀,𝝎⟩.

Proof. Since 𝝀∈𝜕𝑓(𝝎) and 𝑓 is closed, and convex, we know that 𝑓(𝝎^′)−𝑓(𝝎)≥⟨𝝀,𝝎^′−𝝎⟩ for all 𝝎^′∈𝑆. Equivalently ⟨𝝀,𝝎⟩−𝑓(𝝎)≥sup𝝀,𝝎^′−𝑓𝝎^′∶𝝎∈𝑆. (A.2) The right-hand side of the previous equation equals to 𝑓^∗(𝝀), and thus ⟨𝝀,𝝎⟩−𝑓(𝝎)≥𝑓^∗(𝝀)⟶⟨𝝀,𝝎⟩−𝑓^∗(𝝀)≥𝑓(𝝎). (A.3) The assumption that 𝑓 is closed and convex implies that 𝑓 is the Fenchel conjugate of 𝑓^∗. Thus, 𝑓(𝝎)=sup𝝀^′,𝝎−𝑓𝝀^′∶𝝀^′∈𝑆≥⟨𝝀,𝝎⟩−𝑓(𝝎). (A.4) Combining the two inequalities, we have 𝑓(𝝎)+𝑓^∗(𝝀)=⟨𝝀,𝝎⟩. (A.5)

Lemma A.2. Let 𝑓(𝝎)=∑^𝑘_𝑖=1[𝑏_𝑖−⟨𝝎,𝐱_𝑖⟩]₊, where for all 𝑖∈{1,2,…,𝑘}, 𝑏_𝑖∈ℝ and 𝐱_𝑖∈ℝ^𝑛. The Fenchel conjugate of 𝑓(𝝎) is 𝑓^∗(𝝀)=−∑^𝑘_𝑖=1𝛼_𝑖𝑏_𝑖if𝝀∈−∑^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖∶∀𝑖∈{1,2,…,𝑘},𝛼_𝑖∈[0,1]∞𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. (A.6)

Proof. We first rewrite the 𝑓(𝝎) as follows: 𝑓(𝝎)=^𝑘_𝑖=1𝑏_𝑖−𝝎,𝐱_𝑖+=max_{𝛼1,𝛼2,…,𝛼𝑘∈[]}^𝑘_0,1𝑖=1𝛼_𝑖𝑏_𝑖−𝝎,𝐱_𝑖, (A.7) where 𝛼_𝑖∈[0,1] for all 𝑖∈{1,2,…,𝑘}. Based on the definition of Fenchel conjugate, we can obtain that 𝑓^∗(𝝀)=max_𝝎⟨𝝀,𝝎⟩−𝑓(𝝎)=max_𝝎⟨𝝀,𝝎⟩−max_{𝛼1,𝛼2,…,𝛼𝑘∈[]}^𝑘_0,1𝑖=1𝛼_𝑖𝑏_𝑖−𝝎,𝐱_𝑖=max_𝝎min_{𝛼1,𝛼2,…,𝛼𝑘∈[]0,1}⟨𝝀,𝝎⟩−^𝑘_𝑖=1𝛼_𝑖𝑏_𝑖−𝝎,𝐱_𝑖=min_{𝛼1,𝛼2,…,𝛼𝑘∈[]0,1}max_𝝎−^𝑘_𝑖=1𝛼_𝑖𝑏_𝑖+𝝀+^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖,𝝎=min_{𝛼1,𝛼2,…,𝛼𝑘∈[]0,1}−^𝑘_𝑖=1𝛼_𝑖𝑏_𝑖+max_𝝎𝝀+^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖,𝝎. (A.8) Since the aforementioned third equality follows from the strong max-min property, it can be transferred into a min-max problem. If 𝝀+∑^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖≠0, max_𝝎⟨𝝀+∑^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖,𝝎⟩ is ∞; otherwise, if 𝝀+∑^𝑘_𝑖=1𝛼_𝑖𝐱_𝑖=0, we have 𝑓^∗(𝝀)=−∑^𝑘_𝑖=1𝛼_𝑖𝑏_𝑖.

Lemma A.3. Let ‖⋅‖ be any norm on ℝ^𝐧, and let 𝑓(𝝎)=(1/2)‖𝝎‖^𝟐 with 𝑆=ℝ^𝐧. Then 𝑓^∗(𝝀)=(1/2)‖𝝀‖²_∗ where ‖⋅‖_∗ is the dual norm of ‖⋅‖. The domain of 𝑓^∗ is also ℝ^𝐧. For example, if 𝑓(𝝎)=(1/2)‖𝝎‖²₂ then 𝑓^∗(𝝀)=(1/2)‖𝝀‖²₂ since ℓ₂ norm is dual to itself.

Lemma A.4. Let 𝑓 be a function, and let 𝑓^∗ be its Fenchel conjugate. For 𝑎>0 and 𝑏∈ℝ, the Fenchel conjugate of 𝑔(𝝎)=𝑎𝑓(𝝎)+𝑏 is 𝑔^∗(𝝀)=𝑎𝑓^∗(𝝀/𝑎)−𝑏.