On the Variability of Neural Network Classification Measures in the Protein Secondary Structure Prediction Problem

Abstract

We revisit the protein secondary structure prediction problem using linear and backpropagation neural network architectures commonly applied in the literature. In this context, neural network mappings are constructed between protein training set sequences and their assigned structure classes in order to analyze the class membership of test data and associated measures of significance. We present numerical results demonstrating that classifier performance measures can vary significantly depending upon the classifier architecture and the structure class encoding technique. Furthermore, an analytic formulation is introduced in order to substantiate the observed numerical data. Finally, we analyze and discuss the ability of the neural network to accurately model fundamental attributes of protein secondary structure.

1. Introduction

The protein secondary structure prediction problem can be phrased as a supervised pattern recognition problem [1–5] for which training data is readily available from reliable databases such as the Protein Data Bank (PDB) or CB513 [6]. Based upon training examples, subsequences derived from primary sequences are encoded based upon a discrete set of classes. For instance, three class encodings are commonly applied in the literature in order to numerically represent the secondary structure set (alpha helix (𝐻), beta sheet (𝐸), coil (𝐶)) [7–11]. By applying a pattern recognition approach, subsequences of unknown classification can then be tested to determine the structure class to which they belong. Phrased in this way, backpropagation neural networks [7, 12–14], and variations on the neural network theme [8, 10, 11, 15–18] have been applied to the secondary structure prediction problem with varied success. Furthermore, many tools currently applying hybrid methodologies such as PredictProtein [19, 20], JPRED [8, 17, 21], SCRATCH [22, 23] and PSIPRED [24, 25] rely on the neural network paradigm as part of their prediction scheme.

One of the main reasons for applying the neural network approach in the first place is that they tend be good universal approximators [26–30] and, theoretically, have the potential to create secondary structure models. In other words, after a given network architecture has been chosen and presented with a robust set of examples, the optimal parameters associated with the trained network, in principle, define an explicit function that can map a given protein sequence to its associated secondary structure. If the structure predicted by the network function is generally correct and consistent for an arbitrary input sequence not contained in the training set, one must be left to conclude that the neural network has accurately modeled some fundamental set of attributes that define the properties of protein secondary structure. Under these circumstances, one should then be able to extract information from the trained neural network model parameters; thus, leading to a solution to the secondary structure prediction problem as well as a parametric understanding of the underlying basis for secondary structure.

The purpose of this work is to revisit the application of neural networks to the protein secondary structure prediction problem. In this setting, we consider the commonly encountered case where three structure classes (alpha helix (𝐻), beta sheet (𝐸), and coil (𝐶)) are used to classify a given protein subsequence. Given the same set of input training sequences, we demonstrate that, for the backpropagation neural network architecture, classification results and associated confidence measures can vary when two equally valid encoding schemes are employed to numerically represent the three structure classes (i.e., the “target encoding scheme”). Such a result goes against the intuition that the physical nature of the secondary structure property should be independent of the target encoding scheme chosen.

The contribution of this work is not to demonstrate improvements over existing techniques. The hybrid techniques outlined above have been demonstrated to outperform neural networks when used alone. Instead, we focus our attention on the ability of the neural network model-based approach to accurately characterize fundamental attributes of protein secondary structure given that certain models presented within this work are demonstrated to yield variable results. Specifically, in this work, we present (1)numerical results demonstrating how secondary structure classification results can vary as function of classifier architecture and parameter choices;(2)an analytic formulation in order to explain under what circumstances classification variability can arise; (3)an outline of specific challenges associated with the neural network model-based approach outlined above.

The conclusions reported here are relevant because they bring into discussion a body of literature that has purported to offer a viable path to the solution of the secondary structure prediction problem. Section 3 describes the methods applied in this work examine the total number that retained their classification using. In particular, this section provides details concerning the encoding of the protein sequence data (Section  3.1), the encoding of the structure classes (Section  3.2) as well as the neural network architectures (Sections  3.3-3.4), and the classifier performance measures (Section  3.5) applied in this work. Section 4 then presents results from numerical secondary structure classification experiments. Section 5 presents an analytic formulation for the linear network and the backpropagation network described in Section 3 in order to explain the numerical results given in Section 4.

2. Notation for the Supervised Classification Problem

In the supervised classification problem [1, 2], it is assumed that a training set consists of 𝑁 training pairs: 𝑋₁,𝑌₁,𝑋₂,𝑌₂,…,𝑋_𝑁,𝑌_𝑁, (1) where 𝑋_𝑖∈𝑅^𝑛 are 𝑛-dimensional input column vectors and 𝑌_𝑖∈𝑅^𝑝 are 𝑝-dimensional output column vectors. The goal of the supervised classifier approach is to ensure that the desired response to a given input vector 𝑋_𝑖 of dimension 𝑛 from the training set is the 𝑝-dimensional output vector 𝑌_𝑖. Furthermore, when the training data can be partitioned into 𝑀 distinct classes, a set ℰ≡{𝑒₁,…,𝑒_𝑗,…,𝑒_𝑀} of target 𝑝-dimensional column vectors 𝑒_𝑗∈𝑅^𝑝 are chosen to encode (i.e., mathematically represent) each class for 𝑗=1,…,𝑀. Under these circumstances, each output training vector 𝑌_𝑖 is derived from the set ℰ. Based upon this discussion, we summarize the use of following symbols: (i)𝑛 is the dimension of a classifier input vector; (ii)𝑝 is the dimension of a classifier output vector; (iii)𝑀 is the number of discrete classes for the classification problem; (iv)𝑁 is the number of training pairs for the supervised classification problem.

3. Methods

In order to apply the neural network paradigm, two numerical issues must be addressed. First, since the input data comes in the form of an amino acid sequence, Section  3.1 discusses a simple encoding scheme for converting the amino acid alphabet into a usable numerical form. Second, for this work, our secondary structure target alphabet consists of elements from the set {𝐸,𝐻,𝐶}={BetaSheet,AlphaHelix,Coil}. Hence, an encoding scheme must also be chosen for representing the neural network classifier output. Section  3.2 discusses two approaches to encoding the output in fine detail because it is critical to the main point of this paper. Specifically, we choose two different target vector encoding schemes that can be related by a simple mathematical relationship. Such an approach will allow us to compare classifier performance measures based upon the target vector encoding; in addition, it will facilitate the analytic formulation presented in Section 5. Finally, Sections  3.3–3.5 review the neural network architectures and the specific classifier performance measures employed in this work. Section 6 then concludes with some final observations regarding the neural network model-based approach to the protein secondary structure prediction problem.

3.1. Encoding of Protein Sequence Input Data

For the numerical experiments, the training set was constructed using one hundred protein sequences randomly chosen from the CB513 database [6] available through the JPRED secondary structure prediction engine [21]. Furthermore, we employ a moving window of length 17 to each protein sequence where, in order to avoid protein terminal effects, the first and last 50 amino acids are omitted from the analysis. The secondary structure classification of the central residue is then assigned to each window of 17 amino acids. For the one hundred sequences analyzed, a total of 12000 windows of length 17 were extracted. The window size value of 17 was chosen based upon the assumption that the eight closest neighboring residues will have the greatest influence on the secondary structure conformation of the central residue. This assumption is consistent with similar approaches reported in the literature [7, 12–14].

To encode the input amino acid sequences of length 17, we employ sparse orthogonal encoding [31] which maps symbols from a given sequence alphabet onto a set of orthogonal vectors. Specifically, for an alphabet containing 𝐾 symbols, a unique 𝐾-dimensional unit vector is assigned to each symbol; furthermore, the 𝑘th unit vector is one at the 𝑘th position and is zero at all other positions. Hence, if all training sequences and unknown test sequences are of uniform length 𝐿, an encoded input vector will be of dimension 𝑛 where 𝑛=𝐿𝐾. In our case, 𝐾=20 and 𝐿=17; hence, the dimension of any given input vector is 𝑛=340.

The above input vector encoding technique is commonly applied in the bioinformatics and secondary structure prediction literature [7, 15]. While many different and superior approaches to this phase of the machine learning problem have been suggested [3–5], we have chosen orthogonal encoding because of its simplicity and the fact that the results of this work do not depend upon the input encoding scheme. Instead, our work specifically focuses on potential neural network classifier variabilities induced by choice of the target vector encoding scheme.

3.2. Target Vector Encoding

Analytically characterizing the invariance of classifier performance measures clearly involves first establishing a relationship between different sets of target vectors ℰ and ℰ. As a means of making the invariance formulation presented in this paper more tractable, we assume that two alternative sets of target vectors can be related via an affine transformation involving a translation, 𝑇, a rigid rotation, 𝑅, where 𝑅 is an orthogonal matrix and a scale factor, 𝑎, Γ≡𝑎𝑅(Γ−𝑇), (2) where Γ≡𝑒₁𝑒₂…𝑒_𝑀,Γ≡𝑒₁̃𝑒₂̃…𝑒_𝑀̃,𝑇≡𝑡₁𝑡₂…𝑡_𝑀 (3) is a matrix of translation column vectors applied to each target vector. Many target vector choices regularly applied in the literature can be related via the transformation in (2). For instance, two equally valid and commonly applied encoding schemes for the three class problem are orthogonal encoding [31] where 100010001𝐻→,𝐸→,𝐶→, (4) and −1𝐻→2√320−1,𝐸→2−√320100,𝐶→ (5) where class encodings are chosen on the vertices of a triangle in a two-dimensional plane [14]. It turns out that (4) and (5) can be phrased in terms of (2) [32]; hence, the numerical results presented in this work will apply this set of encodings. More precisely, the secondary structure classification associated with a given input vector is encoded using (4) and (5) (hence, 𝑝=3). The set of target vectors ℰ is derived from (4) and the set of target vectors ℰ is derived from (5). Both the linear and the backpropagation networks are tested first by training using ℰ and then comparing classifier performance with their counterparts trained using ℰ. In all numerical experiments, MATLAB has been used for simulating and testing these networks.

3.3. The Linear Network

When the supervised classifier model in (1) assumes an affine relationship between the input and output data sets (as in the case of multiple linear regression), matrices of the form 𝑋≡𝑋₁…𝑋_𝑁,𝑌≡𝑌₁…𝑌_𝑁 (6)(7) are generally introduced. Specifically, the linear network seeks to determine a 𝑝×𝑛 matrix 𝛽 of coefficients and a constant 𝑝-dimensional column vector 𝑐 such that the 𝑖th output vector 𝑌_𝑖 in the training set can be approximated by 𝛽𝑋_𝑖+𝑐. (8) Given this model, we can form a weight matrix of unknown coefficients 𝑊≡𝛽𝑐 (9) that, ideally, will map each input training vector into the corresponding output training vector. If the bottom row of the input data matrix 𝑋 is appended with a row of ones leading to the (𝑛+1)×𝑁 matrix 𝑋𝐴≡¹…𝑋_𝑁1…1, (10) in matrix form, the goal is then to find a 𝑝×𝑛+1 weight matrix 𝑊 that minimizes the sum squared-error 𝐸^linear≡^𝑁_𝑖=1𝑊𝐴_𝑖−𝑌_𝑖^𝑇𝑊𝐴_𝑖−𝑌_𝑖 (11) over the set of data pairs 𝐴_𝑖≡𝑋_𝑖1 (12) by satisfying the first derivative condition 𝜕𝐸^linear/𝜕𝑊=0. The least squares solution to this problem is found via the pseudoinverse 𝐴^† [33] where 𝑊=𝑌𝐴^†. (13)

Once the optimal set of weights has been computed, the network response to an unknown 𝑛×1 input vector 𝑥 can be determined by defining the (𝑛+1)×1 vector 𝑧≡[𝑥1]^𝑇 (14) and calculating 𝑓(𝑧)=𝑊𝑧, (15) where 𝑓(𝑧) is a 𝑝×1 column vector.

3.4. The Backpropagation Network

Given an input vector 𝑥∈𝑅^𝑛, the model 𝑓∶𝑅^𝑛→𝑅^𝑝 for a backpropagation neural network with a single hidden layer consisting of 𝑚 nodes is described as 𝑓(𝑥)=𝑓₁(𝑥)𝑓₂(𝑥)⋮𝑓_𝑝(𝑥)=𝛼𝜎(𝑊𝑥−𝜏), (16) where 𝛼∈𝑅^𝑝×𝑚, 𝑊∈𝑅^𝑚×𝑛, and 𝜏∈𝑅^𝑚 define the set of network weights 𝒲={𝑊,𝜏,𝛼} and 𝜎∶𝑅^𝑚→𝑅^𝑝 is a “sigmoidal” function that is bounded and monotonically increasing. To perform supervised training, in a manner similar to the linear network, 𝒲 is determined by minimizing the objective function: 𝐸^𝑏𝑝=^𝑁_𝑖=1𝑓𝑋_𝑖−𝑌_𝑖^𝑇𝑓𝑋_𝑖−𝑌_𝑖 (17) given the training data defined in (1). Since 𝑓(𝑋_𝑖) is no longer linear, numerical techniques such as the gradient descent algorithm and variations thereof are relied upon to compute the set 𝒲 that satisfies the first derivative condition 𝜕𝐸^𝑏𝑝/𝜕𝒲=0.

Consider the following definitions: 𝛿𝐹_𝑘≡𝑓𝑋_𝑘−𝑌_𝑘=𝑓₁𝑋_𝑘−𝑌_𝑘1𝑓₂𝑋_𝑘−𝑌_𝑘2𝑓_𝑝𝑋_𝑘−𝑌_𝑘𝑝,𝑆_𝑘≡𝜎∑^𝑛_𝑗=1𝑊_1𝑗𝑋_𝑘𝑗−𝜏₁𝜎∑^𝑛_𝑗=1𝑊_2𝑗𝑋_𝑘𝑗−𝜏₂⋮𝜎∑^𝑛_𝑗=1𝑊_𝑚𝑗𝑋_𝑘𝑗−𝜏_𝑚,𝑆^′_𝑘≡𝜎∑^′𝑛_𝑗=1𝑊_1𝑗𝑋_𝑘𝑗−𝜏₁𝜎∑^′𝑛_𝑗=1𝑊_2𝑗𝑋_𝑘𝑗−𝜏₂⋮𝜎∑^′𝑛_𝑗=1𝑊_𝑚𝑗𝑋_𝑘𝑗−𝜏_𝑚, (18) where 𝜎^′(𝑧)=𝑑𝜎/𝑑𝑧. The first derivative conditions for the network weights prescribed by (16) and (17) can then be written in matrix form as follows: 𝜕𝐸^𝑏𝑝=𝜕𝛼^𝑁_𝑘=1𝛿𝐹_𝑘𝑆^𝑇_𝑘=0, (19) (where “𝑇” denotes the matrix transpose), 𝜕𝐸^𝑏𝑝𝜕𝜏=−^𝑁_𝑘=1diag𝑆^′_𝑘𝛼^𝑇𝛿𝐹^𝑇_𝑘=0,𝜕𝐸^𝑏𝑝=𝜕𝑊^𝑁_𝑘=1diag𝑆^′_𝑘𝛼^𝑇𝛿𝐹^𝑇_𝑘𝑋^𝑇_𝑘=0, (20) where diag(𝑆^′_𝑘) is a square diagonal matrix such that the diagonal entries consist of components from the vector 𝑆^′_𝑘.

3.5. Classification Measures

After a given classifier is trained, when presented with an input vector 𝑥∈𝑅^𝑛 of unknown classification, it will respond with an output 𝑓(𝑥)∈𝑅^𝑝. The associated class membership 𝐶(𝑥) is then often determined by applying a minimum distance criterion: 𝐶(𝑥)≡argmin_𝑖𝑑𝑓(𝑥),𝑒_𝑖, (21) where the target vector 𝑒_𝑖 that is closest to 𝑓(𝑥) implies the class. Furthermore, when characterizing the performance of a pattern classifier, one often presents a set 𝑆 of test vectors and analyzes the associated output. In addition to determining the class membership, it is also possible to rank the distance between a specific target vector and the classifier response to 𝑆. In this case a similar distance criterion can be applied in order to rank an input vector 𝑥∈𝑆 with respect to class 𝐽∈{1,…,𝑀}, 𝜌_𝐽(𝑥)=𝑑𝑓(𝑥),𝑒_𝐽. (22) For the purposes of this work, (15) facilitates the determination of the class membership and ranking with respect to class 𝐽 for the linear network. Similarly, assuming a set of weights 𝒲={𝑊,𝜏,𝛼} for a trained backpropagation network, (21) and (22) would be applied using (16).

It is well established that, in the case of a normally distributed data, the classification measures presented above minimize the probability of a classification error and are directly related to the statistical significance of a classification decision [1]. Given the neural network as the supervised classification technique and two distinct choices for the set of target vectors ℰ and ℰ, we demonstrate, in certain instances, that the classification and ranking results do not remain invariant such that 𝐶(𝑥)≠𝐶(𝑥) and 𝜌_𝐽(𝑥)≠𝜌_𝐽(𝑥) for any input vector 𝑥.

4. Noninvariance of Secondary Structure Predictions

In this section, we numerically demonstrate that, when different target vector encodings are applied, neural network classifier measures outlined above, in certain cases, are observed to vary widely. For each neural network architecture under consideration, an analytic formulation is then presented in Section 5 in order to explain the observed numerical data.

As mentioned in Section  3.2, numerical experiments are performed first by training using ℰ and then comparing classifier performance with their counterparts trained using ℰ. Multiple cross-validation trials are required in order to prevent potential dependency of the evaluated accuracy on the particular training or test sets chosen [7, 15]. In this work, we apply a hold-n-out strategy similar to that of [14] using 85% of the 12000 encoded sequences as training data (i.e., 𝑁=10200) and 15% as test data to validate the classification results. Recognition rates for both the linear and backpropagation rates using either set of target vector encodings were approximately 65% which is typical of this genre of classifiers that have applied similar encoding methodologies [7, 12–14]. Although these aggregate values remain consistent, using (21) and (22) we now present data demonstrating that, while class membership and ranking remain invariant for the linear network, these measures of performance vary considerably for the backpropagation network which was trained with, 𝑚=17, seventeen hidden nodes and a mean squared training error less than 0.2. Ranking results from a representative test for the linear and backpropagation networks are presented for the top 20 ranked vectors in Tables 1 and 3. Class membership data are presented in Tables 2 and 4. Observe that, for the linear network, indices for the top 20 ranked vectors remain invariant indicating ranking invariance; in addition, no change in class membership is observed. On the other hand, Tables 3 and 4 clearly indicate a lack of consistency when considering the ranking and class membership of test vectors. A particularly troubling observation is that very few vectors ranked in the top 20 with respect to 𝑒₁ were ranked in the top 20 with respect to 𝑒₁̃. Furthermore, Table 4 indicates that the class membership of a substantial number of test vectors changed when an alternative set of target vectors was employed. The data also indicates that the greatest change in class membership took place for alpha helical sequences; thus implying that there is substantial disagreement over the modeling of this secondary structure element by the backpropagation network due to a simple transformation of the target vectors.

Table 1 

Ranking results for the linear network where 𝑖 represents the test vector index and 𝜌₁(𝑥) and 𝜌₁(𝑥) represent the distance with respect to the helix class vectors 𝑒₁ and 𝑒₁̃. Out of 1800 vectors tested, vectors referred to in this table were ranked from 1 to 20.

Table 2 

Class membership results for the linear network. For each class, the total number of vectors classified using ℰ is analyzed to examine the total number that retained their classification using ℰ.

Table 3 

Ranking results for the backpropagation network where 𝑖 represents the test vector index and 𝜌₁(𝑥) and 𝜌₁(𝑥) represent the distance with respect to the helix class vectors 𝑒₁ and 𝑒₁̃. Out of 1800 vectors tested, vectors referred to in this table were ranked from 1 to 20.

Table 4 

Class membership results for the backpropagation network. For each class, the total number of vectors classified using ℰ is analyzed to examine the total number that retained their classification using ℰ.

5. Analysis

The results in Section 4 clearly show that while the pattern recognition results for the linear network remain invariant under a change in target vectors, those for the backpropagation network do not. In this section, we present analytic results in order to clearly explain and understand why these two techniques lead to different conclusions.

5.1. Invariance Formulation

Let us begin by considering two definitions.

Definition 1. Given two sets of target vectors ℰ and ℰ, the class membership is invariant under a transformation of the target vectors if, for any input vector 𝑥, 𝑑𝑓(𝑥),𝑒_𝑖̃>𝑑𝑓(𝑥),𝑒_𝑗̃⇒𝑑𝑓(𝑥),𝑒_𝑖>𝑑𝑓(𝑥),𝑒_𝑗, (23) where 𝑓(𝑥) is the output of the classifier with target vectors {𝑒_𝑗̃}^𝑀_𝑗=1.

Definition 2. Given two sets of target vectors ℰ and ℰ, the ranking with respect to a specific class 𝐽 is invariant under a transformation of the target vectors if, for any input vectors 𝑥₁ and 𝑥₂, 𝑑𝑓𝑥₁,𝑒_𝐽̃>𝑑𝑓𝑥₂,𝑒_𝐽̃⇒𝑑𝑓𝑥₁,𝑒_𝐽>𝑑𝑓𝑥₂,𝑒_𝐽. (24)

Based upon these definitions, the following has been established [32].

Proposition 3. Given two sets of target vectors ℰ and ℰ, if the ranking is invariant, then the class membership of an arbitrary input vector 𝑥 will remain invariant.

In the analysis presented, the strategy for characterizing neural network performance depends upon the data from the previous section. For the linear network, since both ranking and classification were observed to remain invariant, it is more sensible to characterize the invariance of this network using Definition 2. Then, based upon Proposition 3, class membership invariance naturally follows. On the other hand, to explain the noninvariance of both class membership and ranking observed in the backpropagation network, the analysis is facilitated by considering Definition 1. The noninvariance of ranking then naturally follows from Proposition 3.

5.1.1. Invariance Analysis for the Linear Network

When the target vectors are subjected to the transformation defined in (2), the network output can be expressed as 𝑓(𝑧)=𝑊𝑧=𝑌𝐴^†𝑧=𝑎𝑅𝑌−𝑇^′𝐴^†𝑧, (25) where 𝑇^′ is derived from 𝑇 such that the translation vector 𝑡_𝑖 associated with 𝑒_𝑖∈ℰ is appropriately aligned with the correct target vector in matrix 𝑌. In other words, when the output data matrix in (7) is of the form 𝑌=𝑒_𝑖1𝑒_𝑖2…𝑒_𝑖𝑁 (26) then 𝑇^′=𝑡_𝑖1𝑡_𝑖2…𝑡_𝑖𝑁, (27) where 𝑖_𝑗∈{1,…,𝑀} for 𝑗=1,…,𝑁. Given this network, the following result is applicable [32].

Proposition 4. If (i)the number of training observations 𝑁 exceeds the vector dimension 𝑛;(ii)the rows of the matrix 𝐴 are linearly independent;(iii)ℰ and ℰ are related according to (2);(iv)for some 𝑡∈𝑅^𝑝, 𝑡_𝑖=𝑡 for all 𝑖=1,…,𝑀 in (2);
then the ranking and, hence, the class membership for the linear network will remain invariant.

In other words, if the columns of the matrix 𝑇^′ in (25) are all equal, then using (15) and (25) will result in, (23) being satisfied. The above result is applicable to the presented numerical data with 𝑛=340 and 𝑁=10200; hence, ranking and class membership invariances are corroborated by the data in Tables 1 and 2.

5.1.2. Invariance Analysis for the Backpropagation Network

In this section, we seek to characterize the noninvariance observed in class membership using the backpropagation network. If the class membership varies due to a change in target vectors, then this variation should be quantifiable by characterizing the boundary separating two respective classes. The decision boundary between class 𝑖 and class 𝑗 is defined by points 𝑥 such that 𝑓(𝑥)−𝑒_𝑖²=𝑓(𝑥)−𝑒_𝑗², (28) where 𝑖,𝑗∈{1,…,𝑀}, 𝑖≠𝑗 and 𝑓(𝑥) is the classifier output. Under these circumstances, if an ℓ₂ norm is applied in (21), 𝑓(𝑥)−𝑒_𝑖^𝑇𝑓(𝑥)−𝑒_𝑖=𝑓(𝑥)−𝑒_𝑗^𝑇𝑓(𝑥)−𝑒_𝑗. (29) The solution set to this equation consists of all 𝑥 such that 𝑓(𝑥) is equidistant from 𝑒_𝑖 and 𝑒_𝑗 where, for the purposes of this section, 𝑓(𝑥) is defined by (16). Expanding terms on both sides of this equation leads to the condition 𝑒_𝑖−𝑒_𝑗^𝑇𝑓(𝑥)1=−2𝑒^𝑇_𝑗𝑒_𝑗−𝑒^𝑇_𝑖𝑒_𝑖. (30) If the class membership of a representative vector is to remain invariant under a change of target vectors, this same set of points must also satisfy 𝑒_𝑖̃−𝑒̃_𝑗^𝑇𝑓(𝑥)1=−2𝑒^𝑇_𝑗̃𝑒_𝑗̃−𝑒^𝑇_𝑖̃𝑒_𝑖̃. (31) Assuming that two networks have been trained using two different sets of target vectors ℰ and ℰ, the set of weights {𝑊,𝜏,𝛼} and {𝑊,𝜏,𝛼} determines the network output in (16). Without loss of generality, we consider the case where all target vectors are normalized to a value of one such that ‖𝑒_𝑖‖²=1 and ‖𝑒_𝑖̃‖²=1 for 𝑖=1,…,𝑀. In this case, the conditions in (30) and (31) become 𝑒_𝑖−𝑒_𝑗^𝑇()𝑒̃𝛼𝜎𝑊𝑥−𝜏=0,_𝑖−𝑒̃_𝑗^𝑇𝛼𝜎𝑊𝑥−𝜏=0. (32)(33)

We first consider a special case where the target vectors are related according to Γ=𝑎𝑅Γ with 𝑇=0 in (2). Under these circumstances, if the choice 𝒲≡𝑊,𝜏,𝛼={𝑊,𝜏,𝑎𝑅𝛼} (34) is made, it should be clear that, since 𝑅^𝑇=𝑅⁻¹ and 𝑌_𝑖=𝑎𝑅𝑌_𝑖, 𝐸^𝑏𝑝=^𝑁_𝑖=1𝑓𝑋_𝑖−𝑌_𝑖^𝑇𝑓𝑋_𝑖−𝑌_𝑖=𝑎^2𝑁_𝑖=1𝑅𝑓𝑋_𝑖−𝑅𝑌_𝑖^𝑇𝑅𝑓𝑋_𝑖−𝑅𝑌_𝑖=𝑎^2𝑁_𝑖=1𝑓𝑋_𝑖−𝑌_𝑖^𝑇𝑅^𝑇𝑅𝑓𝑋_𝑖−𝑌_𝑖=𝑎²𝐸^𝑏𝑝 (35) is minimized by the condition 𝜕𝐸^𝑏𝑝/𝜕𝒲=0. Another way to see this is to observe that (19), (20) remain invariant for this choice of target vectors and network weights. Hence, we have the following.

Proposition 5. For a specific choice of 𝑎 and 𝑅, if 𝑇=0 in (2) and 𝒲=𝑊,𝜏,𝛼={𝑊,𝜏,𝑎𝑅𝛼}, (36) then the class membership for the backpropagation network will remain invariant.

Proof. Simply consider (32) and (33) and choose any 𝑥 satisfying 𝑒_𝑖−𝑒_𝑗^𝑇𝛼𝜎(𝑊𝑥−𝜏)=0. (37) It then immediately follows that 𝑒_𝑖̃−𝑒̃_𝑗^𝑇𝛼𝜎𝑊𝜏=𝑒𝑥−_𝑖−𝑒_𝑗^𝑇𝑅^𝑇𝑅𝛼𝜎(𝑊𝑥−𝜏)=𝑎²𝑒_𝑖−𝑒_𝑗^𝑇𝛼𝜎(𝑊𝑥−𝜏)=0. (38) Therefore, if 𝑥 satisfies (32), then it also satisfies (33) and, hence, is a point on the decision boundary for both networks.

Intuitively, a scaled, rigid rotation of the target vectors should not affect the decision boundary. However, when the more general transformation of (2) is applied with 𝑇≠0, we now demonstrate that, due to the nonlinearity of 𝜎 in (16), no simple relationship exists such that (32) and (33) can simultaneously be satisfied by the same set of points. We first investigate the possibility of establishing an analytic relationship between the set of weights {𝑊,𝜏,𝛼} and {𝑊,𝜏,𝛼} for both networks. In other words, we seek, ideally invertible, functions 𝐺_𝛼, 𝐺_𝜏, and 𝐺_𝑊. 𝛼=𝐺_𝛼(𝑊,𝜏,𝛼),𝜏=𝐺_𝜏(𝑊,𝜏,𝛼),𝑊=𝐺_𝑊(𝑊,𝜏,𝛼) (39) such that the set 𝒲 can be transformed into 𝒲. If this can be done, then an analytic procedure similar to that presented in the proof of Proposition 5 can be established in order to relate (32) to (33) for the general case. Since (19), (20) define the set 𝒲, it is reasonable to rephrase these equations in terms of the objective function 𝐸^𝑏𝑝: 𝜕𝐸^𝑏𝑝𝜕𝛼=^𝑁_𝑘=1𝑓𝑋_𝑘−𝑌_𝑘𝑆^𝑇_𝑘𝜕𝐸=0,_𝑏𝑝𝜕𝜏=−^𝑁_𝑘=1diag𝑆^′_𝑘𝛼^𝑇𝑓𝑋_𝑘−𝑌_𝑘^𝑇𝜕𝐸=0,_𝑏𝑝𝜕𝑊=^𝑁_𝑘=1diag𝑆^′_𝑘𝛼^𝑇𝑓𝑋_𝑘−𝑌_𝑘^𝑇𝑋^𝑇_𝑘=0, (40) where 𝑓(𝑋_𝑘)=𝛼𝜎(𝑊𝑋_𝑘−𝜏) and 𝑌_𝑘=𝑎𝑅(𝑌_𝑘−𝑡^′) such that 𝑡′ is the translation vector associated with the target vector referred to by 𝑌_𝑘. From these equations, it should be clear that no simple analytic relationship exists that will transform 𝒲 into 𝒲. A numerical algorithm such as gradient descent will, assuming a local minimum actually exists, arrive at some solution for both 𝒲 and 𝒲. We must therefore be content with the assumed existence of some set of functions defined by (39). Again, let us consider any point 𝑥 on the decision boundary such that 𝑒_𝑖−𝑒_𝑗^𝑇𝛼𝜎(𝑊𝑥−𝜏)=0. (41) Such a point must also simultaneously satisfy 𝑒_𝑖̃−𝑒̃_𝑗^𝑇𝛼𝜎𝑊𝜏=𝑥−𝑎𝑅𝑒_𝑖−𝑡_𝑖−𝑎𝑅𝑒_𝑗−𝑡_𝑗^𝑇𝐺_𝛼()𝐺𝑊,𝜏,𝛼×𝜎_𝑊(𝑊,𝜏,𝛼)𝑥−𝐺_𝜏()𝑒𝑊,𝜏,𝛼=𝑎_𝑖−𝑒_𝑗−𝑡_𝑖−𝑡_𝑗^𝑇𝑅^𝑇𝐺_𝛼()𝐺𝑊,𝜏,𝛼×𝜎_𝑊(𝑊,𝜏,𝛼)𝑥−𝐺_𝜏(𝑊,𝜏,𝛼)=0. (42) At first glance, a choice such as 𝐺_𝛼(𝑊,𝜏,𝛼)=𝑎𝑅𝛼 and 𝑡_𝑖=𝑡_𝑗 for 𝑖,𝑗=1,…,𝑀 (as in Proposition 4) appears to bring us close to a solution. However, the term involving 𝜎(𝐺_𝑊(𝑊,𝜏,𝛼)𝑥−𝐺_𝜏(𝑊,𝜏,𝛼)) is problematic. Although a choice such as 𝐺_𝑊(𝑊,𝜏,𝛼)=𝑊 and 𝐺_𝜏(𝑊,𝜏,𝛼)=𝜏 would yield the solution to (42), it should be clear that these values would not satisfy (40).

Another way to analyze this problem is to first set 𝑡_𝑖=𝑡_𝑗 for 𝑖,𝑗=1,…,𝑀. Then, for any 𝑥 on the decision boundary, from (41) and (42), equate the terms 𝛼𝜎(𝑊𝑥−𝜏)=𝑘𝑅^𝑇𝐺_𝛼()𝐺𝑊,𝜏,𝛼×𝜎_𝑊(𝑊,𝜏,𝛼)𝑥−𝐺_𝜏(𝑊,𝜏,𝛼), (43) where 𝑘 is some constant and 𝐺_𝛼, 𝐺_𝜏, and 𝐺_𝑊 satisfy (40). Given an arbitrary training set defined by (1), it is highly unlikely that this constraint can be satisfied. One remote scenario might be that the terms 𝐺_𝑊(𝑊,𝜏,𝛼)𝑥−𝐺_𝜏(𝑊,𝜏,𝛼) and 𝑊𝑥−𝜏 are always small. In this case, given a sigmoidal function 𝜎(𝑧) that is linear near 𝑧=0, a linearized version of (43) could be solved using techniques described in [32]. However, this again is an unlikely set of events given an arbitrary training set. Therefore, given the transformation of (2), we are left to conclude that class membership invariance and, hence, ranking invariance are, in general, not achievable using the backpropagation neural network.

5.2. Discussion

Intuitively, given a reasonable target encoding scheme, one would desire that properties related to protein secondary structure would be independent of the target vectors chosen. However, we have presented numerical data and a theoretical foundation demonstrating that secondary structure classification and confidence measures can vary depending on the type of neural network architecture and target vector encoding scheme employed. Specifically, linear network classification has been demonstrated to remain invariant under a change in the target structure encoding scheme while the backpropagation network has not. As 𝑁 increases, for the methodology applied in this work, recognition rates remain consistent with those reported in the literature; however, we have observed that adding more training data does not improve the invariance of classification measures for the backpropagation network. This conclusion is corroborated by the analytic formulation presented above.

6. Conclusions

As pointed out in the introduction, one major purpose of the neural network is to create a stable and reliable model that maps input training data to an output classification with the hope of extracting informative parameters. When methods similar to those in the literature are applied [7, 12–14], we have demonstrated that classifier performance measures can vary considerably. Under these circumstances, parameters derived from a trained network for analytically describing protein secondary structure may not comprise a reliable set for the model-based approach. Furthermore, classifier variability would imply that a stable parametric model has not been derived. It is in some sense paradoxical that the neural network has been applied for structure classification and, yet, associated parameters have not been applied for describing protein secondary structure. The neural network approach to deriving a solution to the protein secondary structure prediction problem therefore requires deeper exploration.

Acknowledgments

This publication was made possible by Grant Number G12RR017581 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH). The authors would also like to thank the reviewers for their helpful comments.