Abstract
We suggested different structured hybrid systems for the sentence-level subjectivity analysis based on three supervised machine learning algorithms, namely, Hidden Markov Model, Fuzzy Control System, and Adaptive Neuro-Fuzzy Inference System. The suggested feature extraction algorithm in our experiment computes a feature vector using statistical textual terms frequencies in a training dataset not having the use of any lexical knowledge except tokenization. Taking into consideration this fact, the above-mentioned methods may be employed in other languages as these methods do not utilize the morphological, syntactical, and lexical analysis in the classification problems.
1. Introduction
Identification of subjective data from web documents having opinions within are gaining incrementing interest. Opinions are often views formed by individuals about their sentiments, appraisals, or feelings, etc., not necessarily based on fact or knowledge. Identification of subjectivity attempts to recognize if this written piece of work conveys opinions (personal) or a body of objective facts [1]. This analysis has been utilized in many natural language and text mining solutions. With the aim of generating more instructive data, subjectivity detection has been employed as a primary sifting stage in a lot of natural language processing assignments.
Through our experimentation we are aiming to work out techniques in order to establish classifiers able to identify subjective expressions from objective ones. By means of language independent feature weighting, in the experiment the sentence-level subjectivity classification is attained. A subjectivity database from the opinions about films of “Rotten Tomatoes” [2] was deployed as an experiment.
In the paper, we suggested different structures of hybrid systems based on various supervised machine learning algorithms such as Hidden Markov Model (HMM), Adaptive Neuro-Fuzzy Inference System (ANFIS), and Fuzzy Control System (FCS) which achieved sufficient results. These machine learning methods have been employed for subjectivity analysis individually [3, 4] and our aim is to improve performance of classification by using hybrid systems, which is successfully applied by us in sentiment analysis and natural language call routing problem [5, 6]. Our feature extraction algorithm computes a feature vector using the statistical textual terms frequencies in the corpus not having the use of any lexical knowledge except tokenization. Taking into consideration this fact, the above-mentioned methods may be employed in other languages as these methods do not utilize the lexical, grammatical, and syntactical analysis within the classification process.
2. Related Work
With the aim of recognizing the subjective data in written piece of work or speech, various nonsimilar supervised and unsupervised learning algorithms have been recently analyzed.
A bootstrap technique was employed by Riloff and Wiebe [7] with the aim of studying subjectivity classifiers from a nonannotated texts collection. An identical method was applied by Wiebe and Riloff [8]; however, they studied objective sentences aside from subjective sentences.
With the objective to classify subjective and objective sentences, a MinCut based algorithm was deployed by Pang and Lee [9]. Through this experiment they targeted eliminating objective expressions from each analysis and ameliorating the classification accuracy up to 86.4%.
A Web mining technique was offered by Grefenstette et al. [10] in order to identify subjective adjectives.
Techniques of classifying the strength of opinion within personal sentences were suggested by Wilson et al. [11] and Kim et al. [12].
Subsumption relationships were defined by Riloff et al. [13] within n-grams, unigrams, and lexico-syntactic samples. It was discovered that, in case of subsumption of one feature by another, there is no need for subsumed feature. The subsumption pecking order eliminates a feature set; hence the eliminated feature sets may ameliorate classification functionality.
The utilization of prosodic feature, word -grams, character -grams, and phoneme -grams for subjectivity identification and polarity subsumption of conversation acts in multiparty dialog was researched by Raaijmakers et al. [14]. It was figured out that, for subjectivity detection, an association of phoneme-level information, word-level, prosodic, and character-level provides better performance. The best performance is acquired with an association of phonemes, words, and characters in polarity classification.
By utilizing -gram word successions with fluctuating level of lexical instantiation, the study of patterns of subjectivity available both in labelled and unlabelled information was suggested by Carenini and Murray [15]. It was demonstrated that study of subjective written words with fluctuating instantiation levels from both commented as well as unprocessed information may ameliorate subjectivity recognition and polarity labelling for speech in the meetings and email messages.
Delta TFIDF which is the intuitional general purpose method was suggested by Martineau and Finin [16] to weigh word scores effectively prior to classification. The comparison was made between the results of TFIDFs SVM difference and SVM Term Count Baseline for subjectivity subsumption. The comparison outcomes demonstrated that Delta TFIDF SVM gives low variance with high accuracy.
The subjectivity of postings on Twitter was categorized by Barbosa and Feng [17] according to two feature types such as metainformation about the terms on tweets and specifications of how tweets have been written.
With the aim of recognizing sentence-level subjectivity, Yulan He [18] suggested SubjLDA through adjusting the unused Dirichlet allocation method by including an additional layer to describe subjectivity meaning.
The classification of subjectivity at the segment level was suggested by Benamara et al. [19] for discourse-based sentiment analysis. Each component was classified into four groups such as S, OO, O, and SN wherein S components comprise unambiguously lexicalized subjective as well as appraising phrases, OO components are positive or negative expression embodied in an objective component, O components comprise neither a word with lexicalized subjectivity nor embodied opinion, and SN components are subjective, however nonappraising, which are utilized to express thoughts.
Remus [20] showed that using readability formulae and their combinations as features in addition to already well-known subjectivity clues leads to significant accuracy improvements in sentence-level subjectivity classification.
Bayesian model with pecking order grounded on Latent Dirichlet Allocation, called subjLDA, for sentence-level subjectivity recognition that reflexively determines if the used expression is sentiment or fact was proposed by Lin et al., [1].
Each of these models is based on English information and an English subjectivity dictionary is utilized by most of models. Some models have been recently presented on sentence subjectivity classification in Japanese [21], Chinese [22, 23], Romanian [24, 25], Urdu [26], and Arabic [27] and other models used various machine learning algorithms utilizing universal set of features as well as language specific features.
With the aim of automatically generating resources for a new language through utilizing the resources and instruments for English for subjectivity analysis, various models were researched by Mihalcea et al. [25] and Banea et al. [24]. With the objective of establishing most accurate classifiers for subjectivity classification; another method was employed by Banea et al. [28].
Some methods which aim to discover features which can be employed in other languages have been recently investigated. For instance, the classification of subjectivity which utilizes features which does not depend on language was suggested by Mogadala and Varma [29] and tests were conducted on 5 various languages.
Appel et al. [30] proposed a hybrid approach using SentiWordNet [31] and fuzzy sets to estimate the semantic orientation polarity and intensity of sentiment words, before computing the sentence-level sentiments. Muhammad et al. [32] introduced a lexicon-based sentiment classification system for social media genres, which captures contextual polarity from both local and global context. Fernández-Gavilanes et al. [33] proposed a novel approach to predict sentiment in online texts based on an unsupervised dependency parsing-based text classification method [34].
There are many other types of sentiment analysis that have been developed, such as irony detection [35], multilingual support [36], and differentiating negative emotions [37].
The application of supervised models using language independent features for classification of objective and subjective expressions is our primary target within the present experiment.
3. Feature Extraction
Feature extraction algorithms are an important element of any machine learning model. This algorithm is characterized by us as instinctive and computationally effective which does not demand any supplementary comment by individuals as well as lexical knowledge except tokenization. The mentioned algorithm comprises two parts: preprocessing (information preparation) and computation of the feature vectors.
4. Preprocessing
5000 objective and 5000 subjective labelled sentences from movie reviews are utilized for training process [9]. The classification based on machine learning uses two several sets of documents such as a training set and a test set. The training set is utilized by an automatic classifier with the aim of learning the distinguishing features of documents, and the test set is utilized to authenticate the performance classifier. We currently illustrate the information dispatching in the dataset. With the aim of forming the word list, the following proceedings are implemented:(i)Gather all documents from the training set and create one whole document;(ii)Transfigure content of the document to term matrix;
Taking into consideration that our goal is to avoid lexical knowledge utilization, every term is considered as a single code term. According to our algorithm characteristics, verbs are not sorted in particular tenses (i.e., “contribute” in present tense and “contributed” in past tense) as well as nouns in singular or plural (i.e., “infant” in singular and “infants” in plural). In lieu, words are considered as different terms.
We parted the data collection stochastically into 2 parts for testing and training and created 10 folds. 90% of the data in each folder is used for training and 10% for testing, respectively.
5. Calculation of Feature Vectors
Frequency statistics analysis is used in majority of language independent feature extraction algorithms. Its periodically weighting functions are utilized. In the present research, we endeavored to scrutinize which features were appropriate for Neuro-Fuzzy models.
Let us describe some of the parameters:1. is a number of terms in the training set;2. is a number of classes;3. is a number of sentences in the training set;4. are the sentences in the training dataset, where is the length of rth sentence, ;5. describes the association between ith term and the jth class ;6. is the number of times ith term occurred in the jth class;7. denotes the occurrence times of the ith term in the training set;8.frequency formula for the th term in the th class
A new weighting factor having an impact on the system accuracy is applied so that in lieu of the quantity of documents we take the quantity of classes in the prevalent Inverse-Document Frequency (IDF) formula. Like IDF, we distinguish this coefficient as Pruned ICF (Inverse-Class Frequency) [3].where is a term, is the number of classes containing the term , in which , where is found empirically for the current problem.
The membership degree of the terms () for relevant classes may be evaluated by specialists or can be computed by means of empirical methods. Taking into consideration that we target not to utilize human comments or lexical knowledge, we estimated the membership degree of every word through an analytical method as mentioned below :
6. Subjectivity Detection Using Fuzzy Control System
Fuzzy inference is the process wherein the mapping is formulated from given data to outcome(s) through utilizing fuzzy logic. By means of a basis provided by this mapping decisions can be made, or samples can be distinguished. Fuzzy inference process draws in logic operations if-then rules and membership functions [38]. In lieu of expert knowledge, in the primary phase with the aim of computing membership function, we apply a statistical method. Afterwards, we employ fuzzy operations and through the backpropagation algorithm we adjust parameters. The general structure of Fuzzy Control System is illustrated in Figure 1.
Our algorithm () is applied as follows.(1)() - membership degree of terms of the th sentence is computed by using (4)-(5).
The Center of Gravity Defuzzification (CoGD) technique is employed for defuzzification. This method prevents defuzzification ambiguities, in case output degree of membership is calculated from several crisp output values.
Objective and subjective expressions classified in accordance with classes are trained by a FCS.
The cost function is expressed as follows [39]:, is the desired output.
The function’s partial derivatives are computed as follows:
We applied conjugate gradient decent method for minimization of function (6) and used optimal value in next stage.
Rounding value of gives index of appropriate class, if it satisfies the following condition:where and is the indicator of the relevant class. In this case, is the primary quantity, which affects to the system reliability.
Inspection of feature vector providing the best outcomes for Fuzzy Control System is obvious. Average accuracies of FCS over 10-fold cross validation are illustrated in Table 1. It must be emphasized that these outcomes hinge on the classification technique and might be dissimilar for every classifier.
Delta TFIDF features [16] related to FCS were inspected as well. Due to fact that Delta IDF weighting factors of both classes are identical, the accuracy of the system is not modified in case of employing Delta IDF weighting factor. As it is seen from Table 1, the accuracy of the system increases following the deployment of Pruned ICF weighting coefficient. Recognition results of FCS with different values of are given in Table 2.
The rejection result is 0.01 for . It means after applying Pruned ICF weighting 0.01% expressions’ terms become 0 in the testing process.
7. Adaptive Neuro-Fuzzy Inference System For Subjectivity Detection
General structure of Adaptive Neuro-Fuzzy Inference System is demonstrated in Figure 2 [40]. Regarding the linguistic notes, the fuzzy interface block gives an input vector to a Multilayer Artificial Neural Network (MANN).
We utilized statistical evaluation of membership degree of words by (5) in lieu of linguistic statements in the primary phase. Afterwards, we employed (3) and (4) fuzzy operations.
MANN is applied to the output of the fuzzification operation. The structure of MANN in ANFIS is illustrated in Figure 3.
The main parameters of MANN are described as follows.1. is the number of MANNs layers;2. is the number of neurons on layer , ;3. is set of neurons of layer , which connected to the neuron on layer ;4. is bias of neuron on layer ;5. is weighted coefficient (synapse) of connection between neuron on layer and neuron on layer ;6.is the input vector; the output of fuzzification is given to the MANN as input, ;7. and are state and output value of neuron on layer for input signal of the neural networks.where is a given nonlinear activation function for which we use the hyperbolic tangent
Let the training set , pairs be given, where is desired output for input signal. The training of MANN consists in finding such and , herewith on input signal that MANN has output , which is optimally close to the desired output . MANN’s outputs are taken as indexes of classes appropriate to the sentences. Usually, training quality is defined by the mean square error function [41]: where is coefficient, which determines the belonging “quality” of input to its “ideal” pattern .
The task of MANN training stipulates to minimize the criterion (14) according to parameters with (10)-(12) requirements. The MANN in the developed system was trained using the conjugate gradient method.
We set two thresholds for the acceptance recognition:1.,2..
is the output of Multilayer Artificial Neural Network, and are maximum two elements of the vector . It can be described as follows:
Subjectivity detection results of ANFIS with various and are given in Table 3.
The accuracy of FCS (92.02%) is less than the accuracy of ANFIS (92.16%) due to extra variables in the hidden layers of the MANN.
8. Subjectivity Detection Using Hidden Markov Model
HMMs are a substantial statistical method for modeling generative sequences that can be categorized by a fundamental process generating a perceptible sequence. HMMs have been successfully applied in speech recognition. They have also been used successfully in some NLP problems such as part-of-speech tagging, expression yielding, and excerpting necessary data from documents [42].
With the aim of classifying objective and subjective expressions, a discrete ergodic HMM was employed. We parted expressions into a lot of states and yielded terms comprising these states. Gathering such states provides us with better outcomes and does not use knowledge of language.
Let us describe main parameters of HMM as follows [4]:1. is the initial state distributions: , where N is the number of states.2. is the state transition probability matrix, , .3. are the state-dependent observation probabilities. Here, for every state j, is the probability distribution of words occurring in states.4. are the observation sequences, where R is the number of observed sequences, is the length of r-th observed sequence, and , T is the given quantity, .
We applied Baum-Welch algorithm for the training of HMMs for all classes. The scaled-forward algorithm is applied for the estimating probability of classes for each observed sentence in testing process. The maximum probability among probabilities of classes is found as a decision (Figure 4).
The HMM outcomes with various states are illustrated in Table 4. Experiments were carried out for the ergodic and left-right Hidden Markov Models. According to experiments ergodic HMMs with 2 states show best performance.
9. Hybrid Systems
With the aim of classifying the subjectivity and objectivity of sentences, we suggest employing a combined system utilizing the HMM, FCS, and ANFIS methods. Within the experiment process we analyze each sentence by all methods. The outcomes of HMMs, FCS, and ANFIS are directed to the decision-making block and compared therein.
Two types of hybrid systems are suggested.
9.1. Hybrid-I
The outcomes verified by the HMMs, FCS, and ANFIS methods are confirmed by this system. In case of decline of some of these models’ classification, no decision is accepted by the system. As the system forestalls the error in the testing process, it is considered more reliable. The results of Hybrid-I are shown in Table 5.
9.2. Hybrid-II
We suggest a sequential method in this system. In accordance with the procedure, in case of failure by one classifier to classify a sentence, the unclassified sentence will be passed to the next classifier up to classification of the given sentence or no availability of another classifier. By means of this method, the number of rejected sentences is reduced.
At first ANFIS is applied to rejected sentences by FCS and then consequently ergodic HMMs with 2 states are applied to the rejected sentences by ANFIS in Hybrid-II. By this way we increase classification rate up to 92.24%. The results of Hybrid-II are given in Table 6.
10. Conclusion
Three various classification system methods such as FCS, ANFIS, and HMM were depicted and employed by us for detecting the sentence-level subjectivity in a film analysis database. Training and testing mechanisms of these methods have been particularly demonstrated by us to classify sentences as objective and subjective. Through this analysis we targeted developing methods which would be capable of being applied in various languages since they do not depend on linguistic knowledge. The feature extraction proceeding is a significant element of these methods. We concentrated on study of informative features enhancing the precision of the systems which do not have language specific restrictions. Thus “Pruned ICF Weighting Function” novel was conceived with a parameter specially calculated for the subjectivity data set. While bringing the already created system in comparison with other different systems, we should foreground that, in fact, the utilization of linguistic knowledge increases precision. Due to fact that linguistic knowledge is not used in our method, our output may solely be compared with methods having same restrictions which utilize components based on bag of terms that are investigated on the same data collection. Researches of Pang and Lee [9] and Martineau and Finin [16] are included in those examples. By means of Naıve Bayes classifiers Pang and Lee achieve 92% accuracy on sentence-level subjectivity classification and through utilization of SVMs they report 90% on the same set of data. 91.26% accuracy was achieved by Martineau and Finin [16] utilizing SVM Difference of TFIDFs. The presently achieved outputs are similar as follows: FCS (92.02%), ANFIS (92.16%), and HMM (89.63%). Notwithstanding these indications, the methods we suggested have certain superiorities. Since function (6) is reduced only regarding (in the identified problem N=2), FCS has low complexity compared to other sophisticated supervised machine learning methods. Through extra fluctuations added in the central layer of the neural network, ANFIS may slightly increase accuracy. In case of applying IF-THEN rules in ANFIS and FCS, it is assumed that the accuracy of the system will increase to the level where it gets abreast with human discernment. FCS and ANFIS categorize sentences by dint of occurrence of the words in the output; however the HMM categorizes sentences according to the arrangement of the sentences. The given interpretation can be utilized to establish two various types of hybrid systems. Through combining multiple classifiers, a better performance than the result of any classifier can be achieved. The Hybrid-I system forestalls particular errors in the classification process, and the Hybrid-II system improved the accuracy to 92.24%.
Conflicts of Interest
The author declares that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This work has been carried out in Center for Data Analytics Research at ADA University.