Abstract
Objective. The Bayesian model plays an important role in diagnostic test evaluation in the absence of the gold standard, which used the external prior distribution of a parameter combined with sample data to yield the posterior distribution of the test characteristics. However, the correlation between diagnostic tests has always been a problem that cannot be ignored in the Bayesian model evaluation. This study will discuss how different Bayesian model, correlation scenarios, and prior distribution affect the outcome. Methods. The data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. The diagnostic character of T-SPOT.Tb and KD38 tuberculosis antibody test were evaluated in different Bayesian model, and discharge diagnosis as a gold standard was used to verify the model results in the end. Result. The comparison of four models under the conditional independence situation found that Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. The results were mainly affected by prior information. The sensitivity and specificity of the two tests in Conditional Covariance Bayesian model in prior constraint situation were considerably higher than the Bayesian probabilistic constraint model in prior constraint situation. The results of the four models under the conditional dependence situation were similar to the conditional independence situation; was also negative with no prior constraint situation in both model Bayesian probabilistic constraint model and Conditional Covariance Bayesian model. The Deviance Information Criterion of Bayesian probabilistic constraint model was close to model Conditional Covariance Bayesian model, but of Conditional Covariance Bayesian model in Prior constraint situation (=2.40) was higher than the Bayesian probabilistic constraint model in Prior constraint situation (=1.66). Conclusion. The result of Conditional Covariance Bayesian model in prior constraint with conditional independence situation was closest to the result of gold standard evaluation in our data. Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of the gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.
1. Introduction
Sensitivity and specificity as the reference value of the ability to detect sick and healthy patients are used in diagnostic test evaluation with a gold standard test. However, in clinical practice, the gold standard tests are not given in patients due to expensive or invasive reasons [1]. The absence of a gold standard is a common problem in clinical practice and diagnostic research studies.
Some studies try to evaluate the diagnostic test characteristics by combining multiple diagnostic tests in the absence of a gold standard [2, 3]. Due to the fact that the sensitivity and specificity of diagnostic tests in the estimation process are unknown variables, the biggest difficulty is that the number of parameters of estimation exceeds the number of degrees of freedom provided by the data. For example, when two nongold standard diagnostic tests are used, only three degrees of freedom are provided, but the sensitivity and specificity of the two tests and the prevalence of the disease need to be estimated for at least five unknown parameters; if the correlation between the two tests is considered, there are more parameters to be estimated.
In classical statistical view, sensitivity and specificity are regarded as fixed parameters and the population prevalence is calculated from them. However, it has been proved that sensitivity and specificity are not fixed values, but change with external factors [4, 5]. The sensitivity and specificity of diagnostic tests in the estimation process are unknown, and their values are often independent of the sample data [6]. According to the Bayesian view, any unknown parameter can be regarded as a random variable, and its unknown state can be described by a probability distribution. This probability distribution is called a prior distribution; the prior constraints on Bayesian methods can compensate for the lack of freedom. Of course, prior information needs to be specified by external data, which can be the expert opinion or historical research.
Bayesian methods have been increasingly used to evaluate the true accuracy of diagnostic tests in the absence of a gold standard [7–9] for two reasons. On the one hand, prior information in the Bayesian framework about the sensitivities and specificities of the tests can be obtained from experimental results or other studies; if there is no data source, it can be replaced by an expert prior [10]. The Bayesian analysis allows us to combine external prior information with the data likelihood to yield the posterior estimation of unknown parameters such as the prevalence and diagnostic test characteristics [11]. On the other hand, with the development of computer technology and professional Bayesian analysis software such as OpenBUGS, the computational problems in the Bayesian method have been solved by the efficient Markov chain Monte Carlo (MCMC) algorithms for sampling and summarizing posterior distributions [10].
In the combined application of multiple diagnostic tests, the interdependence between different tests also needs to be considered in the Bayesian model. If two tests have the same biological attribute, it is logical to believe that the tests are conditionally dependent; if the result is positive in one test, the result of another test is likely to be positive [12]. Several approaches try to take conditional interdependence of different tests into account in Bayesian models. One method is to calculate the correlation coefficient directly and incorporate the covariance into the Bayesian model [13]. The other method is to use probabilistic constraints to transform the interdependence of different tests into conditional probabilities and then to construct a Bayesian model based on conditional probabilities [14]. Both approaches will have their application scenarios.
Under the basic Bayesian framework, the Bayesian method is very flexible when considering various influencing factors. The correlation scenarios, prior distribution, and the number of the prior parameters are the important factors that cannot be ignored in the Bayesian estimation. The objective of this study is to compare the two Bayesian methods under different scenarios with tuberculosis (TB) data and to explore the application scenarios for each of the two Bayesian methods.
2. Methods
2.1. Study Patients and Diagnostic Tests
The data analyzed in this study was gathered during studies of patients presenting to the Nanjing Chest Hospital with suspected tuberculosis. In brief, a case report of patients was collected between June and October 2015 at the Nanjing Chest Hospital. Informed consent was completed for all participated in the study. T-SPOT.Tb and KD38 tuberculosis antibody test was combined as the nongold standard diagnostic test to estimate the prevalence and diagnostic test characteristic. The discharge diagnosis was used to verify the model results.
2.2. Conditional Covariance Bayesian Model
2.2.1. The Description of Conditional Covariance Bayesian Model
Conditional covariance Bayesian model directly considers the correlation between the two tests and estimates the conditional correlation between two diagnostic tests using the covariance between tests within the diseased and nondiseased populations. Two diagnostic test evaluation models without a gold standard were shown in Table 1, and parameter explanation was listed as follows: (1) : i=0,1 represents the negative and positive results of the Test1, respectively, and j=0,1 represents the negative and positive results of the Test2, correspondingly; (2) t represents the number of real patients; (3) n represents the total number; (4) : is the positive potential true value corresponding to ; and (5) : is the negative potential true value. Potential true values are those that cannot be observed directly but are close to the gold standard under certain conditions. In the Conditional Covariance Bayesian model, the conditional correlation between the two diagnostic tests was estimated by calculating the covariance between tests within each disease class [12, 15]. Bayesian conditional covariance models use the method of Nandini Dendukuri et al. and the description of the methods partly reproduces their wording [13]. Model construction in both independent and dependent scenarios was provided as supporting information (Text S1).
2.2.2. Likelihood Function of the Conditional Covariance Bayesian Model
Vector represents the actual result of two diagnostic tests (1 is positive, 0 is negative); the probability value P corresponding to vector a is equal to . The sensitivity of the first and second tests is and , the specificity of the two tests is and .
(a) The Covariance under Positive Conditions of the True Result
(b) The Covariance under Negative Conditions of the True Result
(c) The Correlation Coefficient under the True Disease Condition is Positive
(d) The Correlation Coefficient under the True Disease Condition is Negative
(e) Likelihood Function. The likelihood function of the Conditional Covariance Bayesian model is a multinomial likelihood function.
2.2.3. Prior Information about the Conditional Covariance Bayesian Method
According to the Bayesian principle, the conjugate distribution of the binomial distribution is the beta distribution. The prevalence, sensitivities, and specificities are assumed to follow beta prior distribution, list like the following:The prior information of the above parameters () was gathered from the previous study in China. For example, the prior information of sensitivity and specificity for the T-SPOT method was gathered from 18 previous similar published researches in a different area of China. According to the data (mean, standard deviation) calculated from the historical prior, the parameters (a and b) of the prior beta distribution of unknown variables are obtained (Table 2).
In practical application, there is a lack of available prior information in covariance and ; the covariance is random variables varying in a finite range as follows:Since the positive correlation between the two tests is the actual consideration, but the lower limit value in the above expression is always negative, the lower limit value was artificially fixed to zero. Only when the distribution is uniform, the entropy can reach the maximum value. So the prior distribution is defined as follows:
2.3. Bayesian Probabilistic Constraint Model
2.3.1. The Description of Bayesian Probabilistic Constraint Model
The conditionally independent assumption is usually made when two diagnostic tests were combined. However, the conditionally independent assumption cannot easily be made when the two diagnostics have a similar biologic mechanism; extra information will be required in the estimation process [16]. When the number of estimable parameters exceeds the number of parameters to estimate, the Bayesian probabilistic constraint model is to add the constraint on the parameters. These constraints usually come from external information, such as historical study or expert opinion. In the Bayesian method, we call it prior information, and it is also the explanation for the constraint of the Bayesian probabilistic constraint model.
However, in some cases, it is often difficult to directly specify external prior information for some parameters, such as the covariance in the Conditional Covariance Bayesian model. The prior distributions for the covariance are quite difficult to elicit from experts or other studies, because they are not the indicator used in a real-life situation. In Bayesian probabilistic constraint model, prior information on conditional probabilities is easier to specify [14]. Therefore, in the Bayesian probabilistic constraint model, the correlation coefficients between the two diagnostic tests are not calculated directly, some restriction will be imposed on the parameter estimates. We just elicited the information from experts on the conditional performance on one test given the results of another test. The Bayesian probabilistic constraint model uses the method of Berkvens, D. et al. and the description of the method partly reproduces their wording [14]. Model construction in both independent and correlated scenarios was provided as supporting information (Text S2).
2.3.2. Likelihood Function of the Bayesian Probabilistic Constraint Model
The likelihood function was used to express the cell probabilities of the collapsed , the table in terms of the prevalence of the disease, indicated that the subject was (was not) diseased; indicated a positive (negative) result with test T. also indicated the test condition (0 indicated a negative result, and 1 indicated the positive result). An example was listed as follows:
Likelihood function listed as follows:
2.3.3. Prior Information about the Bayesian Probabilistic Constraint Model
The prior information about Bayesian probabilistic constraints model was collected from four experienced tuberculosis physicians. The tuberculosis physician answered the probability of the parameter under the defined question. After obtaining the expert answer, the mean of each parameter had been calculated, and the prior distribution also had been specified. This study was a joint evaluation of two diagnostic tests, - were used for conditional probabilities, and the meaning of the specific reference to each conditional probability was found in S2. In this article, we assumed that the prior distribution of each conditional probability obeyed the beta distribution with two parameters (alpha and beta); specific information was listed in Table 3.
2.4. Model Evaluation and Verification
All parameters in two Bayesian methods were estimated with 95% credible intervals using OpenBUGS 3.2.3[17]. The OpenBUGS code of this study was provided as an attachment file (Text S3). Deviance information criteria (DIC) were used to evaluate the models fit and to verify whether the prior information is against data results [18]. During the model building process, the DIC was minimized, it aims to find the simplest and best-fit model, and the lower the DIC value, the simpler and fitter the model [19]. The number of parameters also represented the complexity of the model and indicated the final reduction in the number of parameters needs to be estimated.
The prediction accuracy of the different models was evaluated using clinical discharge diagnosis as the gold standard. The clinical discharge diagnosis was a comprehensive judgment made by doctors according to various diagnostic tests, expert experience, and disease progression.
3. Results
In total, 637 patients with suspected tuberculosis were included in the study. The mean age was 50.12 years (range 15-90 years); 61.3% of the patients were male and 38.7% were female. 130 patients (20.41%) were negative for T-SPOT.TB test and KD38 tuberculosis antibody test, 235 patients (36.89%) were positive for both of them, the four possible combinations of results for the two tests were listed (Table 4).
3.1. Conditional Independence Situation
Four models under conditional independence situation were applied to the data for the two tests, which assumed that the result of the first test had no influence on the result of the second test. Using the observed of the two tests as the sample data, combined with prior information, we calculated the posterior distribution of sensitivity and specificity of the two tests (Table 5). Under the premise of the conditional independence situation, the Bayesian probabilistic constraint model was consistent with the Conditional Covariance Bayesian model. Therefore, the results of the two models were the same with no prior constraint. Under the condition of prior situation, the results were affected by prior information. The sensitivity and specificity of the two tests in model PC were considerably higher than those predicted in the model PP. The tuberculosis prevalence was estimated to be 63.6% (95% credible interval 43.5%-77.3%) in model PC, being considerably higher than model PP (53.4%, 95% credible interval 45.2 %-61.4%). DIC and of the different model under conditional independence situation were compared (Table 6). was negative with no prior constraints in both NP and NC models, which indicated all our parameters were estimable. The model PC reduced the number of parameters () which was 2.26 and had smaller DIC than the model PP.
3.2. Conditional Dependence Situation
Conditional dependence situation assumed that the two diagnostic tests could be correlated. The posterior distributions of sensitivity and specificity of the two tests under conditional dependence situation were evaluated by four models (Table 7). Whether or not there was a prior constraint, the posterior estimation results of five parameters in the Conditional Covariance Bayesian models were higher than Bayesian probabilistic constraint model. The result of the four models under conditional dependence situation was similar to the conditional independence situation, especially in the case of models with prior constraint. The DIC of the different model under conditional dependence were compared (Table 8), which were also negative with no prior constraints in both model NP and NT. The DIC of model PP were close to model PC, but the of the model PC (=2.40) was higher than model PP (=1.66).
3.3. Impact of Prior Number
The Conditional Covariance Bayesian model was chosen to explore the influence of the prior number on the posterior estimation because it has only five unknown parameters corresponding to only five prior distributions, which was convenient for simulation studies. When the number of priors was equal to n, it means that the rest of the prior (5-n) was prior without information. From the results of simulation under conditional independence, when the prior number was three, the estimation result and the model were stable (Table 9), which were very close to the full prior estimation results. Similarly, models and the results were stable when the number of prior information was three in conditional dependence situation (Table 10).
3.4. Model Validation
The patient discharge diagnosis was used as the gold standard to evaluate the sensitivity and specificity of two diagnostic tests (Table 11), tuberculosis prevalence in our population was estimated to be 82.9% (95% confidence interval 79.7%-85.6%), the sensitivity and specificity of T-SPOT.TB test were 0.739 and 0.670, and the sensitivity and specificity of KD38 tuberculosis antibody test were 0.549 and 0.761. The result of the model PC in conditional independence situation was closest to the result of the gold standard evaluation.
4. Discussion
With the development of computer technology and Bayesian theory, Bayesian model has been widely used in the practice of medical research. In the evaluation of diagnostic tests, when the real disease status is unknown and there is no gold standard, Bayesian method can be used to integrate external prior information and sample data to evaluate diagnostic test characteristics by combining two or more imperfect tests [20, 21]. However, due to the flexibility of the Bayesian model, its estimated results are affected by many factors. The consideration of correlation and the choice of prior distribution are the most important influencing factors for the posterior estimation.
The result of the different model indicated that the estimate of prevalence rate and diagnostic test characteristics depends on the model chosen, prior selection, and dependencies between tests. Compared with the gold standard verification, the result of model PC in conditional independence situation was closest to the result of the gold standard evaluation. The reasons for the above phenomenon may be as follows: firstly, the weak correlation between diagnostic tests cannot have a significant effect on the result; secondly, the prior constraint model can reflect the real situation of the diagnostic test than the nonpriority constraint model; thirdly, the objective prior information from previous studies is more accurate than expert opinion.
The dependencies between diagnostic tests have always been a key issue for Bayesian models. The results of the two Bayesian models showed that the change for the possibility of conditional dependence between diagnostic tests had a certain impact on the posterior estimates of diagnostic test characteristics. The two Bayesian methods deal with the conditional dependencies between tests in different ways. Conditional covariance Bayesian method combined prior information on covariance parameters with the test result to calculate the posterior distribution of the correlation coefficients. However, obtaining the prior distribution of covariance from experts or literature is pretty hard, because it is not specific parameters in a real-life situation. In addition, the complex correlation will be difficult to estimate with multiple diagnostic tests. In order to overcome this problem, the Bayesian probabilistic constraint model does not directly calculate the correlation coefficient, it just elicits prior information for experts on the conditional performance on one test given the results of another test, and this can be easier to answer by experts in a real-life situation. However, such prior information from this model is the expert subjective opinion, and its credibility is not better than objective prior information.
Our results showed that the likelihood functions of the two Bayesian methods were consistent with the conditions of independence situation, and the posterior estimation strongly depended on the prior information. The results of the two Bayesian methods both illustrated that posterior estimation was mainly affected by the available prior information. Hence, it is very important to elicit the prior distribution accurately. On the one hand, the objectivity of prior information is crucial. In the Conditional Covariance Bayesian method, the prior distribution of unknown parameters can be gathered from previous studies, and objective prior information is suggested to ensure the credibility of the result. In the Bayesian probabilistic constraint model, it may be easier to specify expert prior information for unknown parameters, but it is also significant to realize that the unstable expert opinion may have a great impact on the result; if you use the prior by different experts, you may end up with distinctive conclusions. On the other hand, the number of prior information as well has an important effect on the stability of the results. As we all know, the more the number of a prior, the more accurate the result, but it will increase the burden of obtaining prior information. Our results show that three prior distributions can achieve full prior results in the Conditional Covariance Bayesian method. Therefore, obtaining stable results based on minimal prior information is the best choice.
In fact, the influences of prior information and dependencies on the results are inseparable. Because the correlation coefficient itself is an unknown parameter, it also requires the prior distribution. In the evaluation of diagnostic tests in the absence of the gold standard, many factors should be considered in the method selection. DIC is also an important index of the model selection. Both the Conditional Covariance Bayesian method and the Bayesian probabilistic constraint method have their specific applicable scenarios; the users should choose the appropriate method according to the needs of the actual situation. When there are only two diagnostic tests and the correlation coefficient can be objectively specified, the Conditional Covariance Bayesian method is more applicable. The Conditional Covariance Bayesian method could also be extended to include more than two tests by adding more covariance in the model. At this time, the calculation of covariance will become complex, and the determination of prior distribution will be more difficult. Hence, from the point of view of practical application, the Bayesian probabilistic constraint method is more suitable when there are more than two combined diagnostic tests without gold standards. Finally, although these two methods are not perfect, they provide a feasible way for the evaluation of diagnostic test in the absence of a gold standard diagnostic; at the same time, it is of great significance to promote the application of Bayesian method in medical research.
5. Conclusion
Both of the two Bayesian methods are the feasible way for the evaluation of diagnostic test in the absence of a gold standard diagnostic. Prior source, priority number, and conditional dependencies should be considered in the method selection, the accuracy of posterior estimation mainly depending on the prior distribution.
Data Availability
The data used to support the findings of this study are included within the supplementary information file (Table S1).
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Authors’ Contributions
Taishun Li analyzed and interpreted the patient data and was a major contributor in writing the manuscript. Pei Liu revised the manuscript and provided methodological guidance. All authors read and approved the final manuscript.
Acknowledgments
This work was supported by the Fundamental Research Funds for the Central Universities and Postgraduate Research & Practice Innovation Program of Jiangsu Province (Fund number: KYCX17_0186). We would like to thank Jiaying Yang for her help in the modification of the language of the article.
Supplementary Materials
Supplementary file. Text S1: two scenarios of Conditional Covariance Bayesian model.
Supplementary file. Text S2: two scenarios of the Bayesian probabilistic constraint model.
Supplementary file. Text S3: OpenBugs code.
Supplementary file. Table S1: origin dataset.