Abstract
Objectives. We investigated the cognition and application of the “Evaluation System of Traditional Chinese Medicine Clinical Evidence” among populations with disparate backgrounds. Methods. We performed an online survey using a self-designed questionnaire. Results. Of 307 returned questionnaires, 284 were noted to be valid, and the effective recovery rate was 92.5%. Our analyses showed that the respondents demonstrated a better understanding of clinical evidence-based evaluation systems and that they used these occasionally. For both the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) and the traditional Chinese medicine (TCM) evidence evaluation systems, the respondents generally showed poor overall understanding, rendering the systems impractical. Among the respondents who were exposed to the existing evidence evaluation system for TCM. More than 70% of the respondents remarked that it was difficult to obtain high-quality evidence using any existing methods to evaluate TCM clinical evidence, that there was a lack of clear evaluation criteria, and that it was difficult to grasp the evaluation process. Conclusions. The evaluation systems of TCM clinical evidence have gained a certain degree of recognition among practitioners, who show a great willingness to use it, but practical applications are limited. In addition, it is also expected that an evaluation system would be more in line with the clinical characteristics of TCM.
1. Introduction
Evidence provides the all-important basis for evidence-based decision-making, and it needs to be evaluated appropriately to create a reliable foundation for further study [1]. High-quality evaluation of clinical evidence with respect to traditional Chinese medicine (TCM) requires scientific evaluation criteria with TCM characteristics. Many domestic scholars have explored classification and recommendation systems with respect to TCM evidence over the past 20 years. Based on the characteristics of TCM clinical theory and practice, such as treatment according to syndrome differentiation and inheritance of ancient medical literature and famous experts’ experience, there are two developmental directions: the first is to adapt to the existing international evaluation standards to include the evidence types with TCM characteristics for evaluation and the second is to establish independent evaluation standards based on TCM theories [2]. At the same time, with the rapid development of evidence-based medicine in China and the wide application of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) worldwide, many researchers have investigated the application of this system to evaluate the evidence regarding TCM [3]. With respect to the field of acupuncture, several research teams have combined the GRADE system with historical documents of TCM and well-known expert experience to establish an evidence evaluation system for clinical research on acupuncture. To better understand and apply the TCM clinical evidence evaluation system to the healthcare industry and to better promote the development of TCM clinical evidence-based evaluation, we organized and implemented the current survey questionnaire.
2. Methods
2.1. Survey Methodology
The questionnaire was developed by the project team of Shanghai TCM Evidence-based Medicine Research Center. Its content includes basic information on the respondents and cognition and application of the evidence evaluation system (including general information, the GRADE system, and the characteristic evidence evaluation system of TCM), as well as the current situation of clinical evidence evaluation of TCM, making for a total of 25 questions. Before the questionnaire was issued, we consulted with experts and conducted a presurvey to optimize the survey questions and to evaluate these for scientific approach and reasonableness. Based on the presurvey results, the Raosoft sample size calculator (http://www.raosoft.com/samplesize.html) was used to ensure the appropriate number of participants, given the selected margin of error (5%), confidence interval (90%), and approximate population size (20,000 individuals). In addition, the minimum sample size of this survey was 267. We used a 5-point Likert scale to investigate the cognition and usage of evidence evaluation systems and assigned points when data were collated: 1 for very well known/very frequently used, 2 for relatively well known/frequently used, 3 for moderately/occasionally used, 4 for not well known/rarely used, and 5 for not at all known/never used.
The questionnaire was designed using the questionnaire website (https://www.wjx.cn), conducted in the form of a network survey, and distributed through WeChat and the questionnaire website. The survey was open to all people through social networks, regardless of age, sex, year of experience, or educational background, and respondents completed questionnaires according to their interest in this subject. The survey was filled out anonymously and completed independently by the respondents.
2.2. Data Analysis
The questionnaire was collected from the questionnaire website, and the data were downloaded in the form of an Office Excel spreadsheet. Office Excel software was then used to sort the data and eliminate any invalid questionnaire data, and the sorted data were imported into SPSS 24.0 software for statistical analysis. Questionnaires with missing data and inconsistent answers were considered invalid. The effective recovery rate was the ratio of effective questionnaires to total recovery questionnaires [4]. The Cronbach alpha reliability coefficient method was used to test the questionnaire’s reliability, and we utilized the factor analysis method for the construction-validity test, with differences with < 0.05 considered statistically significant.
3. Results
The questionnaire survey was conducted between June 15 and July 15 of 2020. A total of 307 questionnaires were collected, of which 284 were considered valid, constituting an effective recovery rate of 92.5%. The calculated Cronbach’s alpha reliability coefficient was 0.95, the KMO value for the validity test was 0.905, and the Bartlett spherical test resulted in < 0.01. The questionnaire showed high construct validity and scale reliability.
3.1. Characteristics of the Participants
In terms of occupation, the respondents were mainly clinicians (49.6%), followed by scientific researchers (15.1%) and nurses. Organizations were mainly higher medical institutions, accounting for 54.9%, followed by colleges and universities (26.1%). Chinese medicine (57.7%) and integrated medicine (29.2%) dominated the professional fields. Of the respondents, 74.6% had acquired a master’s degree or above, and 24.6% possessed a bachelor’s degree. In terms of professional titles, the percentage of senior titles was the highest (36.3%) (see Table 1 for details). Statistically, there were significant differences among the occupations, educational backgrounds, organizations, and professional fields of the respondents ( < 0.01), which may be related to the distribution of questionnaires and respondent professions.
3.2. Cognition and Application of Evidence Evaluation System
Respondents showed a better understanding of the “clinical evidence evaluation system”; however, they only used it infrequently or occasionally. For the GRADE and TCM evidence evaluation systems, cognition ranged from moderate to a low level of understanding, and the use status was at the no-use level (Figure 1). The difference was statistically significant compared with the three points of normal/occasional use ( < 0.05).
Further analysis of respondents’ occupations, education, titles, and professional fields revealed significant differences in cognition and usage ( < 0.05). In terms of cognition and use of the clinical evidence system, GRADE system, and TCM evidence system, researchers showed a better understanding and more frequent use. There were also significant differences in cognition and usage across educational levels ( < 0.05); i.e., the respondents with doctorate and master’s degrees were more familiar with the evidence evaluation system and more typically used the evaluation system than individuals with a bachelor’s degree. However, those with intermediate and advanced titles were inclined to know more and use TCM clinical evidence evaluation more often, and there was no significant difference in the cognition and application of the GRADE system. In terms of professional field, respondents in the fields of Western medicine and integrated medicine used the “clinical evidence evaluation system” more often, and their cognition and use score of the GRADE system were higher than those of respondents in the field of TCM. In the field of integrated medicine, the TCM clinical evidence system was applied more often.
In terms of the application of evidence evaluation in certain situations, more than 70% of the respondents indicated that they would apply the existing evidence evaluation system (such as the Oxford system, GRADE system, or other evaluation criteria) when encountering clinical problems, conducting research design, and writing professional/scientific manuscripts.
3.3. Evaluation of TCM Clinical Evidence
Of the 284 valid questionnaires, 54 respondents had no knowledge of the “TCM clinical evidence evaluation/TCM clinical evidence evaluation system.” Of the 230 individuals who exhibited knowledge regarding the evaluation of clinical evidence for TCM, over half of the respondents knew the “Composition of traditional medical evidence body and classification of evidence” proposed by Liu [5] or Wang et al. [6] “Evidence-based Chinese medicine clinical practice guidance evidence grading system.” Some interviewees had also read the “TCM clinical evidence grading and scoring system” proposed by Wang et al. [7], the “TCM clinical evidence evaluation method” proposed by Zhong [8], the “Acupuncture clinical research evidence evaluation system” proposed by Ren et al. [9], or other self-created evaluation systems (see Table 2).
Of all respondents, 267 indicated that they had used existing methods to evaluate the clinical evidence for TCM. Over one-third (34.5%) of the respondents had used the “Five-level system of TCM version (Jianping version [4]),” and 28.5% had used the GRADE system to evaluate the TCM evidence. Some respondents indicated that they had evaluated the TCM evidence using the “Oxford system” (16.5%), the self-created evaluation method (13.5%), or other methods (28.1%) (Table 3).
Over 70% of the respondents indicated that when using any existing method to evaluate the clinical evidence underlying TCM, they encountered difficulties in obtaining high-quality evidence, lacked clear evaluation criteria, or faced an unclear evaluation process (Table 4). There were also many other problems, including too many evaluation systems from which to choose, large disparities in the levels of the same evidence in different systems, opaque evaluation processes, and a lack of involvement in the formation of recommendations (Table 5).
3.4. Application in the Evaluation of TCM Evidence Using the GRADE System
Of all the respondents, 218 (76.8%) had heard of the GRADE system, while 66 (23.2%) had no knowledge of it at all. Among the 218, 110 (39.1%) had used the GRADE system to evaluate TCM evidence or had read articles published by others using the GRADE system. More than half (56.7%) of all respondents believed that the GRADE system could be used to evaluate TCM clinical evidence but that an appropriate modification was needed first to reflect the clinical characteristics of TCM. Only 3.9% of the respondents believed that the GRADE system was fully applicable for use without any modifications, while others thought that it was not applicable at all, that an independent evaluation system of TCM was needed, or that because it was not applied, respondents did not make a statement.
When we surveyed the opinions of the respondents who had used or read that the GRADE system being used by others to evaluate TCM evidence (Table 6), the practitioners indicated the following: “generally lower level of evidence limited the usage of GRADE” (88.2%); “the characteristics of treating the same disease with different treatments in TCM lead to higher risk of consistency bias” (74.5%); “higher-priority evidence, such as ancient TCM case reports, rated lower in GRADE” (51.8%); and “it can be well applied and improve the decision-making level” (38.1%). A small number of respondents believed that the results of other evaluation systems were better, or they did not take a position.
Most of the respondents thought that the level of characteristic TCM evidence such as classical TCM or expert experience should be improved (60.5%), and 16 of them thought that it should be decided according to the specific evidence.
3.5. Suggestions for the Evaluation of Clinical TCM Evidence
To promote research and application into a clinical evidence evaluation system for TCM, an open question was designed at the end of the questionnaire to collect the respondents’ suggestions on an evaluation system of clinical evidence regarding TCM. We ultimately collected 31 suggestions. After classifying similar opinions, we determined the order of frequency (Table 7).
4. Discussion
With the development of modern Chinese medicine, clinical studies have been widely conducted, accumulating a large amount of clinical evidence for Chinese medicine. TCM clinical evidence and its evaluation methods are the foundation of TCM clinical practice guide formulation, TCM diagnosis, and treatment plan formation, evidence management, and other associated efforts. Therefore, research on the evaluation of TCM clinical evidence plays a vital role in promoting its production and utilization.
Through an electronic questionnaire, the present study investigated the cognition and application status of TCM clinical evidence evaluation systems in the medical industry. The results showed that the respondents possessed a better understanding of clinical evidence evaluation systems and used them occasionally. However, with respect to the TCM evidence and evaluation system, they possessed only a moderate understanding, with the use aspect rated at the “infrequently” level. More than half of the respondents who had been exposed to TCM evidence evaluation knew and used Liu Jianping’s or Wang Shouchuan’s versions of the “Five-level system of TCM improved version,” which may relate to the development of evidence-based medicine in China and the formulation and promotion of TCM clinical practice guidelines in recent years. In one study [10], the authors reviewed the evidence-based clinical practice guidelines of TCM published within the past five years. Of the 24 guidelines, 11 applied the “Five-level system of TCM improved version” by Wang Shouchuan, and 11 applied the Jianping version; the majority of medical practitioners learned and used the guidelines in clinical practice [11]. The results of the survey also showed that most practitioners used the existing evidence evaluation system for clinical problems, research design, and manuscript writing, indicating that it was necessary to improve and popularize the evaluation method of TCM clinical evidence.
When they applied any existing methods to evaluate the clinical evidence for TCM, most respondents said that they encountered difficulties in obtaining high-quality evidence and that clear evaluation criteria were lacking. The most common complaint of the respondents was that the evaluation standards of various evaluation systems were inconsistent, and the grades for the same evidence in disparate evaluation systems were quite different. Research on the evaluation of TCM clinical evidence is currently primarily performed by medical colleges and evidence-based medical centers at all levels in China. The lack of recognized evaluation standards, especially the evaluation standards of ancient and modern literature evidence of TCM, leads to the same type of evidence, such as the experience of ancient doctors, and is simply evaluated as the highest or lowest level of evidence and thus may create problematic issues regarding the formation of recommendations and even clinical decision-making. In addition, the lack of dissemination also affects the application of evidence evaluation systems [12].
The proportion of respondents who used the GRADE system to evaluate TCM evidence was not large: over half of the respondents believed that the GRADE system was suitable for TCM clinical evidence evaluation but that appropriate modifications needed to be made to reflect the clinical characteristics of TCM (such as improving the level of TCM evidence). Some investigators [13] believe that the introduction of the GRADE system is significant in constructing the evidence evaluation system for TCM. In addition, they recommend that we should pay attention to TCM syndrome differentiation and treatment, the improvement of TCM clinical research quality, and the evaluation of historical TCM literature and expert experience. The GRADE system has the characteristics of strong practicality, exhibiting a transparent evaluation process and ease of operation, and being used widely internationally. Combined with the suggestions of respondents, the evaluation of TCM clinical evidence may ultimately be unified based upon the GRADE system.
The survey was mainly designed for individuals in the medical and related industries who were interested in this topic. Although voluntary response bias cannot be ruled out in these voluntary and anonymous questionnaires, the results do not reflect a large number of strong opinions. While the sample size outnumbered the calculated minimal value, it may still be limited, which might engender some biases in the results. However, in the absence of relevant data in this field, we believe that the conclusion based on 284 valid samples is meaningful. In addition, due to the strong professionalism of the questionnaire, the distribution of the respondents was not uniform, and the investigated population was relatively concentrated; thus, this questionnaire had some limitations in sampling representativeness. The use of the limited media of questionnaires, WeChat, and web links, and the accessibility of the Internet may also have influenced the results.
5. Conclusions
In conclusion, practitioners exhibited a certain degree of awareness regarding the evaluation of clinical evidence for TCM. Although they showed a substantial willingness to use it, its practical application was not acceptable in its current situation. Practitioners expected to possess an evidence-based evaluation system that would be more congruent with the clinical characteristics of TCM. The principal obstacles to its application appear related to its own development, including the numerous evaluation systems, variable standards, an opaque process, a lack of training and promotion, and other external factors. It may be an effective method to establish a unified evaluation system on the basis of the GRADE system that is widely used, but with additional in-depth research, improvement, and application according to the characteristics of clinical practice and the current standards of TCM. Standardized evidence-based evaluation of TCM would provide strong support for the development of clinical practice guidelines for TCM and for evidence management and information services. Scientific classification and recommendation systems of clinical evidence for TCM would promote the development of evidence-based Chinese medicine and internationalization of TCM.
Data Availability
The data used to support the findings of this study are available from the corresponding author on request.
Conflicts of Interest
The authors have no conflicts of interest to declare.
Acknowledgments
This work was supported by the National Key R&D Program of China (Grant no. 2019YFC1709803) and National Natural Science Foundation of China (Grant no. 81873183). The authors would like to thank Prof. Tiansong Zhang (Fudan University), Prof. Peiyong Zheng (Longhua Hospital, Shanghai University of Traditional Chinese Medicine), and Prof. Zhaoqin Fang, Prof. Xueyong Shen, and Prof. Shuijin Shao (Shanghai University of Traditional Chinese Medicine) for the excellent and helpful suggestions in the questionnaire design and presurvey stages.
Supplementary Materials
Questionnaire on clinical evidence evaluation of traditional Chinese medicine. (Supplementary Materials)