Abstract

Introduction. With Chinese health care reform increasingly emphasizing the importance of primary care, the need for a tool to evaluate primary care performance and service delivery is clear. This study presents a methodology for a rapid assessment of primary care organizations and service delivery in China. Methods. The study translated and adapted the Primary Care Assessment Tool-Adult Edition (PCAT-AE) into a Chinese version to measure core dimensions of primary care, namely, first contact, continuity, comprehensiveness, and coordination. A cross-sectional survey was conducted to assess the validity and reliability of the Chinese Rapid Primary Care Assessment Tool (CR-PCAT). Eight community health centers in Guangdong province have been selected to participate in the survey. Results. A total of 1465 effective samples were included for data analysis. Eight items were eliminated following principal component analysis and reliability testing. The principal component analysis extracted five multiple-item scales (first contact utilization, first contact accessibility, ongoing care, comprehensiveness, and coordination). The tests of scaling assumptions were basically met. Conclusion. The standard psychometric evaluation indicates that the scales have achieved relatively good reliability and validity. The CR-PCAT provides a rapid and reliable measure of four core dimensions of primary care, which could be applied in various scenarios.

1. Introduction

Primary care is a fundamental part in the health care systems of both high and low income countries and there is ample evidence that primary care is closely related to the improvement of health outcomes [1, 2]. In 1994, the Institute of Medicine (IOM) proposed a function-orientated definition of primary care, which is consistent with the widely acknowledged multidimensional concept of primary care with its emphasis on the four core dimensions of primary care (i.e., first contact, continuity, comprehensiveness, and coordination) [35].

In urban China, primary care is defined as “the delivery of comprehensive, continuous, and convenient episodic and preventive health care services to families in the community” [6], which are mostly provided by general practitioners (GPs) in community health centers (CHCs) or other health stations or clinics [7]. Since 2009, the Chinese central government has launched a new health care reform, an essential part of which is to strengthen the primary care system [810]. Significant government investments have been made on developing GP workforce, constructing CHCs infrastructure, improving CHCs distribution network in order to improve the geographic accessibility, providing 12 basic public health services in CHCs, and so on [9, 11]. Meanwhile, the social health insurance have covered approximately 95% of the population and its reimbursement strategy has strengthened the use of primary care and the promotion of CHCs as a potential gatekeeper [1214].

As an economically developed district in southern China, Guangdong province has long recognized the importance of primary care in promoting health for the general public and has launched pilot programs for regional primary care reform since 1996 [15]. There are some favorable changes; for example, general practitioners (GPs) who are at the frontier in providing primary care have gradually acquired a good local reputation and attracted large numbers of patients. In the meantime, a notable characteristic of the province is that various management patterns of CHCs emerged after years of reform and the performance of CHCs varied under different management patterns (discussed later).

Whilst there has been considerable government investment and policy attention on primary care, evidence of the quality of primary care services is urgently needed [10, 16]. Previous researches in China have been carried out to assess the quality of primary care in disease-focused or task-orientated manners [1720]. Although these studies have been essential, they largely fell short on evidence of the core dimensions of primary care. A previous systematic review has demonstrated that the four core dimensions determine the primary care process [21]. Primary care is deemed stronger with better fulfillment of these dimensions [22]. Therefore, it is reasonable and necessary to develop a valid tool to measure the quality of primary care in a multidimensional manner and include its four core dimensions [23]. Several studies in developed countries have successfully developed valid tools in this way [2427] and such evaluation tools have been used in China before [7, 28]. Our preliminary research tried out such tools in practice. However, the results indicated that they were time-consuming and generated a significant amount of “not sure/don’t remember” responses. Therefore, constructing a rapid assessment tool seems imperative so that the Chinese primary care evaluation tool becomes concise and user-friendly in order to obtain key information with minimum time cost as well as to ensure patient compliance and quality of data.

This study was undertaken in close collaboration with health authorities of Guangdong province in response to the need for a rapid assessment tool to measure the quality of primary care service delivery. The objective of this paper was first to construct a rapid assessment tool that was based on the core dimensions of primary care and, secondly, to apply analytic methods to assess the validity and reliability of this tool. In addition, analysis was undertaken to compare primary care achievements across different health care settings.

2. Methods

2.1. Instrument

Primary Care Assessment Tool (PCAT) has been developed based on the theoretical model of primary care attributes established by John Hopkins [29] and Donabedian’s quality framework [30]. PCAT has been highly recommended over other measurement tools when assessing the process of primary care and is available in various formats [26, 3032]. Primary Care Assessment Tool-Adult Edition (PCAT-AE) is designed to measure adult patients’ experience of primary care. It has originally been used in the US [26] and has been adopted in several other countries with different health systems, including Brazil [33], Korea [34], Spain [35], and China [7, 28]. After certain adaptions, all these tools had been validated in their corresponding countries, and some of them have been used to measure quality of primary care [33, 36, 37].

In our study, the translation and adaption of PCAT-AE involved several steps. First, the questionnaire was translated into Chinese. Second, a group of four general practitioners and two experts of primary care policy reviewed the translated PCAT and some of the items that did not fit in the Chinese context were rephrased. During this stage, any item that did not adapt to the Chinese context, shared similar conceptual basis, or lacked conciseness was rewritten, combined, or deleted. Modifications were only made on condition that a consensus was reached; for example, “home safety, such as getting and checking smoke detectors and storing medicines safely” was eliminated, because it was considered to be inappropriate or unpractical in the current Chinese context; “When your PCP is closed and you get sick during the night, would someone from there see you that night” was deleted since most of the CHCs in China do not offer medical services at night. Then pilot tests, which focused on item wording, were conducted in three facilities. Twenty patients were randomly selected to complete the questionnaire by themselves in each of the pilot test and items were further revised into more understandable words to ensure that elderly or subjects of lower socioeconomic status can fully comprehend (specific item changes can be found in Table 9).

In this stage, the Chinese Rapid Primary Care Assessment Tool (CR-PCAT) consisted of 42 items. 23 items assessed the four core dimensions of primary care—first contact, continuity, comprehensiveness, and coordination—and three derivative dimensions—family centeredness, community orientation, and cultural competence. Two items were developed to identify individuals’ usual source of care and another 2 measured the frequency of visits to the general practitioners while the rest of the items mainly reflected patients’ social demographic information. The services received by patients were represented by a 4-point Likert scale (1 = never; 2 = sometimes; 3 = often; 4 = always). An additional option of Don’t know/Not sure was added in case of lack of knowledge of a certain item. The Don’t know/Not sure response and missing data were assigned a neutral value of 2.5 when conducting the analyses to be consistent with the methods used in other countries [34, 35].

2.2. Data Collection

A stratified, three-stage sampling approach was used to decide study sample. Three cities, namely, Guangzhou, Shenzhen, and Dongguan, were chosen, since they displayed significant socioeconomic variations and also CHCs were under different management pattern in these cities. In Guangzhou, 90% of the CHCs were owned and operated by government, and the remaining 10% were private-owned. The study chose one government-owned CHC and one private-owned CHC that were both granted as a state-level demonstration site, and another ordinary government-owned CHC. In Dongguan, all CHCs were government-owned and government-run; one state-level demonstration site along with one ordinary CHC was selected. In Shenzhen, all CHCs were affiliated with local hospitals. We selected two state-level demonstration sites and one ordinary CHC. Convenience sampling was then used to select patients in each of the CHC to participate in the survey.

The standard sample size formula for cross-sectional survey was used to calculate the sample size, where = standard normal variant corresponding to 5% type I error of 1.96; = expected proportion in population based on previous study; = absolute error or precision [38]. An estimated of the proportion was drawn from previously published research [7]. Typically, a conservative estimate of the proportion (i.e. 50/50) was made, if data was not sufficient. In addition, the sample size was adjusted for refusal rate of approximately 10%. Finally, a target sample size of 400 was set within each area. The CHCs in Guangzhou were oversampled because of additional planned analysis (not the focus of this paper). Furthermore, expert consultation was also used to ensure the precision and feasibility of the sample size, which confirmed the calculation above.

Data collection began in June 2014 and lasted for three months. Interviewers who were postgraduate students from Sun Yat-sen University were trained by two researchers in advance in order to assist the patients to complete the questionnaires. One-to-one interviews were conducted to guarantee the quality of survey data. Adult patients who were 18 years or older and could speak Mandarin or Cantonese were selected at the waiting area of each site. Subjects selected were exclusively to patients who were visiting general practitioners. Those who had visited the same general practitioners at least three times were eligible since they were considered to have a better understanding of primary care services provided by GPs. These patients were asked for permission to participate in the interview with a full explanation of the research purpose and were told that the survey would not influence their GP visits. Patients who took part in the survey were given a small gift as a token of appreciation upon completion of the questionnaire.

2.3. Data Analysis

Data analysis consisted of description of the sample population, validation of the CR-PCAT, and descriptive summary of the primary care achievement of each city. First, socio-demographic characteristics and health data of the sample population were summarized to demonstrate a diverse sample of populations and their utilization of primary care, meanwhile, the characteristics of each city and CHC were also presented.

Second, validation of the CR-PCAT was conducted, which was applied exclusively to the 23 items evaluating the seven dimensions of primary care [39]. For construct validity, factor analysis (principal component analysis and varimax rotation) was undertaken to test the dimensionality of the hypothesized scales. The item selection criteria were: eigenvalues >1.0, factor loading >0.35, and all retained items should not have a secondary loading over 0.35. Then internal consistency reliability of each scale was assessed by Cronbach’s coefficient alpha () and item-total correlation. Cronbach’s coefficient alpha over 0.50 was recommended as the benchmarks of adequacy of reliability coefficients [4042], while the minimum acceptable item-total correlation was 0.30 [41].

Next, five Likert scaling assumptions were tested, which included (1) item-convergent validity (a test by item-scale correlations); (2) item-discriminant validity (a test by the scaling success rate, i.e. items within a scale correlate more substantially with their hypothesized scale than with any other scale); (3) equal item variance (a test by examining item means, standard deviations, the equivalence of the intra-class correlation and Scotts homogeneity ratio for each scale); (4) equal item-scale correlation (a test by examining the range of item-scale correlations); and (5) score reliability (a test by Cronbach’s coefficient alpha) [43, 44]. The analysis was only applied to the retained items.

Then, we examined the following score distribution characteristics of the revised scales: mean, standard deviation, quartiles, skewness, range, and Kurtosis. Each of the five scales was also assessed for inter-factor correlations by computing a matrix of inter-factor correlations [45]. Finally, primary care achievement of different cities was compared using analysis of variance.

All analyses were performed using the statistical package STATA 13.

3. Results

3.1. Subjects

A total of 1831 eligible individuals were asked to participate in the survey, and 81.9% () of them completed the questionnaire. After eliminating samples with more than 5 missing responses (), 1465 effective samples were included for analysis.

Table 1 shows the socio-demographic features and health care utilization of the survey participants. The analytic sample included adults aged 18 to 89 years (mean = 46.1 years). They were predominantly female (60.7%), and were resident or temporary resident population (98.6%) with generally lower level of household income (only 10.9% over 10000 yuan/month). Over half of respondents were employed (54.6%) with relatively lower education (69.1% did not finish high school). Approximately one third (41.8%) indicated having chronic disease and 23.3% rated their health as fair or bad. Most participants (78.7%) were covered by health insurance and had been visiting their PCP for more than 1 year (82.3%). 73.3% reported over 3 times of visits to their PCP and over half of the respondents had visited a specialist in the past year (53.4%). Less than one tenth of all users had signed contracts with their PCP in spite of government’s vigorous promotion (8.9%) [46].

Table 2 displays the characteristics of the included cities and CHCs. Of the eight CHCs being studied, one CHC from Guangzhou was private-owned with the other 7 owned by the local government. Gatekeeping system was not implemented in Guangzhou, while in Dongguan and Shenzhen it has been implemented in practice. It is also worth noting that only the patients who were enrolled in a specific insurance plan were under the impact of the gatekeeping system (not the focus of this paper). Of the three CHCs in Shenzhen, they were all affiliated with the local public hospitals while CHCs in the other two cities were operated independently.

3.2. Construct Validity and Reliability

Twenty-three items were included in the initial principal component analysis. Based on the criteria that eigenvalue was larger than 1.0, five components that corresponded to the four core dimensions of primary care remained: first contact utilization, first contact accessibility, ongoing care, comprehensiveness, and coordination (Table 3). These extracted factors explained 58.91% of the total variance, with eigenvalues ranging from 2.08 to 1.39.

Eight of the 23 items were eliminated based on the criteria imposed for factor analyses (Table 4). No items were deleted for first contact utilization, ongoing care, and comprehensiveness. One item was eliminated for first contact accessibility and coordination dimension, respectively. Although first contact accessibility dimension contained only two items after elimination, it was retained since it represented unique conceptual meaning of primary care. All items were deleted for family centeredness, community orientation, and cultural competence.

Table 4 presents the item descriptive results, as well as results of reliability tests for both the original items and the final items. The distribution of the items varied significantly from a mean of 1.09 (When your PCP is closed, can you get advice quickly over the phone if you need it?) to 3.63 (When you have a new health problem, do you go to your PCP before going somewhere else?) on the 4-point Likert scale. The distribution tends to skew toward less favorable answers, with only five items falling above 2.5. The first contact utilization and ongoing care scales achieved the highest mean scores, whereas scales with lower means were first contact accessibility, comprehensiveness, and coordination.

Reliability tests include item-total correlation and Cronbach’s alpha coefficient reliability. Item-total correlations show that each item was strongly associated with its corresponding scale, with value ranging from 0.56 (When you have a new health problem, do you go to your PCP before going somewhere else? and When your PCP is closed, can you get advice quickly over the phone if you need it?) to 0.83 (When you go to your PCP’s, are you taken care of by the same doctor or nurse each time?). Cronbach’s alpha demonstrates internal consistency reliability that was higher than or equal to the original scales, despite dropping items from original scales.

3.3. Testing the Likert Scaling Assumptions

Table 5 demonstrates a summary of the results of the tests of Likert scaling assumptions using the five multi-item revised scales. All item-scale correlations exceeded 0.5 with the majority greater than 0.6 and all scales demonstrated a relatively narrow range of item-scale correlations (from 0.04 for first contact accessibility to 0.19 for first contact utilization). All 5 multi-item scales achieved 100% scaling success, indicating that all items in these scales got a greater correlation with items in their hypothesized scale than with items in other scales. Formal evidence of equal item variance was supported by the proximity of the intraclass correlation and Scotts homogeneity ratio for each scale. Two multi-item scales achieved Cronbach’s alpha level over 0.70, and two scales were below the threshold, but exceeded 0.50. One scale, first contact utilization, was far below the standard, verifying the low homogeneity of variances among items within the scale (0.2). However, the scale was retained because of their conceptual significance.

3.4. Descriptive Feature of CR-PCAT

Table 6 presents estimates of central tendency, dispersion, and other features of scale score distributions for the five primary care scales. The full range of possible scores was observed for all scales. Except for first contact utilization and ongoing care, the other three scales were positively skewed, indicating distributions with more negative ratings of primary care. The percentage of respondents scoring at the floor (the lowest score) or ceiling (the highest score) was acceptably low for all scales. The most significant floor effects existed in the first contact accessibility scale where over 80% of the participants reported minimum score.

Table 7 compares the internal reliability coefficient and interfactor correlation for each primary care scale. Cronbach’s alpha of each scale was substantially greater than its correlation with all other primary care scales. None of the interfactor correlations were excessively high, indicating the uniqueness of each primary care scale. All significant correlations were positive, implying that each scale was complementary to some extent. Relatively high and positive interfactor correlations were observed between first contact utilization and coordination (0.32), with the latter and comprehensiveness (0.27), and with comprehensiveness and ongoing care (0.29).

3.5. Primary Care Achievements in Three Cities

Table 8 illustrates that the overall scores of primary care achievements were relatively low in general, with Shenzhen achieving the highest total score (2.19), followed by Dongguan (2.10), and Guangzhou achieving the lowest score (2.09). For each scale, Guangzhou got the highest scores on two scales (ongoing care and comprehensiveness) and Shenzhen on first contact accessibility and coordination while Dongguan achieved the highest score on one scale (first contact utilization). Besides, most of the scores of each scale differed significantly among the three cities.

4. Discussion

The CR-PCAT introduced in this paper is an explicit and brief adaptation of the original PCAT-AE to the Chinese context. The final version of CR-PCAT comprises 15 items with five scales measuring four core dimensions of primary care. The core dimensions and items in corresponding dimensions display high consistency with the original PCAT-AE.

A standard psychometric evaluation method was used to validate the CR-PCAT. The analytic results indicate that the hypothesized scales have achieved relatively good reliability and validity. Principal component factor analysis supports the scale’s construct validity and the extracted five factors explained 58.91% of the total variance in the item scores. The panel of experts in primary care helped facilitate the best possible content validity [39]. Reliability tests indicated adequate item-total correlation and internal consistency of the tool. All of the five Likert scaling assumptions, including item-convergent validity, item-discriminant validity, equal item variance, equal item-scale correlation, and score reliability, were basically met, which suggests the appropriateness of the usage of Linker’s method.

The three derivative dimensions—family centeredness, community orientation, and cultural competence—were eliminated, because retaining all these three derivative dimensions did not increase the explained variations. Besides, the items within the three dimensions did not converge together as the other studies had displayed [26]. To balance between the amounts of information captured and time costs, only the four core dimensions with five scales were retained, which also met our goal of developing a compact measure tool. In addition, we also ran factor analysis using data from each of the three cities separately. The results confirmed the validity of the factor analysis done with combined dataset (results are not shown due to large amount of information but are available upon request).

Item descriptive results indicated that most of the items scores skew toward less favorable answers, which contrasted with the results of other studies [7, 34]. High rate of favorable response and “Don’t know/Not sure” answers was common among Asian populations, since they were not used to commenting or critiquing. However, the study adopted one-to-one interview to ensure that all participants fully understood each item and reported their actual perception of primary care.

4.1. Application of the Tool

CR-PCAT has multiple uses given its small number of items. Compared to the PCAT-AE and the other version of PCAT, which needs approximately 20–40 minutes to complete [26], the CR-PCAT questionnaire took about 5 to 7 minutes to administer. The tool greatly reduces the burden of patients and interviewers while insuring the key information was captured. CR-PCAT can be applied in periodic evaluation activities to assess the latest performance of primary care provided by CHCs. Also, since the tool is readily linked to specific implementations of primary care, it can be used to assess the effectiveness of policy changes on the delivery of care from both CHCs and district levels. On condition that primary care is provided by GPs in China, CR-PCAT can also serve as a part of the performance evaluation tools of GPs to provide evidence for further improvement.

In addition, from the primary care achievements of three cities, low scores of the overall primary care quality and each dimension suggest that primary care in China is far from perfect. The low scores of each dimension were consistent with our previous studies, which focused specifically on continuity or coordination of care [4749]. Besides, significant differences of scale scores between three cities were observed, implying that there might be system or organizational level factors that cause the differences [50]. For example, the score of first contact utilization in Dongguan was the highest among the three cities, which might imply the effects of gatekeeping system; and Shenzhen achieved the highest scores on the scale of coordination, indicating that patients experience better referral services when specialist care is needed, which may be due to the reason that CHCs in Shenzhen were affiliated to hospitals while the CHCs in the other two cities were independent of hospitals.

4.2. Limitations

Potential limitations of the study are discussed as follows. First, the CR-PCAT is a very short version of the original measure tool and the development and adjustment of the tool were straightforward. Only quantitative analyses were conducted in the study, which might imply that qualitative methods such as focus groups or in-depth interviews with patients might be necessary to fully explore their perception of primary care. Also, CR-PCAT does not measure disease specific quality of care, but rather general experience of primary care. Further research could combine CR-PCAT with disease specific quality indicators to have more extensive representation.

Secondly, the CR-PCAT was developed based on CHCs samples obtained in Guangdong province, where local people mainly speak Cantonese, suggesting that the linguistic habits were unique. If the tool were to be used in other provinces in China where there were distinct differences between the dialects, slight revisions of the wording were deemed necessary.

Thirdly, the measurement of primary care achievement was based on patients’ self-report. While this may be the best way to ascertain individual experience, it is subject to recall and response bias. Nevertheless, one-to-one interviews were undertaken in this study in an attempt to minimize the bias. In addition, criterion-related validity (or more specifically concurrent validity) and stability were not tested due to time and economic reasons; however, the tool will be applied in more regions and populations in the future, the results of which could add to the validation evidence of CR-PCAT.

Finally, as the CR-PCAT only measured adult patient’s perception of primary care, which was adapted from one part of the original methodology, our future research would center on developing rapid assessment tools for providers and children.

5. Conclusion

CR-PCAT appears to be a valid and effective tool to capture reliable performance of four core dimensions of primary care in China. Because of its simplicity and easy administration, it is a feasible and practical tool that can be used in CHCs’ daily administration, performance evaluation and monitoring, and policy assessments. The poor performance of primary care highlights the urgency of recognition and understanding of its core dimensions and the development of corresponding policies in the future to strengthen them in China. The next phase of the study will focus on identifying characteristics at the organization and health care system levels that account for the observed differences in primary care performance.

Abbreviations

PCAT:Primary Care Assessment Tool
PCAT-AE:Primary Care Assessment Tool-Adult Edition
CR-PCAT:Chinese Rapid Primary Care Assessment tool
IOM:Institute of Medicine
GP:General practitioner
CHC:Community health center.

Ethical Approval

The study protocol was reviewed and approved by the Ethics Review Committees of Sun Yat-sen University School of Public Health.

All participants gave verbal consent.

Conflict of Interests

None of the authors have any competing interests.

Authors’ Contribution

Jie Mei carried out the study, was involved in the acquisition and interpretation of data, and drafted the paper. Yuan Liang carried out the study and was involved in collection and interpretation of data. LeiYu Shi critically revised the paper and supported analysis and interpretation of data. JingGe Zhao and YuTan Wang were involved in collection and interpretation of data. Li Kuang conceived and designed the study and was involved in the collection and interpretation of data. All authors read and approved the final paper.

Acknowledgments

This study was supported by a grant from the Health Department of Guangdong province of China. The authors thank all the participating public health experts and health care staffs for their expertise and support during the translation and adaption and data collection process.