Abstract
This paper outlines how to compute community priority indices in the context of multicriteria decision making in community settings. A simple R function was developed and validated with community needs assessment data. Particularly, the first part of this paper briefly overviews the existing methods for priority setting and reviews the utility of a multicriteria decision-making approach for community-based prioritization. The second part illustrates how community priority indices can be calculated using the freely available R program to handle community data by showing the computational and mathematical steps of CPI (Community Priority Index) with bootstrapped 95% confidence intervals.
1. Introduction
Providing public health practitioners and community development advocates with reliable measures for priority setting is a necessary step to foster accountability of the decision-making process in community settings. Community engagement is considered to be the pivotal element for a successful community-based organization [1, 2], particularly during the implementation of needs assessment projects and the selection of priorities for community action. By involving all relevant community stakeholders in the development of community action plans, community-based organizations not only ensure an equitable decision-making process but also enhance the cultural acceptability of interventions [3–5]. Although priority setting is an essential decision-making step for community-based organizations and participatory action research, there is little guidance on how to approach priority setting with quantifiable indicators while adopting community engagement principles [6].
Techniques for prioritization in community settings range from simple to more complex consensus building techniques, such as straight voting, weighted voting, nominal group technique, consensus panels, focus groups, Delphi technique, and others. The simplest form of community prioritization often occurs in town hall meetings or board meetings using simple voting, which typically implies giving each stakeholder the opportunity to vote on a list of issues. A variation of simple voting is the assignment of a certain number of votes to each stakeholder (e.g., 3 votes) and then sorting ideas or ranking to select the top one. Although democratic, this form of prioritization can only be used when the number of choices is small, and it is sensitive to issues of representativeness and generalizability. However, such a decision-making approach becomes increasingly cumbersome and impractical as the number of priorities increases. Another issue with straight voting is that it takes up only the majority of opinions and may inadvertently alienate a minority group, which can result in detrimental consequences for the community partnership and the engagement process. Some community advocates also use weighted voting, in which stakeholders assign different points (e.g., 1 = low importance, 2 = medium importance, and 3 = high importance) to a list of community issues in order to rank the items posteriorly. Although such a process tends to be more equitable, this method assumes that decision-makers are capable of mentally assigning reliable weights of diverse issues. This assumption is impractical because unguided stakeholders will reflect their personal preference and there is no guarantee that they will use uniform and consistent defensible criteria for prioritization every time they vote.
Because of some limitations of simple weighted methods, community development scholars recommend that voting methods are complemented with group discussions toward building a consensus to capture the community perspective rather than personal references. The two most frequently used consensus building methods are the nominal group technique (NGT) [7, 8] and Delphi technique [9–15]. By combining ranking procedures and participatory discussions, these techniques can be very effective in gathering consensus across diverse groups of stakeholders in a democratic and unbiased manner [10–12, 14–16]. The nominal group technique uses a one rank-ordered feedback, followed by a discussion that results in community consensus. However, a group of stakeholders that surpasses ten or twelve members cannot be easily managed and consensus may not be achieved. In contrast, the Delphi technique can encompass larger numbers of stakeholders and include several iterations of ranking and reranking (typically three or four) and consensus discussions. Because of its iterative nature and capacity to incorporate larger numbers of stakeholders, the Delphi technique is more robust than other methods. However, it can result in a lengthy process of several weeks or months. Because its implementation often requires the technical skills of highly skilled facilitators, the Delphi technique may be difficult to be implemented in community settings [9–11, 14, 15].
Community practitioners may also utilize qualitative techniques such as focus groups or key informant interviews. Qualitative methods provide credible and transferrable contextual data but in order to obtain generalizability, we need to implement mixed methods approaches. The value of qualitative techniques lies in gathering culturally relevant and richly experiential data, whereas quantitative techniques can complement qualitative findings (e.g., focus group themes) to generate measureable indicators that are comparable across settings (priority scores), different populations (mothers, children, and different geographical areas), and over different periods of time (longitudinal/repeated measures assessment).
None of the techniques mentioned previously explicitly differentiate between criteria of importance and changeability for the issues under consideration. For instance, stakeholders may decide to address highly important topics that are very difficult to change, which will result in projects that are ineffective and with discouraging results for the community. Conversely, stakeholders may decide to prioritize those issues that are highly changeable, but those issues may be of relatively low importance. The later situation would result in inefficient use of the scarce resources. We consider that community-based organizations must aim to address highly changeable and highly important issues. Since there is little guidance for community-based organizations on how to integrate these two criteria, we felt the need to develop a combined measure that indicates priority based on both importance and changeability.
2. The Need for the Development of the Community Priority Index
There is a clear need for quantifiable indicators for priority setting that integrate importance and changeability and permit cross settings comparisons but at the same time permit the wide participation of community stakeholders. Therefore, we developed the Community Priority Index using the following stepwise approach.
2.1. Adoption of Multiple Decision-Making Criteria
The adoption of a multicriteria decision-making approach in community-based priority setting has been widely recommended [17–19]. We recommend that decision-makers at least adopt a minimum of two criteria: importance and changeability. We selected importance and changeability because these are commonly used decision criteria in community-based program planning, but their utilization remains based on judgment and it is difficult to replicate due to the lack of quantifiable and comparable measures [20, 21]. Importance pertains to how relevant the issue was to the community context, which could be based on the magnitude of a particular problem (e.g., how prevalent, how much healthcare cost burden, or contribution to life expectancy or quality of life, or how relevant the problem is for the community under discussion). It is important to note that decision-makers can adopt separate importance criteria, such as importance based on cost, importance based on number of people dying from associated diseases, and importance based on impact on community quality of life. To simplify the present analysis, we use only one overall importance criterion. The second criterion we recommend is changeability, which refers to how easily the issue could be changed in the community if a designated intervention would be made available within the scope of a particular community-based organization.
2.2. Importance and Changeability Ratings for Each Decision-Making Criterion
Separate stakeholders’ rating for the criteria of importance and changeability for each community issue must be identified, using weighted numerical scores (from 1 = low to 3 = high). We used a 3-point Likert-type scale, as follows: for importance: 0 = not important, 2 = intermediate importance, and 3 = very important; for changeability: 0 = not changeable, 2 = intermediate changeability, and 3 = highly changeable.
The mathematical computation consists of the following steps. Let be a number of stakeholders (interviewers or decision-makers); each interviewer will prioritize the questions (issue) using criteria for each question. Let be a -Likert scale representing the score of the th criteria of the th question of the th interviewer; thus, for all , , .
2.3. Computation of Item Average Scores by Importance and by Changeability
An important caveat is that, in community settings, it is typical to get an unequal number of participants’ responses per item. This situation occurs because some items are responded by all members, while a few are responded by only a subset of the members (e.g., some stakeholders may leave blank spaces for abstaining from voting or just missingness at random). The use of simple sum of item scores is inappropriate in such a situation. Thus, we used the arithmetic mean or average and computed mean importance scores and mean changeability scores. Accordingly, each sum of item scores was divided by the number of respondents for the particular item, which resulted in the item mean importance as well as item mean changeability. Forced responses are not recommended, since it may be perceived as coercion and a threat to the democratic process.
The mean of the th criteria of the th question, , is calculated as follows:
2.4. Multiplication of Mean Importance and Mean Changeability Item Scores to Generate a Summary Statistic That We Refer to Here as Community Priority Index (CPI)
We used the following formula: . A single summary index was computed for each item or issue which integrated both perceived importance and perceived changeability, with higher values indicating higher priority.
The CPI is the product of the mean of the th criteria of the th question, calculated as follows:
2.5. Stratification by Target Population
If two or more subpopulations are targeted, then we recommend the stratification of CPI scores by types of population to identify priorities for action. This is a final step in which the issues are organized by type of population and ordered in descending fashion to identify the top highly important and highly changeable issues by target population. In this manner, community stakeholders will be able to better determine the scope of community-based strategies and to allocate project resources more effectively. Notably, the process is systematic and democratic, from beginning to the final selection of top priorities and by diverse populations separately (e.g., women, children, men, and the elderly).
2.6. Construction of 95% Confidence Intervals and Bootstrapping
This step of CPI pertains to the evaluation of the precision of asymptotic approximations in small samples, which is an important step if the number of stakeholders is relatively small (e.g., less than 30). For this purpose, we constructed 95% confidence intervals.
The lower bound (LB) of can be calculated by assuming for all , , . Thus, LB of CPI is . Similarly, the upper bound (UB) of can be calculated by assuming for all , , . Thus, UB of CPI is . That is, the range of is .
It is important to highlight that the traditional confidence interval estimator that is based on the normal assumption of the sampling distribution cannot be used with small samples [22]. To overcome this limitation, we complemented the classic analysis with bootstrap methods to construct 95% confidence intervals [23]. Bootstrapping samples were created by ten thousand samples with replacement from the original dataset. The 2.5th percentile and 97.5th percentile are represented as the 95% CI of CPI. In bootstrapping, data collected for a single experiment is used to simulate what the results would have been if the experiment was repeated over and over with new samples (e.g., sampling with replacement from the original dataset). Specifically, we used bootstrap samples to estimate the mean score of 3-point Likert-type scaled items and their 95% confidence interval. By using bootstrap methods the distribution of the data normalizes permitting the use of the mean as a reference cut point [23]. Therefore, we generated via computer program (S+ 8.2) 5000 bootstrap samples of community stakeholders ratings [24]. The following algorithm was used to generate the bootstrap samples.(1)We constructed an empirical distribution function, , from the observed data. places probability on each observed data point .(2)We then drew a bootstrap sample of size 6 with replacement from . The mean of this bootstrap sample was calculated achieving a normally distributed population.(3)Step 2 was repeated 10,000 times. The percentile method was used to compute a 95% confidence interval around the mean by ranking the bootstrap sample means and then selecting the 2.5 percentile as the lower confidence limit and the 97.5 percentile as the upper confidence limit. In other words, the lower bound value is the least CPI score possible within the 95% confidence interval, while the upper bound value is the highest possible CPI score within the interval. In this regard, the mean value of CPI scores for each issue represents the group consensus, whereas the width of 95% confidence intervals indicates the range of agreement.
2.7. Standardization of CPI Scores
Up to this point, CPI results are still scale-dependent and lack comparability potential with other community settings if different Likert-type scales are used (e.g., 3-point scale versus 5-point scale or 7-point scale). Comparability becomes particularly important for nation-wide programs or coalitions that have local or county chapters. Thus, we standardized each CPI indicator to have a range from 0 to 1 by applying the following conceptual formula:
Accordingly, the mathematical formula standardizes each CPI indicator to have a range from 0 to 1 by applying the following formula:
Given the above formula, the CPI can only range from 0 to 1 and it is scale-free, which now permits comparisons across different studies and populations. The entire computational process of the CPI can be summarized in Figure 1.
3. R Code for Computation Process of CPI
In order to make the benefits of the CPI freely available to community practitioners, we developed a statistical program using the software R, which is free and widely available for download through the worldwide web. For any R users, we provide the R code for CPI computation.
3.1. Data Preparation
(1)Its extension is .csv.(2)The 1st column is ID of interviewee.(3)The 2nd column is question number.(4)The 3rd is -Likert scale for the first criteria (importance).(5)The 4th is -Likert scale for the second criteria (changeability).
3.2. R Functions
3.2.1. LikertCheck Function
This function will return the data replacing all inappropriate scores that are outside the appropriate range () with NA (see Algorithm 1).
| ||||||||||||||||||||||||
3.2.2. CPICal Function
This function will return the CPI of data (see Algorithm 2).
| ||||||||||||||
3.2.3. CI95.CPI Function
This function will resample the data with replacement and calculates its CPI for 10,000 times (default). It will return the 2.5th and 97.5th percentile of 10,000 CPI representing the 95% confidence interval of CPI (see Algorithm 3).
| ||||||||||||||||||||||||||||||||||||||||||||
3.2.4. CPIReport Function
This function will calculate the CPI, its 95% CI, and their standardized form (see Algorithm 4).
| ||||||||||||||||
Table 1 depicts a hypothetical set of scenarios where CPI was calculated for three populations that had unbalanced lists of issues, which is likely a situation that will arise in diverse community contexts. For this exercise, we derived CPI scores from a community needs assessment data conducted as part of previous R24 grant (i.e., purpose was to identify the top health priorities for community action to eliminate disparities in maternal and child health populations). However, we omitted the names and specific populations to permit a standard presentation of the CPI computation. The following cut-off points for prioritization were adopted for CPI scores, based on combined importance and changeability: values <0.3 were considered low priority, 0.3–0.7 intermediate priority, and >0.7 high priority. We recognize that these cut-off points are arbitrary, and practitioners may need to exercise more strict and narrow ranges according to their needs.
It can be appreciated in Table 1 that, for hypothetical population A, there were four standard CPI scores with values higher than 0.70, which indicates those issues were perceived as highly important and highly changeable priorities. However, only the top issue had a lower bound 95% higher than 0.70 (CPI for = 0.85; 95% CI: 0.70, 1.00), which indicated a stronger consensus among decision-makers. In contrast, for issues and , the standard CPI scores were higher than 0.70, but the respective 95% CI was wider (opinions vary more). Therefore, decision-makers can clearly select one top issue for population and be also confident that there is a strong consensus that this issue is highly important and changeable.
For comparison purpose, we included subpopulations and , which had their own list of issues that were ranked and CPI scores computed. The main reason for adding populations is to illustrate how CPI could help prioritize populations too by carefully analyzing the CPI scores. In Table 1, it can be appreciated for population that three issues presented standard CPI scores higher than 0.70. However, when looking at their 95% CI, all three issues had lower bounds with less than 0.70. This may indicate that decision-makers were less consistent regarding their assessment of importance and changeability or that there is less agreement in the selection as top issue. Since the CPI scores are standardized, practitioners can compare across populations and evaluate the consistency of the evidence across populations. In this particular exercise, decision-makers had stronger consensus regarding the importance and changeability of issue than for (CPI = 0.87; 95% CI = 0.69, 1.00) or issues (CPI = 0.88; 95% CI = 0.69, 1.00). Therefore, if practitioners need to only prioritize one issue due to scarce resources, they can confidently select as their top more efficient way to allocate resources. However, depending on availability of resources and programmatic focus, they can also justify additional issues that were also important and changeable, but for which consensus was weaker.
4. Conclusion
This study examined the development of a quantifiable indicator for priority setting that we have referred to as the Community Priority Index (CPI). We utilized two criteria (importance and changeability), as well as stratified by subpopulations. We consider that our CPI is a new measure that fosters the accountability of decision making, while flexibly allowing for the application of diverse participatory methodologies. For example, the list of issues that must be prioritized can be generated through nominal groups, focus groups, community surveys, or even expert opinions. However, if the CPI is used in conjunction with democratic participation and adequate community engagement processes, the decision of prioritizing will explicitly incorporate aspects of relevance and changeability. Thus, rather than replacing other methods, we recommend the CPI as an additional tool that can be incorporated to community-based efforts for priority setting, if acceptable and relevant for the community and project context.
Our R code can be easily modified to suit the needs of different projects and help decision-makers in the application of multiple decision-making criteria. Notably, since the R package is free of cost and available for multiple operating systems, community-based organizations from all over the world can use the CPI to foster accountability in the selection of priorities for action. We recommend that practitioners use the 95% CIs for the final assessment of top priorities and weight their decision against willingness to pay and budget allocations, as well as ethical factors, social justice, and cultural competence of interventions. In particular, the bootstrapped 95% CI provides an opportunity to assess the precision of the CPI scores with even small samples of decision-makers. While the mean CPI score for a particular issue (standardized CPI) indicates the level of group consensus on importance and changeability, the width of the 95% CI indicates the scope of agreement or group consensus. Wider 95% CI indicates that group opinions were more diverse. Diversity of opinion among group leaders may directly indicate more controversy in the community since their opinion represents the community perspective to a certain extent. Using our recommended cut-off values, practitioners can choose priorities that were highly important and highly changeable, as well as selecting those for which there is greater agreement. CPI is presented here as a tool in tandem with advancing the science of mixed methods analysis by incorporating both qualitative and quantitative data in the way to enhance precision in the assessment of group consensus, as well as enhanced interpretation of the multiple decision-making criteria. In this context, researchers wishing to use the CPI must acknowledge the ecological and political nature of community-based participatory research by assessing individual opinions through adequate participatory methods. Indeed, the partnership synergy is an extremely important aspect of community-based organizations and participatory action research that describes how early disagreements and conflicts can produce more productive stakeholder interactions in the future. Thus, the CPI is not a replacement of stakeholders’ consensus building discussions. Instead, the CPI is a way to implement multicriteria decision-making approach and to foster accountability of decisions.
Disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Minority Health and Health Disparities, or the Baylor College of Medicine.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by funding from the National Institute on Minority Health and Health Disparities through a R24 grant on “Community-Based Participatory Research” (Award no. 5R24MD8056-02).