The Context, Process, and Outcome Evaluation Model for Organisational Health Interventions

Fridrich, Annemarie; Jenny, Gregor J.; Bauer, Georg F.

doi:https://doi.org/10.1155/2015/414832

BioMed Research International

On this page

Abstract Introduction Limitations Conclusions Acknowledgments References Copyright Related Articles

Special Issue

Psychosocial Factors and Workers Health and Safety

View this Special Issue

Research Article | Open Access

Volume 2015 | Article ID 414832 | https://doi.org/10.1155/2015/414832

The Context, Process, and Outcome Evaluation Model for Organisational Health Interventions

Annemarie Fridrich,¹Gregor J. Jenny,¹and Georg F. Bauer¹

Academic Editor: Sergio Iavicoli

Received18 Dec 2014

Revised09 Feb 2015

Accepted23 Mar 2015

Published18 Oct 2015

Abstract

To facilitate evaluation of complex, organisational health interventions (OHIs), this paper aims at developing a context, process, and outcome (CPO) evaluation model. It builds on previous model developments in the field and advances them by clearly defining and relating generic evaluation categories for OHIs. Context is defined as the underlying frame that influences and is influenced by an OHI. It is further differentiated into the omnibus and discrete contexts. Process is differentiated into the implementation process, as the time-limited enactment of the original intervention plan, and the change process of individual and collective dynamics triggered by the implementation process. These processes lead to proximate, intermediate, and distal outcomes, as all results of the change process that are meaningful for various stakeholders. Research questions that might guide the evaluation of an OHI according to the CPO categories and a list of concrete themes/indicators and methods/sources applied within the evaluation of an OHI project at a hospital in Switzerland illustrate the model’s applicability in structuring evaluations of complex OHIs. In conclusion, the model supplies a common language and a shared mental model for improving communication between researchers and company members and will improve the comparability and aggregation of evaluation study results.

1. Introduction

Recent years have seen a rise of work-related health interventions targeting the entire organisation as complex social systems, thus referred to as organisational health interventions (OHIs) [1]. This paper refers to comprehensive OHIs, which are usually comprised of a mix of individual-directed interventions targeting employee stress management capacities and leadership behaviour combined with work-directed interventions, such as collective workshops targeting working conditions and social relations. Such interventions are portrayed as intricate change processes in complex social systems where implementation and outcomes are never a priori predictable [2, 3].

Currently available meta-analyses and reviews of evaluation studies reveal mixed results for the effectiveness of comprehensive OHIs [4, 5]. Thus, systematic evaluations of OHIs are considered to be ultimately needed, to understand intended and unintended and desirable and undesirable changes and moreover to pass on process and outcome knowledge to other researchers and practitioners in the field [2, 3, 6]. However, since OHIs vary in content, composition, context, targeted audience, and the desired outcomes of change, it is difficult to develop generally applicable evaluation principles [7]. A variety of different evaluation approaches for OHIs have evolved, which either focus on specific aspects of interventions to be evaluated or present general rationales to be further specified and operationalized in regard to a specific project [2, 7, 8]. Additionally, these approaches make use of a diverse terminology, all of which provide a challenge for practitioners and intervention researchers to integrate findings across studies and to select an appropriate evaluation approach for their own intervention project.

So far, only a few researchers have published generic models for the evaluation of complex OHIs. Some models focus on one evaluation category such as process evaluation [7], others are limited to special types of interventions such as preventive occupational health and safety interventions [9], and still others are rather applicable to single component interventions than to multicomponent interventions [10]. Many of them display context as the underlying frame of an intervention but limit it to its hindering or facilitating role [7, 8, 10]. Finally, most of the existing models and frameworks [8, 9] classify effectiveness research/effect evaluation as an autonomous phase after the intervention instead of considering it as a process of continuous observation and assessment throughout the intervention.

All these models and frameworks demonstrate the considerable effort that has been made to standardise and improve the evaluation of organisational health interventions. However, to make the results revealed by evaluations of comprehensive interventions more comparable, a model with clearly defined evaluation categories is needed. Such a model has to be applicable to different intervention types, like single component and multicomponent interventions, interventions that address psychosocial or physiological aspects, and those that focus on the individual, the organisation, or both. Thus, it must be general enough to cover all these different intervention types but at the same time concrete enough to distinguish between the context, process, and outcome aspects at different organisational levels in different interventions phases.

Furthermore, the model has to consider context not only as a static and confounding factor that hinders or facilitates the implementation process, but also as a transformable and essential part of the intervention. The model should also consider the outcome evaluation as a continuous process rather than as a particular, time-limited intervention phase. Moreover, it should be accompanied by examples and research questions that help intervention researchers conduct context, process, and outcome evaluations.

All of the aforementioned models and frameworks only partially meet these needs. Thus, this paper aims at developing a model defining and relating generic evaluation categories for OHIs. First, we develop, define, and relate the generic categories and subcategories of the model. Second, concepts to specify the main categories for developing measurable indicators are compiled, considering different intervention approaches and principles from the literature. Third, research questions that might guide the evaluation of an OHI according to the CPO categories and a list of concrete themes/indicators and methods/sources applied within the evaluation of an OHI at a hospital in Switzerland illustrate the model’s applicability in structuring evaluations of comprehensive OHIs.

2. Overview of the Model

The present model differentiates between the three categories context, process, and outcome evaluation of organisational health interventions and is thus labelled CPO evaluation model (see Figure 1). Context is seen as the underlying frame within an organisational health intervention is implemented, change occurs, and outcomes emerge. We further distinguish between the omnibus context which refers to the general intervention and implementation setting and the discrete context which refers to specific situational variables. In regard to the category of process, two subcategories are differentiated: the implementation process as the time-limited enactment of all steps and elements of the original intervention plan and the triggered change process as all the intended and unintended and observable and nonobservable mechanisms of alteration in the intervention context. This leads to outcomes, defined as all results of the change process observable and measurable in the intervention context. According to the phase of the change process, alterations of proximate, intermediate, and long-term outcomes can be distinguished.

Another important aspect of the model is its grid of phases and levels applied to these categories and subcategories. Whereas phases refer to temporal distinctions within the implementation process and its discrete context (preparation, action cycle, and appropriation), the levels refer to hierarchical aspects of the context and the intended intervention outcomes, spanning from the individual to the organisation. The following description starts with this overlaying grid of phases and levels and then proceeds to the categories and subcategories of context, process, and outcome.

3. Intervention Phases and Levels

3.1. Intervention Phases

OHIs usually consist of several elements such as participatory workshops, survey feedbacks, and information events. These planned intervention elements are comprised within overall intervention architecture, that is, the combination and sequence of intervention elements. Furthermore, intervention planning and evaluation concepts usually distinguish between three and five intervention phases [8, 9, 11–13]. They share in common the notion that an analysis is needed before actions can be planned and implemented. To reduce complexity, the CPO evaluation model proposes three intervention phases: the preparation phase, the action cycle phase, and the appropriation phase.

The first, so-called preparation phase comprises all activities needed to fit the intervention to the specific context and to obtain the commitment of the organisation for the following phases. This includes, for example, presentations and workshops with decision makers, a qualitative analysis of the intervention context, the planning of the intervention architecture (who is involved, when, and how in the OHI), the establishment of a steering group, and project leader.

The second phase is referred to as the action cycle phase that comprises all activities needed to trigger a change process that will improve organisational health on a broad scale. It encompasses the subphases analysis, action planning, enactment, and monitoring. Analysis, for example, an organisation-wide stress assessment, is considered part of the action cycle phase as it not only serves to generate information but also is an active intervention element since its mere implementation might trigger small changes, such as increased awareness, readiness for change, or sensibility for stressful issues [11]. It should be noted that we use the term enactment to replace the commonly used term implementation [8, 9, 12], as the activities in the preparation phase and the final appropriation phase should also be considered as part of the implementation process.

The third phase is named appropriation phase and comprises all activities needed to ensure the continuation, advancement, and diffusion of the change process triggered by the previous two phases. This phase refers to the period when intervention implementers and researchers usually have left the organisation. At that time, capacities for self-optimization have been built up and the organisation and its members have to take over the responsibility for the continuation of the triggered change processes, for example, in the form of continued, repeated action cycles and optimisation processes. As appropriation is a precondition for achieving sustaining long-term effects, we consider it a crucial element of any OHI evaluation.

3.2. Intervention Levels

OHI aims to impact different levels of an organisation, often referred to as combined, individual-organisational or multilevel interventions [14]. In this regard, the CPO evaluation model distinguishes the levels of individuals/leaders and groups/organisation (cf. the IGLO-levels by Nielsen and colleagues [15]). Thus, outcome evaluation should be conducted in consideration of these levels in order to make differentiated statements concerning the effectiveness of an intervention. This differentiation is also important for the evaluation of the intervention context, where different levels can be of importance during different phases. For example, during the preparation phase, organisational level factors such as strategic goals in regard to employee health are of particular importance; during the action cycle phase, leadership level factors such as line manager attitudes are crucial; during the appropriation phase, group level factors such as team climate and capacities for continued optimisation are relevant.

4. The Main Categories and Subcategories of the Model

4.1. Context

For a long time, researchers have considered context to be crucial for understanding the causes of success and failure of interventions [16]. Context is often considered as a process indicator [16–18], but in recent years, the perceived importance of and the attention to contextual issues have increased. Slowly but surely, context has dissociated itself from process issues, becoming an autonomous and meaningful factor in intervention research [19]. Research findings demonstrate that context is a very complex and intervention-specific factor [20] and that its effects can vary from subtle to powerful [21]. For instance, it might occur that an intervention aiming to improve individual resources is implemented in two teams of different sizes, and outcome evaluation may reveal the intervention to be effective in the smaller team while ineffective in the larger team. In the case that evaluators ignore further contextual aspects, they might conclude that the intervention works only in small teams. However, a more differentiated context evaluation might reveal that the line manager of the smaller team strongly supported the intervention while the line manager of the larger team tended to be critical of the intervention. This shows that neglect of contextual aspects could lead to fundamental fallacies concerning the effectiveness of interventions. Accordingly, the concept of context is broadly discussed in OHI research [2] and more broadly in organisational behaviour [21–23]. Various researchers treat context as an unspecific setting parameter that concerns environmental and situational aspects and limit its conceptualization to its facilitating and hindering functions [7, 18, 21]. Moreover, the context of an intervention is often considered as an undesired, uncontrollable, and unmanageable constraint that could be neither predicted nor controlled. For instance, Kompier and Kristensen [24] state that “interventions always take place in context, and that this context is not under control of scientists.”

The conceptualisation of context in the CPO evaluation model follows a taxonomy proposed by Bauer and Jenny [1] (in reference to human resource management practices distinguished by Delery and Doty [25]): “… the organization as the context of [organisational health] interventions might be considered for selecting and targeting the intervention [universalistic approach], for adapting the intervention to this context [contingency approach], or as the final target and actor of change [configurational approach].” Considering this reciprocal, transformational relationship between context and intervention, the CPO evaluation model defines context as the underlying frame that influences and is influenced by an organisational health intervention. As such, context is more than a static, nonchangeable boundary condition; it is a malleable riverbed directing the river of change and simultaneously being shaped by the river. When we consider context as a static boundary condition, we must accept the critical line manager in the above example as a nonchangeable hindering factor and run the risk of implementation failure. When we, on the other hand, consider context as an alterable condition, we can initially adapt the intervention (river) to the critical line manager (river bed) by assigning him/her a specific role, for example, by explicitly representing the critical perspective in the steering committee of the project. By this active involvement, his/her critical position might be transformed into one of constructive support of the intervention.

4.1.1. Omnibus and Discrete Context

The CPO evaluation model distinguishes the omnibus context and the discrete context, as recommended by Johns [21]. The omnibus context describes the general intervention and implementation setting: occupation (who), location (where), time (when), and rationale (why), whereas the discrete context refers to “specific situational variables that influence behaviour directly or moderate relationships between variables” [21]. According to Johns [21], specific situational variables may comprise task aspects (autonomy, uncertainty, accountability, resources, etc.), social aspects (social density, structure, influence, etc.), and physical aspects (temperature, light, built environment, decor, etc.). Relating to OHI, the omnibus context describes the overall setting in which the OHI takes place independently of the three intervention phases. For instance, it may occur that an intervention at the team-level leads to positive changes in team climate, but these changes do not become apparent because of interfering effects triggered by another project that is implemented at the same time (e.g., introduction of shift work). In this case, if evaluators ignore contextual aspects, such as conflicting projects, they might conclude that the team-level intervention was ineffective; however, it actually had positive effects that were superimposed by another project.

The discrete context refers to specific individual, leader, group, and organisational (IGLO) aspects directly relevant to the implementation and change process. Randall and colleagues [26] further distinguish between two temporal kinds of context: the baseline preintervention context and the context of activated intervention. The CPO evaluation model considers this temporal differentiation by referring to the three above-described phases. In the preparation phase, the discrete context is often evaluated in regard to the question of whether the organisation and its members are ready for the next phase so that the implementation process can gain momentum and flow in the action cycle phase as described in the scenario with the critical line manager (see Section 4.1). In the action cycle phase, the discrete context is most commonly evaluated in regard to the factors that, passively or actively, hindered or promoted the flow of the implementation and change process. In the appropriation phase, the discrete context is evaluated in regard to aspects that will maintain and further develop the induced changes or if they will be reversed.

The differentiation between the two kinds of contexts is important with respect to the extent to which the intervention implementers can influence them. Often, aspects of the omnibus context are hardly or not at all manipulable by intervention implementers, for instance, economic developments. It is nevertheless important to monitor and record the omnibus context for an understanding of the implementation process and an interpretation of intervention outcomes. For example, high dismissal rates caused by an economic crisis (omnibus context) will lead to lower absolute participation rates (implementation process) as a matter of course.

The discrete context, on the other hand, can be changed—not easily, but easier than the omnibus context—and, thus, should be considered as a target of change from the beginning. Due to the close proximity of the discrete context to the implementation process, the discrete context (e.g., leaders’ and employees’ readiness for change) can have an immediate and stronger influence on the implementation process than the omnibus context.

4.2. Process

Referring to Bauer und Jenny [1], process covers “both the implementation processes … and the intended and unintended process of change triggered in organizations and their employees, leading to alterations in …outcomes.” Following this understanding, the CPO evaluation model distinguishes between the implementation process and the change process.

4.2.1. Implementation Process

The CPO evaluation model defines the implementation process as the time-limited enactment of all steps and elements of the original intervention plan. The intervention plan, or so-called overarching intervention architecture, concerns the combination and sequence of single intervention elements that may vary in terms of level (IGLO), target audience (e.g., aging workers and specific departments), type (e.g., workshops, survey, presentations, and practical lessons), implementer (internal or external consultants), and so on. Further, the intervention elements can be arranged in parallel (e.g., if they address different target groups) or in a sequential order (e.g., if they build on each other). There are a range of intervention architectures for OHIs, which usually also incorporate a set of general implementation principles to be followed during the process (cf. Table 1). For example, during the preparation phase, successful building of a strong coalition of project leaders, defining goals, and raising awareness and commitment are evaluated (all of which will be part of the subsequent discrete context of the action cycle phase). During the action cycle phase, it is evaluated if the sequence and linkage of the intervention elements are implemented as planned, how employees, managers, and other stakeholders perceive the implementation process, and if the intervention successfully shapes a favourable discrete context for the appropriation phase. The latter means that capacities for self-optimisation are built up so that the organisation and its members are capable and willing to further develop the triggers of change processes autonomously. In the case of formative evaluation assignments, the progress of implementation is monitored continuously in order to make adjustments to the original intervention plan if necessary. In the appropriation phase, it is evaluated whether and how the further development, maintenance, and sustainability of the intervention effects are ensured.

For the evaluation of the implementation of single intervention elements, many researchers focus on quantitative indicators such as reach [27] or dose received [17]. But qualitative implementation indicators are also applied [28]. Commonly, researchers apply measures capturing the perceived quality of an intervention element [29, 30], which has proven to be an important factor when doing process evaluation [17, 30]. However, more research is needed on which indicators concerning the implementation of intervention elements are useful and how the appraisal of particular intervention elements influences the overall impact of an intervention.

Distinction between aspects of the implementation process and the discrete context is sometimes difficult; thus, in the past, it has often been ignored. Figure 2 illustrates the main questions for evaluating the implementation process and the discrete context with regard to the three intervention phases.

4.2.2. Change Process

The CPO evaluation model defines the change process as all intended and unintended individual and collective dynamics triggered by the implementation process, leading to alterations in the organisation and its members. Thus, the change process potentially involves all levels of the intervention context from the individual to the organisation (and their environments). As a current overview of OHI approaches shows [1], change processes include, for example, individual and organisational learning, social processes, taking over others’ perspectives, realisation of jointly developed action plans for improving work, organisational structure, and strategy.

Regarding the timeline of change, the CPO evaluation model is based on the assumption that the implementation and change processes in OHIs are initiated with the beginning of the preparation phase. During the action cycle phase and the appropriation phase, the change process gains drive and develops its intended dynamics. This is illustrated in Figure 1 by the increasing colour density of the change process arrow. Furthermore, the arrow suggests that the change process does not end with the appropriation phase. The implementation process should trigger changes in the everyday processes and structures in organisations leading to short- and long-term outcomes of interest. Due to the multilevel nature of OHIs, change can spread beyond the single topics (e.g., coping with stress, transformative leadership, and teamwork) or particular target audiences (e.g., single divisions and risk groups) initially addressed by the intervention [31]—leading to unforeseen desirable or undesirable side effects.

Evaluation of the change process can help to reveal what changes were triggered on which level by the implementation process and thus help to better understand the mechanisms of change. There is a lot of research on the different types of change. Authors discuss whether change should be considered as episodic or continuous [32], exceptional or natural [33], but when it comes to the assessment of change, most researchers focus on aspects of the discrete context that influence or shape the change process in a hindering or facilitating way [34, 35]. As the change process is very complex and only partly observable, it still remains as a kind of black box. In practice, intervention implementers usually compare baseline and follow-up measures of outcome variables in order to make induced individual and organisational changes visible. Research on valid change process indicators is still scarce, but the psychosocial mechanisms of change described by Karanika-Murray and Biron [36] demonstrate a promising approach to fill this gap (cf. Table 1). They present “six psychosocial mechanisms that can explain how an intervention can exert its impact on individuals and workgroups: diffusion, sharing, identifying, comparing, influencing and learning” [36]. Evaluating whether, how, and to which extent these mechanisms have evolved, for example, whether and how intervention effects at the individual level have diffused throughout an entire team, might help to make the change process more visible.

4.3. Outcomes

Outcome evaluation is mainly concerned with what effects an intervention has had [37]. The CPO evaluation model defines outcomes as all results of the change process that are measurable and at the same time meaningful for the organisation, its members, researchers, and other stakeholders. The CPO model utilises the following three outcome categories: proximate outcomes, intermediate outcomes, and distal outcomes. These categories are based on a trichotomy of outcomes commonly used in the public health community [38–41]. As all outcomes manifest in the intervention context, proximate, intermediate, and distal outcomes can be further observed on all levels of individuals, leaders, groups, and the organisation.

Proximate outcomes, often also labelled as immediate-, initial- or short-term outcomes or first-level targets, refer, for example, to individual skills and collective capacities needed for the change process (i.e., as part of the discrete context) as well as quick-wins in the form of minor but instant structural changes. As such, they can be classified as results of the change process that immediately arise.

Intermediate outcomes, often also labelled as medium-term outcomes or second-level targets, comprise, for example, changes in job demands and resources with regard to factual (job-related) processes such as work load or time pressure and social (people-related) processes such as leadership behaviour or social support [42]. As such, we can define them as results of the change process with regard to factual (job-related) and social (people-related) processes.

Distal outcomes refer to the distal objectives of the interventions such as improved individual health and increased organisational performance, which often depend on change in the intermediate outcomes. These outcomes are often labelled as the overall objectives/goals or, simply, impacts, and we, thus, define them as higher-level results of the change process that evolve over time.

In the literature, most intervention process cycles end up with the evaluation phase measuring the outcome, impact, or effect of the intervention [8, 11, 12]. The CPO evaluation model assumes that changes in outcomes happen continuously as a result of the change process induced by the continuous implementation process; therefore the CPO model depicts evaluation of proximate, intermediate, and distal outcomes in a separate box of the CPO model. Thus, outcome evaluation should be considered not only as an important intervention element where results of the change process are fed back into the organisational system at the end of the intervention, but also as a continuous observation and assessment of change results accompanying the entire implementation and change process, for example, by continuously measuring proximal outcomes to show successful growth in these variables. In this regard, change in proximate outcomes might already be visible after a project information event (preparation phase) or after a particular intervention element, such as a stress management workshop (action cycle phase). Thus, collection and reflection of outcome data are also parts of an intervention and can influence the change process [11].

5. Practical Application of the CPO Evaluation Model

5.1. Possible Concepts to Specify the Main CPO Categories

The CPO model defines generic categories, which means that they have to be further specified and operationalised by concrete indicators for the purpose of measurement and evaluation. Table 1 presents a variety of possible concepts and related indicators for these generic categories of the CPO model, considering different intervention approaches and principles. The list is not exhaustive and is mainly compiled on the basis of recent intervention and evaluation research in occupational health psychology [1]. The concrete selection of indicators will depend on the specific intervention theories and logic; interest of stakeholders; and available resources such as budget, time, and manpower. The concepts proposed for evaluating the main CPO categories reveal considerable variance in specification, terminology, and objective—although common themes emerge and could be condensed in future work.

5.2. Practical Application of the CPO Model Using the Example of an Evaluation of an OHI at a Hospital in Switzerland

Table 1 presents a list of relatively general concepts to specify the main CPO categories. In order to clarify how the theoretical concepts of the CPO categories can be translated into practice and to show the practical application of the CPO evaluation model in a more concrete way, Table 2 exemplarily illustrates the model’s use for the conceptualization of an OHI evaluation by an OHI project that included 31 nursing divisions from a hospital in Switzerland (see Acknowledgments) implemented between 2013 and 2015.

The data collection, data analyses, and the evaluation report of this OHI were structured using the CPO logic. In the first column of Table 2, research questions that might guide the evaluation of an OHI along the CPO categories are presented. The second and third columns show the applied themes/indicators and methods/sources for each CPO category. To better understand the applied themes/indicators and methods/sources, a brief overview of the project with regard to the intervention’s architecture, goals, elements, and evaluation instruments is presented in the following paragraph.

The intervention focused on optimizing the working processes in the nursing divisions with regard to lean principles (i.e., reducing waste and enhancing value), fostering interdisciplinary collaboration between nursing and medical staff, strengthening team climate within the divisions, and improving the balance of resources and demands of the nursing teams. A four-day workshop was implemented in each division by the hospital’s internal process managers as the main intervention component. A representative selection of employees participated in the workshop; workshop participants consisted of registered nurses at all hierarchical levels, including ward managers. The workshop was evaluated using a paper-based evaluation questionnaire. To evaluate the intervention as a whole, an online survey (comprising items on resources, demands, team climate, work-life balance, etc.) was applied three times at six-month intervals in each division.

The evaluation had a waitlist control group design; that is, the 31 nursing divisions were randomly assigned to either an intervention () or control group (). For the intervention groups, the workshop took place four to six weeks after the first online survey; for the control group, the workshop took place four to six weeks after the second online survey. Furthermore, four focus group discussions were conducted at the end of the intervention project to gather data on the implementation and change process as well as on the discrete context. The focus group discussion participants were composed as follows: (1) workshop participants who rated the intervention impact positive; (2) workshop participants who rated the intervention impact negative; (3) nonworkshop participants who rated the intervention impact positive; and (4) nonworkshop participants who rated the intervention impact negative.

6. Limitations

A thorough evaluation of all CPO categories and subcategories is a difficult endeavour requiring a lot of resources and instruments. In many cases, it will not be possible to evaluate all these aspects due to scarce resources or limited access to information. Furthermore, it can be difficult to specify the CPO categories and develop suitable indicators due to the variety of possible concepts. However, the CPO model at least offers an overview of which categories could be evaluated and how these categories could be specified. Depending on the particular intervention project, evaluators can then consciously decide which of these categories should be evaluated and to what extent. It can sometimes be difficult to distinguish between aspects of the implementation process and the discrete context but the questions displayed in Figure 2 may facilitate the distinction of these two categories. The most arguable category of the CPO model is the change process, as it still remains a kind of black box. Consequently, more research is needed on how to assess the change process. Furthermore, the CPO model was only tested in two intervention projects so far. Use of the CPO model across diverse OHIs and evaluation studies may produce a systematic evidence base, building on generic categories for collecting and reporting data. This will help researchers and practitioners to develop more effective and sustainable interventions in the future. Finally, preexisting concepts and indicators have to be tested and compared in different intervention projects in future research in order to identify the critical concepts and develop corresponding valid indicators.

7. Conclusions

The CPO evaluation model provides a basis for structured evaluation of combined OHIs in the field by combining context, process, and outcome evaluation. It offers generic evaluation categories and subcategories that are further differentiated in terms of time and hierarchy and displayed by a grid of intervention phases and organisational levels that facilitates detecting changes at different times and levels. Furthermore, it provides a clear taxonomy for a wide range of possible concepts specifying the evaluation categories and subcategories. Development, testing, and selection of concrete indicators need to be realized in future research. However, descriptions of the evaluation of the OHI project at the hospital demonstrate how an evaluation guided and structured by the CPO evaluation model might look. The research questions presented in Table 2 will support intervention researchers in selecting appropriate indicators for a particular intervention project.

In comparison to similar models, the CPO model uses a clearly defined terminology for OHIs, which might facilitate the development of a common language for improving both communication between researchers and company members and the comparability and aggregation of evaluation study results.

Second, it distinguishes the implementation process from the change process. This distinction is essential as it helps differentiate between cause (i.e., the implementation process) and effect (i.e., the triggered change process), which in turn helps researchers and company members understand the mechanics of change. Following Harachi and colleagues [43], it also helps identify the possible causes of intervention failure. In the first instance, the failure may be caused by an implementation failure, which means “that the way the intervention was implemented was incomplete or designed in such a way that the intervention would have failed even if the theory behind the intervention was correct” [7]. Alternatively, the theory/programme failure may be based on false assumptions about how the implementation of the intervention translates into desired outcomes through an assumed change process, meaning “that the theory behind the problem did not address the problem” [7].

Third, it assumes a reciprocal relationship between the implementation process and the discrete context. By doing so, it broadens the hitherto existing understanding of a static context by defining it as a dynamic factor that needs to be systematically considered and that can be transformed during the intervention.

Fourth, it considers outcome evaluation as the continuous observation and assessment of the change results accompanying the entire intervention. Conducting an outcome evaluation continuously from the beginning will help to better understand the dynamics of the change process and to prevent evaluation results from being subject to hindsight biases.

Overall, the CPO evaluation model can serve as a shared mental model throughout the complex intervention evaluation process by supporting organisational members, project leaders, implementers, and evaluators in establishing objectives, selecting possible evaluation concepts from OHI literature, developing key indicators, gathering data, and reporting evaluation results in a structured and succinct way.

Conflict of Interests

The authors have declared that no conflict of interests exists.

Acknowledgments

The intervention project described in the paper was financed by “Lotteriefonds,” Suva, as well as the University Hospital Zurich. Annemarie Fridrich was supported by the Swiss National Science Foundation (SNSF).

References

G. F. Bauer and G. J. Jenny, “From fidelity to figuration: current and emerging approaches to organizational health intervention research,” in Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research, G. B. Bauer and G. J. Jenny, Eds., Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
C. Biron, M. Karanika-Murray, and C. L. Cooper, “Organizational interventions for stress and well-being—an overview,” in Improving Organizational Interventions for Stress and Well-Being. Addressing Process and Context, C. Biron, M. Karanika-Murray, and C. L. Cooper, Eds., pp. 1–17, Routledge, Hove, UK, 2012.
View at: Google Scholar
T. Cox, M. Karanika, A. Griffiths, and J. Houdmont, “Evaluating organizational-level work stress interventions: beyond traditional methods,” Work & Stress, vol. 21, no. 4, pp. 348–362, 2007.
View at: Publisher Site | Google Scholar
A. D. LaMontagne, T. Keegel, A. M. Louie, A. Ostry, and P. A. Landsbergis, “A systematic review of the job-stress intervention evaluation literature, 1990–2005,” International Journal of Occupational and Environmental Health, vol. 13, no. 3, pp. 268–280, 2007.
View at: Publisher Site | Google Scholar
K. M. Richardson and H. R. Rothstein, “Effects of occupational stress management intervention programs: a meta-analysis,” Journal of Occupational Health Psychology, vol. 13, no. 1, pp. 69–93, 2008.
View at: Publisher Site | Google Scholar
K. Nielsen, T. W. Taris, and T. Cox, “The future of organizational interventions: addressing the challenges of today's organizations,” Work & Stress, vol. 24, no. 3, pp. 219–233, 2010.
View at: Publisher Site | Google Scholar
K. Nielsen and R. Randall, “Opening the black box: presenting a model for evaluating organizational-level interventions,” European Journal of Work and Organizational Psychology, vol. 22, no. 5, pp. 601–617, 2013.
View at: Publisher Site | Google Scholar
K. Nielsen and J. S. Abildgaard, “Organizational interventions: a research-based framework for the evaluation of both process and effects,” Work and Stress, vol. 27, no. 3, pp. 278–297, 2013.
View at: Publisher Site | Google Scholar
L. M. Goldenhar, A. D. LaMontagne, T. Katz, C. Heaney, and P. Landsbergis, “The intervention research process in occupational safety and health: an overview from the National Occupational Research Agenda Intervention Effectiveness Research team,” Journal of Occupational and Environmental Medicine, vol. 43, no. 7, pp. 616–622, 2001.
View at: Publisher Site | Google Scholar
R. Pawson and N. Tilley, Realistic Evaluation, Sage, London, UK, 1997.
A. Inauen, G. J. Jenny, and G. F. Bauer, “Design principles for data- and change-oriented organisational analysis in workplace health promotion,” Health Promotion International, vol. 27, no. 2, pp. 275–283, 2012.
View at: Publisher Site | Google Scholar
P. Mendel, L. S. Meredith, M. Schoenbaum, C. D. Sherbourne, and K. B. Wells, “Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research,” Administration and Policy in Mental Health and Mental Health Services Research, vol. 35, no. 1-2, pp. 21–37, 2008.
View at: Publisher Site | Google Scholar
E. Wimbush and J. Watson, “An evaluation framework for health promotion: theory, quality and effectiveness,” Evaluation, vol. 6, no. 3, pp. 301–321, 2000.
View at: Publisher Site | Google Scholar
N. K. Semmer, “Job stress interventions and the organization of work,” Scandinavian Journal of Work, Environment and Health, vol. 32, no. 6, pp. 515–527, 2006.
View at: Publisher Site | Google Scholar
K. Nielsen, M. Stage, J. S. Abildgaard, and C. V. Brauer, “Participatory intervention from an organizational perspective: employees as active agents in creating a healthy work environment,” in Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
L. Linnan and A. B. Steckler, “Process evaluation for public health interventions and research: an overview,” in Process Evaluation for Public Health Interventions and Research, A. B. Steckler and L. Linnan, Eds., pp. 1–24, Jossey-Bass, San Francisco, Calif, USA, 2002.
View at: Google Scholar
S. G. Murta, K. Sanderson, and B. Oldenburg, “Process evaluation in occupational stress management programs: a systematic review,” The American Journal of Health Promotion, vol. 21, no. 4, pp. 248–254, 2007.
View at: Publisher Site | Google Scholar
J. van Berkel, C. R. L. Boot, K. I. Proper, P. M. Bongers, and A. J. van der Beek, “Process evaluation of a workplace health promotion intervention aimed at improving work engagement and energy balance,” Journal of Occupational and Environmental Medicine, vol. 55, no. 1, pp. 19–26, 2013.
View at: Publisher Site | Google Scholar
C. Biron, M. Karanika-Murray, and C. L. Cooper, Eds., Improving Organizational Interventions for Stress and Well-Being: Addressing Process and Context, Routledge, Hove, UK, 2012.
R. Randall, K. Nielsen, and S. D. Tvedt, “The development of five scales to measure employees' appraisals of organizational-level stress management interventions,” Work and Stress, vol. 23, no. 1, pp. 1–23, 2009.
View at: Publisher Site | Google Scholar
G. Johns, “The essential impact of context on organizational behavior,” Academy of Management Review, vol. 31, no. 2, pp. 386–408, 2006.
View at: Publisher Site | Google Scholar
M. A. Griffin, “Specifying organizational contexts: systematic links between contexts and processes in organizational behavior,” Journal of Organizational Behavior, vol. 28, no. 7, pp. 859–863, 2007.
View at: Publisher Site | Google Scholar
R. T. Mowday and R. I. Sutton, “Organizational behavior: linking individuals and groups to organizational contexts,” Annual Review of Psychology, vol. 44, no. 1, pp. 195–229, 1993.
View at: Publisher Site | Google Scholar
M. Kompier and T. S. Kristensen, “Organizational work stress interventions in a theoretical, methodological and practical context,” in Stress in the Workplace. Past, Present and Future, J. Dunham, Ed., pp. 164–190, Whurr Publishers, London, UK, 2000.
View at: Google Scholar
J. E. Delery and D. H. Doty, “Modes of theorizing in strategic human resource management: tests of universalistic, contingency, and configurational performance predictions,” Academy of Management Journal, vol. 39, no. 4, pp. 802–835, 1996.
View at: Publisher Site | Google Scholar
R. Randall, T. Cox, and A. Griffiths, “Participants' accounts of a stress management intervention,” Human Relations, vol. 60, no. 8, pp. 1181–1208, 2007.
View at: Publisher Site | Google Scholar
R. E. Glasgow, E. Lichtenstein, and A. C. Marcus, “Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition,” American Journal of Public Health, vol. 93, no. 8, pp. 1261–1267, 2003.
View at: Publisher Site | Google Scholar
G. Baril-Gingras, M. Bellemare, and C. Brisson, “How can qualitative studies help understand the role of context and process of interventions on occupational safety and health and on mental health at work?” in Improving Organizational Interventions for Stress and Well-Being: Addressing Process and Context, C. Biron, M. Karanika-Murray, and C. L. Cooper, Eds., pp. 135–162, Routledge, Hove, UK, 2012.
View at: Google Scholar
B. Aust, R. Rugulies, A. Finken, and C. Jensen, “When workplace interventions lead to negative effects: learning from failures,” Scandinavian Journal of Public Health, vol. 38, no. 3, supplement, pp. 106–119, 2010.
View at: Publisher Site | Google Scholar
K. Nielsen, R. Randall, and K. Albertsen, “Participants' appraisals of process issues and the effects of stress management interventions,” Journal of Organizational Behavior, vol. 28, no. 6, pp. 793–810, 2007.
View at: Publisher Site | Google Scholar
M. Karanika-Murray, C. Biron, and C. L. Cooper, “Concluding comments. Distilling the elements of successful organizational intervention implementation,” in Improving Organizational Interventions for Stress and Well-Being. Addressing Process and Context, C. Biron, M. Karanika-Murray, and C. L. Cooper, Eds., pp. 353–361, Routledge, Hove, UK, 2012.
View at: Google Scholar
L. L. Martins, “Organizational change and development,” in APA Handbook of Industrial and Organizational Psychology, S. Zedeck, Ed., pp. 691–728, American Psychological Association, Washington, DC, USA, 2011.
View at: Google Scholar
H. Tsoukas and R. Chia, “On organizational becoming: rethinking organizational change,” Organization Science, vol. 13, no. 5, pp. 567–582, 2002.
View at: Publisher Site | Google Scholar
A. A. Armenakis and S. G. Harris, “Reflections: our journey in organizational change research and practice,” Journal of Change Management, vol. 9, no. 2, pp. 127–142, 2009.
View at: Publisher Site | Google Scholar
L. E. Tetrick, J. C. Quick, and P. L. Gilmore, “Research in organizational interventions to improve well-being: perspectives on organizational change and development,” in Improving Organizational Interventions for Stress and Well-Being. Addressing Process and Context, C. Biron, M. Karanika-Murray, and C. L. Cooper, Eds., pp. 59–76, Routledge, Hove, UK, 2012.
View at: Google Scholar
M. Karanika-Murray and C. Biron, “The nature of change in organizational health interventions: some observations and propositions,” in Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., pp. 239–258, Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
R. Z. Goetzel and R. J. Ozminkowski, How to Evaluate Workplace Health Promotion Programs, American Journal of Health Promotion, Keego Harbor, Mich, USA, 2000.
D. A. Julian, A. Jones, and D. Deyo, “Open systems evaluation and the logic model: program planning and evaluation tools,” Evaluation and Program Planning, vol. 18, no. 4, pp. 333–341, 1995.
View at: Publisher Site | Google Scholar
M. M. Mark and G. T. Henry, “Logic models and content analyses for the explication of evaluation theories: the case of emergent realist evaluation,” Evaluation and Program Planning, vol. 38, pp. 74–76, 2013.
View at: Publisher Site | Google Scholar
J. A. McLaughlin and G. B. Jordan, “Logic models: a tool for telling your program's performance story,” Evaluation and Program Planning, vol. 22, no. 1, pp. 65–72, 1999.
View at: Publisher Site | Google Scholar
D. Nutbeam, “Evaluating health promotion—progress, problems and solutions,” Health Promotion International, vol. 13, no. 1, pp. 27–44, 1998.
View at: Publisher Site | Google Scholar
G. J. Jenny and G. F. Bauer, “The limits of control: a systemic model-based approach to changing organisations towards better health,” in Salutogenic Organizations and Change: The Concepts behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., pp. 167–187, Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
T. W. Harachi, R. D. Abbott, R. F. Catalano, K. P. Haggerty, and C. B. Fleming, “Opening the black box: using process evaluation measures to assess implementation and theory building,” The American Journal of Community Psychology, vol. 27, no. 5, pp. 711–731, 1999.
View at: Publisher Site | Google Scholar
National Institute for Occupational Safety and Health (NIOSH), The Changing Organization of Work and the Safety and Health of Working People: Knowledge Gaps and Research Directions, National Institute for Occupational Safety and Health (NIOSH), Washington, DC, USA, 2002.
B. J. Weiner, H. Amick, and S.-Y. D. Lee, “Review: conceptualization and measurement of organizational readiness for change. A review of the literature in health services research and other fields,” Medical Care Research and Review, vol. 65, no. 4, pp. 379–436, 2008.
View at: Publisher Site | Google Scholar
J. M. Prochaska, “A transtheoretical model for assessing organizational change: a study of family service agencies' movement to time-limited therapy,” Families in Society, vol. 81, no. 1, pp. 76–84, 2000.
View at: Publisher Site | Google Scholar
P. Ø. Saksvik, S. D. Tvedt, K. Nytrø et al., “Developing criteria for healthy organizational change,” Work and Stress, vol. 21, no. 3, pp. 243–263, 2007.
View at: Publisher Site | Google Scholar
R. Randall, “Process monitoring in intervention research: a ‘dashboard’ with six dimensions,” in Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
A. Linna, A. Väänänen, M. Elovainio, M. Kivimäki, J. Pentti, and J. Vahtera, “Effect of participative intervention on organisational justice perceptions: a quasi-experimental study on finnish public sector employees,” International Journal of Human Resource Management, vol. 22, no. 3, pp. 706–721, 2011.
View at: Publisher Site | Google Scholar
T. L. Veach, R. H. Rahe, R. L. Tolles, and L. M. Newhall, “Effectiveness of an intensive stress intervention workshop for senior managers,” Stress and Health, vol. 19, no. 5, pp. 257–264, 2003.
View at: Publisher Site | Google Scholar
M. Gerkhardt and D. Frey, “Erfolgsfaktoren und psychologische Hintergründe in Veränderungsprozessen: Entwicklung eines integrativen psychologischen Modells,” OrganisationsEntwicklung, vol. 25, no. 4, pp. 48–59, 2006.
View at: Google Scholar
C. Ipsen and V. Andersen, “A multi-level and participatory model for prevention of work-related stress in knowledge work,” in Salutogenic Organizations and Change: The Concepts behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., pp. 127–148, Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
G. J. Jenny, A. Inauen, R. Brauchli, D. Füllemann, F. Müller, and G. F. Bauer, SWiNG Project—Final Report of the Evaluation, Public and Organizational Health Department, Zürich, Switzerland, 2011.
G. J. Jenny and G. F. Bauer, “The limits of control: a systemic, modelbased approach to changing organisations towards better health,” in Salutogenic Organizations and Change. The Concepts behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
R. A. Karasek, “Job demands, job decision latitude, and mental strain: implications for job redesign,” Administrative Science Quarterly, vol. 24, no. 2, pp. 285–308, 1979.
View at: Publisher Site | Google Scholar
J. Siegrist, “Adverse health effects of high-effort/low-reward conditions.,” Journal of Occupational Health Psychology, vol. 1, no. 1, pp. 27–41, 1996.
View at: Publisher Site | Google Scholar
A. B. Bakker and E. Demerouti, “The job demands-resources model: state of the art,” Journal of Managerial Psychology, vol. 22, no. 3, pp. 309–328, 2007.
View at: Publisher Site | Google Scholar
G. J. Jenny, R. Brauchli, A. Inauen, D. Füllemann, A. Fridrich, and G. F. Bauer, “Process and outcome evaluation of an organizational-level stress management intervention in Switzerland,” Health Promotion International, 2014.
View at: Publisher Site | Google Scholar
N. R. Anderson and M. A. West, “Measuring climate for work group innovation: development and validation of the team climate inventory,” Journal of Organizational Behavior, vol. 19, no. 3, pp. 235–258, 1998.
View at: Publisher Site | Google Scholar
M. Salanova, S. Llorens, E. Cifre, and I. M. Martínez, “We need a hero! toward a validation of the Healthy and Resilient Organization (HERO) model,” Group and Organization Management, vol. 37, no. 6, pp. 785–822, 2012.
View at: Publisher Site | Google Scholar
L. Vaandrager and M. Koelen, “Salutogenesis in the workplace: building general resistance resources and sense of coherence,” in Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., pp. 77–89, Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar
K. Vogt, G. J. Jenny, and G. F. Bauer, “Comprehensibility, manageability and meaningfulness at work: construct validity of a scale measuring work-related sense of coherence,” SA Journal of Industrial Psychology, vol. 39, no. 1, article 1111, 2013.
View at: Publisher Site | Google Scholar
T. S. Kristensen, H. Hannerz, A. Høgh, and V. Borg, “The Copenhagen psychosocial questionnaire—a tool for the assessment and improvement of the psychosocial work environment,” Scandinavian Journal of Work, Environment & Health, vol. 31, no. 6, pp. 438–449, 2005.
View at: Publisher Site | Google Scholar
O. Idan, O. Braun-Lewensohn, and S. Sagy, “Qualitative, sense of coherence-based assessment of working conditions in a psychiatric in-patient unit to guide salutogenic interventions,” in Salutogenic Organizations and Change. The Concepts behind Organizational Health Intervention Research, G. F. Bauer and G. J. Jenny, Eds., pp. 55–74, Springer, Dordrecht, The Netherlands, 2013.
View at: Google Scholar

Copyright

Copyright © 2015 Annemarie Fridrich et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

13995

Downloads

3150

Citations