Table of Contents Author Guidelines Submit a Manuscript
Nursing Research and Practice
Volume 2014 (2014), Article ID 896587, 28 pages
http://dx.doi.org/10.1155/2014/896587
Review Article

A Systematic Review of the Cost-Effectiveness of Nurse Practitioners and Clinical Nurse Specialists: What Is the Quality of the Evidence?

1Daphne Cockwell School of Nursing, Ryerson University, 350 Victoria Street, Toronto, ON, Canada M5B 2K3
2Faculty of Nursing, Université de Montreal and Research Centre of Hôpital Maisonneuve-Rosemont, CSA-RC-Aile Bleue-Room F121, 5415 boulevard l’Assomption, Montréal, QC, Canada H1T 2M4
3KJ Research, Rosemere, QC, Canada J7A 4N8
4School of Nursing, McMaster University, 1280 Main Street West, Hamilton, ON, Canada L8S 4L8
5School of Nursing, Dalhousie University, Box 15000, 5869 University Avenue, Halifax, NS, Canada B3H 4R2
6Department of Oncology, McMaster University, 1280 Main Street West, HSC-3N28G, Hamilton, ON, Canada L8S 4L8
7Health Interventions Research Centre, Ryerson University, 350 Victoria Street, Toronto, ON, Canada M5B 2K3
8Department of Community Health Sciences, Faculty of Medicine, University of Calgary, Health Research Innovation Centre, Room 3C56, 3280 Hospital Drive NW, Calgary, AB, Canada T2N 4Z6
9Fraser Health Authority, Suite 400-13450 102nd Avenue, Surrey, BC, Canada V3T 0H1
10International Affairs and Best Practice Guidelines Centre, Registered Nurses’ Association of Ontario, 158 Pearl Street, Toronto, ON, Canada M5H 1L3
11School of Nursing, University of Victoria, P.O. Box 1700 STN CSC, Victoria, BC, Canada V8W 2Y2
12The Hospital for Sick Children, Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, 555 University Avenue, Toronto, ON, Canada M5G 1X8
13Primary Health Care Nurse Practitioner Program, School of Nursing, University of Ottawa, 600 Peter Morand Crescent, Suite 101, Ottawa, ON, Canada K1G 5Z3
14Departments of Pediatrics and Psychology and Neurosciences, Dalhousie University, P.O. Box 15000, 5869 University Avenue, Halifax, NS, Canada B3H 4R2
15School of Health Studies, Western University, Health Sciences Building, Room 403, London, ON, Canada N6A 5B9
16Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, ON, Canada L8S 4L8

Received 2 March 2014; Revised 26 June 2014; Accepted 27 June 2014; Published 1 September 2014

Academic Editor: Patrick Callaghan

Copyright © 2014 Faith Donald et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background. Improved quality of care and control of healthcare costs are important factors influencing decisions to implement nurse practitioner (NP) and clinical nurse specialist (CNS) roles. Objective. To assess the quality of randomized controlled trials (RCTs) evaluating NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization). Design. Systematic review of RCTs of NP and CNS cost-effectiveness reported between 1980 and July 2012. Results. 4,397 unique records were reviewed. We included 43 RCTs in six groupings, NP-outpatient (), NP-transition (), NP-inpatient (), CNS-outpatient (), CNS-transition (), and CNS-inpatient (). Internal validity was assessed using the Cochrane risk of bias tool; 18 (42%) studies were at low, 17 (39%) were at moderate, and eight (19%) at high risk of bias. Few studies included detailed descriptions of the education, experience, or role of the NPs or CNSs, affecting external validity. Conclusions. We identified 43 RCTs evaluating the cost-effectiveness of NPs and CNSs using criteria that meet current definitions of the roles. Almost half the RCTs were at low risk of bias. Incomplete reporting of study methods and lack of details about NP or CNS education, experience, and role create challenges in consolidating the evidence of the cost-effectiveness of these roles.

1. Introduction

Nurse practitioners (NPs) and clinical nurse specialists (CNSs) have practiced for over 50 years in the United States, followed closely by Canada and the United Kingdom, and the roles are increasingly being implemented in other countries [1]. The quest for improved quality of care and control of healthcare costs are important drivers in the decision to implement these roles. We conducted a systematic review to assess the evidence of cost-effectiveness of NP and CNS roles.

2. Background

Both NPs and CNSs are considered advanced practice nurses [2]. NPs are defined as RNs who have additional education in recognized programs, preferably at the graduate level. They demonstrate advanced competencies to practice autonomously and collaboratively to perform assessments, order laboratory and diagnostic tests, diagnose, prescribe medications and treatments, and perform procedures, as authorized by legislation and their regulatory scope of practice [2], as well as performing an advanced nursing role that includes consultation, collaboration, education, research, and leadership. CNSs are registered nurses (RNs) with a graduate degree in nursing who have expertise in a clinical specialty and perform an advanced nursing role that includes practice, consultation, collaboration, education, research, and leadership [3].

NPs and CNSs function in alternative or complementary provider roles. Those working in alternative roles provide similar services to those for whom they are substituting, usually physicians [4]. Those working in complementary roles provide additional services that are intended to complement or extend existing services. The intention of the alternative role is typically to reduce cost or workload or to address workforce shortages while maintaining or improving the quality of care; in contrast, the intention of the complementary role is to improve the quality of care [5].

During the 1970s, the first randomized controlled trials (RCTs) of NPs demonstrated their safety and effectiveness, as well as patient satisfaction with the NP role [617]. NPs improved resource utilization and access to care [14, 1820], increased primary care services in the community [7], and reduced costs [15]. Over the past 30 years, a number of literature reviews and systematic reviews have summarized the findings of studies evaluating NPs [2125]. The reviews have consistently shown no difference in the health outcomes of patients receiving NP care when compared to patients receiving physician care, but often both quality of care and patient satisfaction are higher with NP care.

Most RCTs of CNS roles have been published since 1980 except one. In 1977, Pozen and colleagues [26] found that the CNS increased the knowledge of heart disease in patients with myocardial infarction resulting in an increased rate of return to work and a reduction in smoking. Literature reviews and systematic reviews of CNSs [25, 27] reveal that CNSs are associated with reductions in hospital length of stay, readmissions, emergency room visits, and costs, as well as improvements in staff nurse knowledge, functional performance, mood state, quality of life, and patient satisfaction.

Study findings are consistent that NPs and CNSs, either in alternative or complementary provider roles, deliver high quality patient care that results in high patient satisfaction. To address a question that often surfaces, “are NPs and CNSs cost-effective?,” we conducted a systematic review of RCTs of NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization) entitled “A systematic review of the cost-effectiveness of nurse practitioners and clinical nurse specialists: 1980 to July 2012.” The purpose of this paper is to report on the methodological strengths and threats to internal and external validity of these RCTs.

3. Methods

3.1. Eligibility Criteria

We sought RCTs of NP and CNS cost-effectiveness between January 1980 and July 2012. Due to inconsistencies in the use of titles and lack of role clarity for these two roles [28], we developed specific criteria to decide if the role was an NP, a CNS, or an RN in an expanded role. To be deemed an NP, the nurse had to have completed a formal postbaccalaureate or graduate NP education program or be licensed as an NP. To be deemed a CNS, the nurse had to have completed a graduate degree and the role had to be reflective of the CNS role definition. If necessary, we contacted the lead author and/or experts in advanced practice nursing from the country where the study was conducted to determine eligibility.

The principal outcomes of interest in this review were objective measures of health system utilization. These included length of stay, rehospitalization, costs of healthcare (e.g., hospital, professional, and family costs), and health resource use (e.g., diagnostic tests and prescriptions). Because it is important to examine health system utilization in the context of patient and provider outcomes, we also extracted data on all patient (e.g., mortality, morbidity, quality of life, and satisfaction with care) and provider (e.g., quality of care and job satisfaction) outcomes.

Participants were patients of any age receiving care in all types (e.g., teaching and nonteaching, public and private), sizes (e.g., small, medium, and large), and locations (e.g., rural and urban) of hospitals or community settings (e.g., long-term care, primary care, and home care).

Substantive developments since 1980 (e.g., training, payment models, and scope of practice of NPs) have reduced the relevance of pre-1980 studies to modern-day policy. In consultation with a policy advisor, we chose to exclude pre-1980 studies from this review. Studies were also excluded if (1) the NP or CNS education failed to meet our criteria or if we could not contact the author for clarification despite repeated attempts; (2) the NP or CNS was part of a multicomponent or multidisciplinary intervention in which the impact of their contribution could not be isolated from other healthcare providers on the team; (3) the study evaluated a very specific intervention (e.g., cognitive behavioural therapy) that was delivered by an NP or CNS but could be delivered by other clinicians, such as an RN; (4) the control group was also exposed to an NP or CNS during the study; (5) a measure of health system utilization was not included; (6) true randomization was not used (randomization was predictable, for example, assignment by day of hospital admission and alternating assignment).

3.2. Search Strategy

A search was conducted to identify all relevant published and unpublished RCTs reported from January 1980 to July 2012. No restrictions were imposed on jurisdiction or language. Medical librarians conducted a comprehensive search of the literature using CINAHL, EMBASE, Global Health, HealthStar, Medline, Allied and Complementary Medicine Database (AMED), Cochrane Library Database of Systematic Reviews and Controlled Trials Register, Database of Abstracts of Reviews of Effects (DARE), Health Economics Evaluation Database (HEED), and Web of Science. Relevant Medical Subject Headings (MeSH) keywords, inclusive suffixes, and search strings formed the search strategy (appendix). In addition, the following methods were used to identify primary studies: handsearching of 16 high-yield journals, checking reference lists of all relevant papers and reviews, contacting authors of an early list of relevant studies, searching personal files, reviewing bibliographies, and searching websites of nursing research and professional organizations and national, provincial/state, and territorial governments.

3.3. Study Selection

We uploaded all identified citations to a web-based reference management program (RefWorks) and removed duplicate entries. Two-member teams independently screened titles and abstracts of these citations for relevance using prespecified criteria. Translators assisted with the review of all citations in languages other than French or English. The full-text of a published paper and/or study report was obtained if it appeared to meet the inclusion criteria, if an abstract was unavailable, or if it was not possible to determine relevance from the title and abstract review. In instances where a study was reported in more than one paper, we grouped the study’s papers in a constellation and collectively reviewed them. Two-member teams independently screened these full-text papers for eligibility based on the inclusion criteria. Discrepancies were discussed and resolved by consensus. We catalogued all excluded studies and the reason for exclusion. Studies that met eligibility criteria advanced to the quality assessment phase of the review.

3.4. Quality Assessment

Two team members (AD and KR) independently assessed the methodological quality of the studies for internal validity and disagreements were resolved through discussion and consensus. The internal validity of each study was assessed using a slightly modified version of the Cochrane risk of bias criteria [29]; modifications to the criteria were three-fold. First, we did not assess for blinding of participants and personnel because the nature of NP and CNS interventions precludes this possibility. Second, outcome assessment and completeness of outcome data were evaluated separately for objective and subjective outcomes within a study. We looked for evidence of key outcomes that would typically be measured for each study’s research question [29]. Third, if outcomes had more than 20% missing data, we judged the study to be at high risk of bias for “incomplete outcome data.”

We assessed studies, assigning a high, low, or unclear risk of bias for each of the following eight questions: (1) To avoid selection bias, was the strategy used for random sequence generation likely to produce comparable groups (e.g., random number table, computer random number generator)? (2) To avoid selection bias, was a method used to conceal the allocation sequence so that group allocation could not be foreseen in advance (e.g., sequentially numbered, opaque, sealed envelopes; central allocation office)? (3) To avoid detection bias, was an appropriate method/source used to collect objective (e.g., mortality) measures (e.g., death records, blinding of outcome assessor, trained chart abstracter)? (4) To avoid detection bias, was an appropriate method used to collect subjective (e.g., quality of life) measures (e.g., blinding of outcome assessor; use of reliable, valid, established self-administered questionnaires)? (5) To avoid attrition bias, was outcome data complete for the objective measures (i.e., complete for ≥80% of sample; missing data balanced between groups; missing data imputed using appropriate methods)? (6) To avoid attrition bias, was outcome data complete for the subjective measures (i.e., complete for ≥80% of sample; missing data balanced between groups; missing data imputed using appropriate methods)? (7) To avoid reporting bias, were all outcomes described in the methods section of the study reported in the results and were all key outcomes reported? (8) Were “other” biases detected in the study (e.g., contamination bias in which the control group had exposure to the intervention)?

We sought clarification from 40 of the 43 study authors when there were insufficient details in the paper to determine the risk of bias and we received 28 (70%) responses. An overall risk of bias was assigned to each study as follows: low risk of bias (at risk in 0-1 category), moderate risk of bias (at risk in 2-3 categories), high risk of bias (at risk in 4–6 categories), and very high risk of bias (at risk in 7-8 categories).

External validity refers to the generalization or applicability of the study to other circumstances [30]. To assess external validity, two team members independently assessed the generalizability of the study population, intervention, control, and outcomes (PICO). Disagreements were resolved through discussion and consensus. Historically, RCTs of NPs and CNSs have been criticized because the number evaluated in any study has been small (e.g., one or two NPs) causing concern that those willing to be evaluated may be atypical in training, experience, knowledge, skills, or practice characteristics. We consulted with our policy advisor and together decided that 10 NPs or CNSs either within a single study or across studies combined in meta-analyses would be a reasonable minimum sample necessary to generalize results to similar NP or CNS roles.

As reported in a separate paper, we applied the Quality of Health Economic Studies (QHES) instrument [3133] to evaluate the economic analyses in each study. The quality of the body of evidence for individual outcomes was evaluated using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system [34, 35] and GRADEpro software. The results of the GRADE assessments are reported elsewhere [36].

3.5. Data Extraction

A trained research assistant (KR) extracted data from each study into a summary table regarding general information (i.e., author, country, setting, language of publication, and publication status), characteristics of the study (design and group allocation), characteristics of the participants (number per group, sex, ages, and health conditions), characteristics of the intervention (number and type of NPs or CNSs, education and training, specific role, and comparison intervention), outcomes (health system, patient, and provider), length of follow-up, proportion followed to study completion, and study findings. If the findings of a single study were reported in two or more papers, they were extracted as one study. Team members checked the accuracy of extractions and discrepancies were resolved through discussion and consensus.

3.6. Analysis

Studies were categorized into the following six groupings: NP-outpatient, NP-transition, NP-inpatient, CNS-outpatient, CNS-transition, and CNS-inpatient. In a transition role, the NP or CNS could provide “time-limited services designed to ensure healthcare continuity, avoid preventable poor outcomes among at-risk populations, and promote the safe and timely transfer of patients from one level of care to another or from one type of setting to another” [37, page 747]. Within these groupings, studies were further categorized into alternative or complementary NP or CNS role function.

The strengths and threats to internal and external validity of the included RCTs are summarized narratively, organized by the six groupings identified above.

4. Results

4.1. Results of the Search

The searches yielded 4,397 unique records of which 3,981 were excluded during title and abstract review. Based on full-text review of the remaining 416 papers, 351 were excluded based on reasons listed in Figure 1. The remaining 65 papers described 43 relevant RCTs (28 studies reported in single papers and 15 studies reported in 37 papers). All studies were published in English. In general, the control intervention was “usual care.”

896587.fig.001
Figure 1: Identification and screening of relevant studies. Flow diagram adapted from Moher et al. [109].

The distribution of the 43 RCTs across groupings was NP-outpatient (), NP-transition (), NP-inpatient (), CNS-outpatient (), CNS-transition (), and CNS-inpatient (). We summarize the results by grouping beginning first with a brief overview of the study characteristics (Tables 1 and 2) followed by a description of threats to internal validity (Figures 2 and 3). Finally, threats to internal and external validity across studies will be described.

tab1
Table 1: Summary of NP study characteristics.
tab2
Table 2: Summary of CNS study characteristics.
896587.fig.002
Figure 2: Risk of bias assessment of NP studies ().
896587.fig.003
Figure 3: Risk of bias assessment of CNS studies ().
4.2. Study Characteristics and Internal Threats to Validity
4.2.1. NP-Outpatient Care

Eleven RCTs of NPs in outpatient care [3848] met our inclusion criteria (Table 1). All but one were published in the year 2000 or later. They were conducted in the United States (), United Kingdom (), or the Netherlands (). Six studies evaluated NPs in alternative provider roles and five in complementary provider roles. The number of NPs ranged from one to 20 in NP alternative provider studies and from one to four in NP complementary provider studies. Some of the trials were quite large with over 1000 patients, while most of the trials examining specific patient populations tended to be much smaller. The studies were conducted at between one and 20 sites.

Threats to Internal Validity. Overall, six of the 11 RCTs were judged to be at low risk of bias (in other words, the methods were of high quality), four at moderate, and one at high risk of bias (Figure 2). With regard to selection bias, nine studies used a random sequence generation process that was likely to produce comparable groups; for two studies, we had insufficient information about the sequence generation process to permit judgement, despite contact with one of the authors. Seven trials used an adequate process to conceal allocation so that participants and those enrolling participants could not foresee the group to which the next patient would be assigned. We judged one study as unclear because there was insufficient information and three at high risk of selection bias.

All the RCTs were judged to be at low risk of detection bias with respect to objective outcome measures (e.g., blood levels and medical records abstraction) and all but two trials were assessed at low risk of detection bias for subjective measures because most used established validated self-report instruments (e.g., SF-12 and SF-36). Three trials used blinded assessors for some data collection. Two studies were judged as unclear, one because they used self-reported dietary intake and physical activity which can be subject to recall and social desirability bias and the other because clinicians self-recorded the length of time they spent with each patient.

Seven RCTs were judged to be at low risk of attrition bias for the objective measures; one study reported a follow-up rate less than 80% for a blood cholesterol measure, and two did not report all follow-up rates. The risk of attrition bias for subjective measures was high or unclear for six studies due to failure to report follow-up rates or poor response rates for at least one self- or interviewer-administered questionnaire by last follow-up.

One study was judged at high risk of reporting bias because they did not report any patient outcomes such as child’s health status, quality of life, or parent satisfaction in a study of the appropriateness of follow-up care after attendance at an emergency department. We rated one study at high risk of “other” bias because there was substantial baseline imbalance which was not adjusted for in the analyses.

4.2.2. NP-Transition Care

Five RCTs evaluated NPs in a transition role [4953] (Table 1). Three studies were conducted in the US, one in Canada, and one in the UK. Four of the studies were published in the year 2000 or later. One study evaluated NPs in an alternative provider role and four in complementary provider roles. One or two NPs were evaluated in each study. The number of patients included in the trials ranged from 54 to 750 and they were conducted at between one and 10 sites.

Threats to Internal Validity. Overall, two studies were judged to be at low risk of bias and three at high risk of bias (Figure 2). The trials assessed to be at low risk of selection bias used a random number generator that revealed the intervention assignment when a patient was ready for allocation and a computer generated sequence concealed in sequentially numbered, opaque, sealed envelopes. The other three studies provided insufficient information to fully judge random sequence generation and allocation concealment.

All five RCTs were judged to be at low risk of detection bias for objective measures as they used abstraction of hospital administrative records or blinded outcome assessment. With respect to subjective measures, two trials were at high risk of detection bias because patients self-reported their smoking cessation success and the NP, who delivered the intervention, also collected baseline and outcome data from the comparison groups during a guided interview.

All but one trial were at low risk of attrition bias for objective measures as they followed over 80% of participants and this was balanced across comparison groups within each study. With respect to subjective data, three trials scored high for risk of attrition bias due to poor response rates to self- or interviewer-administered questionnaires.

Two studies identified outcomes that they planned to measure but did not report, placing them at risk of reporting bias.

4.2.3. NP-Inpatient Care

Two RCTs of NPs in inpatient settings met our inclusion criteria, both of which evaluated the NP in an alternative provider role [54, 55] (Table 1). One study was conducted in the US and one in Canada. One study was conducted before and one after the year 2000. The number of NPs in the trials ranged from 2.5 to 4.5 full-time equivalent NPs. The number of patients included in the trials ranged from 381 to 821 and each study was conducted at one site.

Threats to Internal Validity. Overall, the two studies were judged to be at low risk of bias (Figure 2). Both studies were at low risk of selection bias having used acceptable random sequence generation processes (table of random numbers; computer random number generator) and having concealed allocation through the use of sequentially numbered, sealed, opaque envelopes.

Both were at low risk of detection bias as they relied on medical record and hospital database extraction of objective data such as mortality, medical complications, and length of hospital stay. In cases where study participants completed questionnaires, there were reliable, valid measures such as the SF-36 and the Minnesota Infant Development Inventory (MIDI).

Both studies were judged to be at low risk of attrition bias for the objective measures but at high risk of attrition bias for the subjective measures. While many of the primary objective outcome data were available for all study participants (e.g., mortality, complications, and length of stay), subjective self-report measures often had response rates less than 80%. We judged both studies to be at low risk of reporting bias and other biases.

4.2.4. CNS-Outpatient Care

Eleven RCTs [5666] addressed the CNS role in delivering outpatient care (Table 2). Six studies were conducted in the US, two in the UK, two in the Netherlands, and one in China. Nine studies were published in the year 2000 or later. Four trials evaluated one to six CNSs in the alternative provider role, while seven trials evaluated one to nine CNSs in the complementary provider role. The number of patients included in the trials ranged from 20 to 643 and the studies were conducted at between one and six sites.

Threats to Internal Validity. Overall, five of the eleven studies were assessed at low risk, four at moderate risk, and two at high risk of bias (Figure 3). While seven studies used valid methods to generate the random sequence and were at low risk of selection bias, we judged the remaining four to be at unclear risk of bias because the authors did not include this information in their papers and we did not receive responses to our request for further details. With respect to allocation concealment, five trials were assessed at low risk of selection bias (e.g., central allocation and sealed envelopes) and three at unclear risk of bias because methods were not described. We judged one at high risk of bias because the patients were randomly assigned by the CNS to one of the study groups by drawing the next allocation from an envelope; using this method, it is possible that the drawn assignment could be returned to the envelope and redrawn if allocation was deemed unsuitable. Two studies used cluster randomization and allocation concealment was not applicable as the clusters were all randomized at one time.

All the studies were rated as low in risk of detection bias for objective outcome measures (e.g., mortality and rehospitalization). Of the nine studies that included subjective outcomes, eight were judged at low risk of bias as they used established, validated instruments, or blinded outcome assessment and one was at high risk of bias because the CNS who delivered the intervention also collected data from both groups before and after the intervention via telephone interviews.

With respect to attrition bias, all but three studies were assessed at low risk of bias for objective measures. Two of the studies judged at unclear risk of bias provided insufficient information to assess the completeness of all objective outcome measures and one, judged at high risk of bias, did not have cost data for at least 80% of the study participants.

While seven trials were judged to be at low risk of reporting bias, four were judged at high risk because they did not fully report all the outcomes they collected or did not collect all patient-important outcomes that would have been expected (e.g., patient/parent satisfaction with care and quality of life). Finally, one trial was judged at high risk of “other” bias because they did not adjust for cluster randomization.

4.2.5. CNS-Transition Care

Thirteen RCTs [6779] evaluated the CNS in the delivery of transition care in the US () and in the UK () (Table 2). Seven studies were conducted before the year 2000. All trials evaluated the CNS in a complementary provider role. The studies included between one and seven CNSs. The number of patients included in the trials ranged from 40 to 375 and the studies were conducted at between one and six sites.

Threats to Internal Validity. Overall, three of the thirteen trials were at low risk, eight at moderate risk, and two at a high risk of bias (Figure 3). Most trials were not at risk of selection bias. All but one trial used valid methods to generate the random sequence and all but one trial concealed allocation.

All trials were rated at low risk of detection bias for objective measures, except one. In this study, the risk of bias was unclear because healthcare utilization outcomes were based on self-report rather than medical record review data. For subjective measures, two trials were judged to be at unclear risk of detection bias because the validity of their scales was not described, and, in one trial, treatment adherence was based on self-report rather than objective measures, such as pill counts and was assessed at high risk of bias.

For objective measures, six trials had a low risk of attrition bias but, for five trials, the risk was unclear and, for two, it was high. For subjective measures, four trials had a low risk of attrition bias but, for six trials, the risk was unclear and, for three, it was high. For those studies in which it was unclear, the response rates were not specified or data were imputed; for those at high risk of bias, the follow-up rate was less than 80%.

Of the 13 trials, four were at high risk of reporting bias, three of which did not report on all outcomes measured and one of which did not include a measure of health status. One study was at unclear risk of bias because it was unclear if measures reported at baseline should have been reported as outcomes. Finally, seven trials were assessed at high risk of “other” bias because there were baseline differences between the groups for which adjustments were not made.

4.2.6. CNS-Inpatient Care

Only one study, conducted in the US in 1990, evaluated the CNS delivering inpatient care [80] (Table 2). The study examined CNSs in a complementary role. Two CNSs participated in the study, which included 107 patients and was conducted at one site.

Summary of Threats to Internal Validity. Overall, the risk of bias for this study was judged as moderate (Figure 3). We judged the study at low risk of selection bias and detection bias. The study, however, was judged to be at high risk of attrition bias because over 20% of patients were dropped from the study after randomization as the intervention they received was changed (e.g., sitters discontinued and control group receiving CNS consultation) resulting in unequal distribution of patients in the two groups.

The study was also at high risk of reporting bias because they did not report whether the CNS and staff nurse intervention influenced patient risk behaviours as intended. Contamination bias was possible because the same staff nurses who received coaching from the CNS for intervention group patient management and for charting nursing observations cared for the control group and might have provided the same patient management and charting strategies for them. Because the associated risk of bias was unknown, we judged this as unclear “other” bias.

4.2.7. Summary

Overall, we assessed that 18 of the 43 trials (42%) were at low risk, 17 (39%) at moderate risk, and 8 (19%) at high risk of overall bias (Figures 2 and 3). No study was judged to be at very high risk of overall bias. Figures 4 and 5 summarize the studies by type of bias. With respect to the NP trials, many studies were at high risk of detection bias with incomplete (<80%) follow-up for subjective outcomes (e.g., self-administered scales). In CNS trials, a number of studies were at high risk of reporting bias because they either did not report on all outcomes measured or did not include a key outcome that we would have expected. A number of studies (especially smaller studies) had baseline differences with no mention of adjusting the analyses to account for these differences.

896587.fig.004
Figure 4: Risk of bias horizontal graph of NP studies ().
896587.fig.005
Figure 5: Risk of bias horizontal graph of CNS studies ().

Some of the potential threats to validity may not in reality be threats, but rather it may be an issue of lack of reporting. There were many instances that we rated categories as “unclear risk of bias” because there was insufficient information in the paper or from the author to permit judgment of low or high risk of bias.

4.3. Summary of Threats to External Validity

Of the 43 RCTs, 70% of the studies were conducted in the United States () and the remainder in four other countries: the United Kingdom (; 14%), The Netherlands (; 9%), Canada (; 5%), and China (; 2%). Given that healthcare systems and NP and CNS education, role implementation, and scope of practice vary internationally, applicability of study findings from one country to another may be compromised.

Some RCTs evaluating NP and CNS roles were conducted across many sites which may enhance generalizability. However, many trials were conducted in single sites, which likely limits the generalizability of study findings.

Of the 43 RCTs, 13 (30%) studies were published prior to the year 2000. Given the substantive progress that has occurred in the development of NP and CNS roles and dynamic changes in healthcare systems internationally, the results of these studies may be less relevant to current-day policy. Although we found a substantial number of eligible RCTs, when broken down by grouping, we identified only one dated RCT of CNSs in the nontransitional care role for inpatient settings. This RCT evaluated two CNSs providing consultation for a small very particular population of medical-surgical patients requiring sitters due to the risk of self-harm or unpredictable behaviour. Similarly, we identified only two RCTs of NPs in the nontransitional care role in inpatient settings, both of which were published over 10 years ago. One study evaluated NPs caring for a homogeneous population of critically ill infants in a Canadian hospital and the other evaluated NPs caring for a heterogeneous population of adults admitted to general medical wards in a US-based hospital. Given the existence of only three fairly dated RCTs of NPs or CNSs in inpatient settings and somewhat specific populations, caution is needed in generalizing these results to NPs and CNSs in other inpatient settings.

Nine (21%) trials were conducted with small numbers of patients () with specific health conditions. Four trials of NPs in outpatient settings were large with over 1,000 patients. The larger studies with patients experiencing common conditions are more readily generalizable to the general population than smaller trials with patients experiencing a specific condition. However, one of the larger trials [44] limited study entry to poor, non-English speaking Hispanic people which may limit the generalizability of the findings to other patients seeking primary healthcare.

Twenty-seven (63%) of the RCTs evaluated one or two NPs or CNSs, 9 (21%) evaluated three to five, four (9%) evaluated six to nine, and three (7%) evaluated 10 or more all of which were NP-outpatient studies. The small number of NPs and CNSs evaluated in any study raises concern that the results may not be generalizable to colleagues in similar roles. In some cases when study outcomes were similar we were able to combine study findings which increased the number of NPs or CNSs evaluated for that outcome.

About two-thirds of the studies (; 67%) specified that they evaluated experienced NPs or CNSs (i.e., NPs or CNSs who had completed their training at least one year before the evaluation and/or had graduate degrees). Many of the studies did not include information about training and experience. One study posed concern, as it compared novice NPs who had completed a two-year advanced nursing practice graduate degree in the previous two months with general practitioners who had an average of 16 years work experience [39].

Most studies used reliable and valid outcome measures to evaluate patient-important outcomes such as health status, quality of life, and satisfaction with care which strengthens the generalizability of the findings; however, some studies had very short-term follow-up periods (e.g., two weeks after the patient appointment) which may compromise generalizability of study findings over the long term [39, 48, 60].

5. Discussion

The purpose of this paper was to report on the methodological strengths and threats to internal and external validity of RCTs of NP and CNS cost-effectiveness. Based on a comprehensive search of the international literature, we identified 43 RCTs, evaluating NPs () and CNSs (). While 43 RCTs sound like a large number of evaluations of NPs and CNSs, categorizing the studies by NP or CNS role (i.e., alternative or complementary) and by setting (i.e., outpatient, transition, or inpatient) reveals the areas where further research is still required. For example, we found only one RCT of the CNS in a nontransitional role in the inpatient setting and only two RCTs of the NP in a nontransitional role in the inpatient settings, both of which were alternative provider roles.

Of the 43 RCTs, 70% () were conducted in the United States with far fewer conducted in four other countries (Canada, China, The Netherlands, and United Kingdom). In 2011, Newhouse et al. conducted a systematic review of the effectiveness of NPs and CNSs [25]. They chose to restrict the review to studies conducted in the United States to enhance the applicability of study findings to the United States healthcare system. A recent systematic review that also includes studies conducted outside the United States has not been conducted, to our knowledge. Therefore, we chose to broaden our search to include international studies in order to learn more about where NP and CNS role evaluations have been conducted and how the roles are being enacted globally.

5.1. Internal Validity

Our assessment of the risk of bias revealed that about two-fifths (; 42%) of the 43 studies were at low risk of bias, close to the same number (; 39%) were at moderate risk of bias, and about one-fifth (; 19%) at high risk of bias. When examined by date, 31% of the 13 RCTs published before the year 2000 were at high risk of bias compared to 13% of the 30 RCTs published in or after the year 2000 that were at high risk of bias which may mean that study validity is improving over time.

In many cases it was unclear if the authors met the risk of bias criteria because the required information was not reported in the paper. Consequently, we rated a large number of categories as “unclear risk of bias.” To permit complete and accurate assessment of risk of bias, researchers are encouraged to use a guide such as the Cochrane risk of bias criteria [29] when planning and reporting future studies.

Researchers reporting RCTs may also find the following recommendations helpful. A clear brief description of the sequence generation (e.g., random number table; computer random number generator) is needed to allow the reader to determine if the process should provide comparable groups [29]. A description of allocation concealment (e.g., sequentially numbered, opaque, and sealed envelope) is important for the reader to determine if allocation to groups could be manipulated. While blinding of participants is not possible in a study incorporating NPs or CNSs, a description of procedures used to blind outcome assessors and/or the description of valid outcome measures is needed to assess the quality of the study. Completeness of outcome data for each outcome measure and group, including the description of missing data and details of all participants excluded, lost to follow-up (e.g., dropped out of study or died), or reincluded at each stage, also needs to be reported. If researchers do not report outcomes that were measured or key outcomes that would be expected, a clear description is needed of the reasons for failing to report the outcome. A description of how any “other” biases were managed that threaten the quality of the study should also be reported. More detailed recommendations for reporting RCTs can be found in the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement [81, 82]. When authors are faced with cutting back on the number of words in a publication, a suggestion is to reduce the introductory sections to provide sufficient space to describe in detail the strategies used to prevent or minimize threats to internal validity.

5.2. External Validity

As others have found [5, 24], a challenge in conducting this systematic review was determining the fidelity of the intervention. The definition of the role and the education, training, and experience of the NPs or CNSs were often inadequately described or missing. When we contacted authors for this information, we found that some studies were conducted with RNs who had received as little as a few weeks of training or one course and were then called “NPs.”

Over half (; 63%) of the 43 studies evaluated only one or two NPs or CNSs and only three trials, all of NPs in outpatient settings, evaluated 10 or more. Approximately two-thirds of the studies (; 67%) evaluated experienced NPs or CNSs. Researchers are encouraged to include a detailed description of the NPs or CNSs being evaluated in their study (role in the context of an internationally accepted definition [2], education, experience in the role, and training for the specific intervention if applicable). Similar information should be provided for comparison providers. Furthermore, evaluations of these roles should not be initiated while the NPs or CNSs are still novices but rather when they have had sufficient experience in their role (i.e., at least 12 months). Challenging as it is, researchers are encouraged to plan multisite studies, to increase the number of NPs or CNSs evaluated, to increase the number of patients enrolled in the study, and to account for variations in practice to enhance the generalizability of study findings.

5.3. Strengths and Limitations

Restricting this review to RCTs may be viewed as a strength or limitation, depending on the perspective of the reader. Health service settings are complex and research is confounded by multiple variables that challenge the ability to evaluate the effectiveness of an intervention, such as NP and CNS roles. When feasible, randomization of participants to intervention and control groups is considered the optimal design to control known and unknown complexities and confounding variables [8386]. Therefore, we chose to limit our review to RCTs. The quality of evidence in this review demonstrates that it is feasible to conduct well-designed RCTs to evaluate the effectiveness of NP and CNS roles in a variety of settings, remuneration mechanisms, and patient populations.

Strengths of our review include use of numerous strategies to identify all RCTs in any language (published or unpublished) that met our inclusion criteria, contact with authors and international expert advisors when it was unclear whether a study met our inclusion criteria, use of current education and credentialing criteria to verify that the trial was indeed evaluating an NP or CNS, use of duplicate assessment by independent reviewers and a consensus process for every stage of the review, use of an internationally recognized and established tool to assess the overall risk of bias of each trial and contact with authors when additional information was required to make our assessment, use of an established tool (Quality of Health Economic Studies) to evaluate the health economic analysis in each study, use of GRADE to evaluate outcome-specific quality of evidence, consideration of external as well as internal validity, grouping of trials by type (NP or CNS), setting (inpatient, transition, or outpatient), and role (alternative or complementary), and conducting meta-analyses whenever possible.

In future publications, we will summarize our assessment of the quality of the economic analyses of each RCT and outcome-specific quality of evidence using GRADE for each of the six groupings.

With respect to limitations, despite our attempts to identify all relevant RCTs, we may have missed some relevant studies or included some that do not meet our criteria based on author responses, advisor advice, or our interpretation of the description of the education or role. With respect to generalizability, we did not use a specific tool to assess threats to external validity but did consider the country and year of publication, number of NPs or CNSs in the study, the number of settings, and characteristics of the population, setting, intervention, and outcomes. We do not know how the exclusion of observational studies that investigate the effectiveness of NP and CNS roles may have influenced our findings [86].

6. Conclusions

This paper builds on the body of knowledge regarding quality of RCTs of NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization). We have used an international lens and inclusion criteria that meet today’s definitions of the NP and CNS roles. While almost half the RCTs were found to be at low risk of bias, incomplete reporting of study methods and lack of details about NP and CNS education, experience, and roles make it difficult to fully evaluate the internal and external validity of studies of these roles. Future studies that adhere to current standards for internal validity,such as Cochrane risk of bias [29], CONSORT [81, 82], and GRADE [34, 35], will contribute to a stronger body of evidence to address policy makers’ questions regarding the cost-effectiveness of NP and CNS roles.

Appendix

Electronic Database Search Strategies

Database: CINAHLData range: 1981 to July 31, 2012Results: 713# QueryS56 S53 and S55S55 S20 or S21 or S25S54 S26 and S53S53 S50 or S51 or S52S52 S47 or S48 or S49S51 S37 or S38 or S39 or S40 or S41 or S42 or S43 or S44 or S45 or S46S50 S27 or S28 or S29 or S30 or S31 or S32 or S33 or S34 or S35 or S36S49 placebo N4 controlS48 placebo N4 trialS47 placeboS46 (MH “Placebos”)S45 tripl N25 maskS44 tripl N25 blindS43 doubl N25 maskS42 doubl N25 blindS41 singl N25 maskS40 singl N25 blindS39 PT clinical trialS38 (MH “Double-Blind Studies”)S37 double-blind methodS36 single-blind methodS35 (MH “Single-Blind Studies”)S34 experimental trialS33 (MH “Clinical Trials+”)S32 controlled clinical trialS31 randomi?ed experimental trialS30 random allocatS29 (MH “Random Assignment”)S28 rctS27 randomi?ed controlled trialS26 S20 or S24 or S25S25 nurs ledS24 S11 or S12 or S13 or S16 or S17 or S18 or S19S23 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11 or S12 or S13 or S14or S15 or S16 or S17 or S18 or S19 or S20 or S21S22 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11 or S12 or S13 or S14 or S15 or S16 or S17 or S18 or S19 or S20 or S21S21 S11 or S12 or S16 or S17 or S18 or S19S20 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10S19 (MH “Nurse Anesthetists”)S18 nurse anesthetistS17 nurse anaesthetistS16 nurse clinicianS15 NPS14 npS13 “np” and nurseS12 CNS and nurseS11 specialist nurseS10 nurse specialistS9 nurse practitionerS8 (MH “Nurse Practitioners+”)S7 clinical nurse specialistS6 (MH “Clinical Nurse Specialists”)S5 apnS4 advanced practice nursS3 (MH “Advanced Practice Nurses+”)S2 advanced nursing practiceS1 (MH “Advanced Nursing Practice+”)

Database: EMBASEDate: 1980 to July 31, 2012Results: 15521. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (cns and nurs).mp.6. exp advanced practice nurse/7. nurse practitioner.mp.8. (np and nurse).mp.9. nurse specialist.mp.10. specialist nurse.mp.11. nurse clinician.mp.12. nurse an?esthetist.mp.13. nurs led.mp.14. or/1-1315. randomized controlled trial/16. randomi?ed controlled trial.mp.17. rct.mp.18. randomi?ed controlled trial.pt.19. randomized controlled trial.pt.20. RANDOMIZATION/21. random allocat.mp.22. randomi?ed experimental trial.mp.23. controlled clinical trial/24. controlled clinical trial.mp.25. clinical trial/26. clinical trial.mp.27. clinical trial.pt.28. experimental trial.mp.29. single blind procedure/30. double blind procedure/31. triple blind procedure/32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.33. (placebo or (placebo adj4 (trial or control))).mp.34. or/15-3335. 14 and 34

Database: Global HealthDate: 1973 to July 31, 2012Results: 381. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (cns and nurs).mp.6. advanced practice nurse/7. nurse practitioner.mp.8. (np and nurse).mp.9. nurse specialist.mp.10. specialist nurse.mp.11. nurse clinician.mp.12. nurse an?esthetist.mp.13. nurs led.mp.14. or/1-1315. randomized controlled trial/16. randomi?ed controlled trial.mp.17. rct.mp.18. randomi?ed controlled trial.pt.19. randomized controlled trial.pt.20. RANDOMIZATION/21. random allocat.mp.22. randomi?ed experimental trial.mp.23. controlled clinical trial/24. controlled clinical trial.mp.25. clinical trial/26. clinical trial.mp.27. clinical trial.pt.28. experimental trial.mp.29. single blind procedure/30. double blind procedure/31. triple blind procedure/32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.33. (placebo or (placebo adj4 (trial or control))).mp.34. or/15-3335. 14 and 34

Database: HealthStarDate range: 1966 to July 31, 2012Results: 11701. advanced nursing practice.mp.2. advanced practice nurs.mp.3. APN.mp.4. clinical nurse specialist.mp.5. (CNS and nurs).mp.6. nurse practitioners/7. nurse practitioner.mp.8. (NP and nurs).mp.9. nurse specialist.mp.10. specialist nurse.mp.11. nurse clinicians/12. nurse clinician.mp.13. nurse anesthetists/14. nurse anesthetist.mp.15. nurse anaesthetist.mp.16. nurs led.mp.17. or/1-1618. randomized controlled trials/19. randomi?ed controlled trial.mp.20. randomized controlled trial.pt.21. RCT.mp.22. random allocation/23. random allocat.mp.24. randomi?ed experimental trial.mp.25. controlled clinical trial.mp.26. controlled clinical trial.pt.27. randomized controlled trial.pt.28. controlled clinical trial.mp.29. experimental trial.mp.30. single-blind method/31. double-blind method/32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.33. (placebo or (placebo adj4 (trial or control))).mp.34. or/18-3335. 17 and 34

Database: MedlineDate range: 1950 to July 31, 2012Results: 13491. advanced nursing practice.mp.2. advanced practice nurs.mp.3. APN.mp.4. clinical nurse specialist.mp.5. (CNS and nurs).mp.6. nurse practitioners/7. nurse practitioner.mp.8. (NP and nurs).mp.9. nurse specialist.mp.10. specialist nurse.mp.11. nurse clinicians/12. nurse clinician.mp.13. nurse anesthetists/14. nurse anaesthetist.mp.15. nurse anesthetist.mp.16. nurs led.mp.17. or/1-1618. randomized controlled trial/19. randomi?ed controlled trial.mp.20. RCT.mp.21. randomized controlled trials as topic/22. randomized controlled trial.pt.23. random allocation/24. random allocat.mp.25. randomi?ed experimental trial.mp.26. controlled clinical trial/27. controlled clinical trial.mp.28. clinical trial.pt.29. clinical trial.mp.30. experimental trial.mp.31. single-blind method/32. double-blind method/33. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.34. (placebo or (placebo adj4 (trial or control)).mp.35. or/18-3436. 17 and 35

Database: AMEDDate range: all years to July 31, 2012Results: 301. advanced nursing practice.mp.2. advanced practice nurs.mp.3. APN.mp.4. clinical nurse specialist.mp.5. (CNS and nurse).mp.6. nurse practitioner.mp.7. (np and nurse).mp.8. nurse specialist.mp.9. specialist nurse.mp.10. nurse clinician.mp.11. nurse anesthetist.mp.12. nurse anaesthetist.mp.13. or/1-1214. randomized controlled trials/15. randomi?ed controlled trial.mp.16. RCT.mp.17. randomized controlled trial.pt.18. Random allocation/19. random allocat.mp.20. randomi?ation.mp.21. randomi?ed experimental trial.mp.22. controlled clinical trial.mp.23. Clinical trials/24. clinical trial.mp.25. clinical trial.pt.26. experimental trial.mp.27. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.28. (placebo or (placebo adj4 (trial or control))).mp.29. or/14-2830. 13 and 2931. nurs led.mp.32. 31 or 1333. 32 and 29

Database: CochraneDate range: all years to July 31, 2012Results: 1451. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (CNS and nurse).mp.6. nurse practitioner.mp.7. (np and nurse).mp.8. nurse specialist.mp.9. specialist nurse.mp.10. nurse clinician.mp.11. nurse anaesthetist.mp.12. nurse anesthetist.mp.13. or/1-1214. randomi?ed controlled trial.mp.15. RCT.mp.16. random allocat.mp.17. randomi?ed experimental trial.mp.18. controlled clinical trial.mp.19. clinical trial.mp.20. experimental trial.mp.21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.22. (placebo or (placebo adj4 (trial or control))).mp.23. or/14-2224. 13 and 2325. nurs led.mp.26. (13 or 25) and 23

Database: Cochrane CentralDate range: all years to July 31, 2012Results: 4741. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (CNS and nurse).mp.6. nurse practitioner.mp.7. (np and nurse).mp.8. nurse specialist.mp.9. specialist nurse.mp.10. nurse clinician.mp.11. nurse anaesthetist.mp.12. nurse anesthetist.mp.13. or/1-1214. randomi?ed controlled trial.mp.15. RCT.mp.16. random allocat.mp.17. randomi?ed experimental trial.mp.18. controlled clinical trial.mp.19. clinical trial.mp.20. experimental trial.mp.21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.22. (placebo or (placebo adj4 (trial or control))).mp.23. or/14-2224. 13 and 2325. nurs led.mp.26. (13 or 25) and 23

Database: DAREData range: all years to July 31, 2012Results: 851. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (CNS and nurse).mp.6. nurse practitioner.mp.7. (np and nurse).mp.8. nurse specialist.mp.9. specialist nurse.mp.10. nurse clinician.mp.11. nurse anaesthetist.mp.12. nurse anesthetist.mp.13. or/1-1214. randomi?ed controlled trial.mp.15. RCT.mp.16. random allocat.mp.17. randomi?ed experimental trial.mp.18. controlled clinical trial.mp.19. clinical trial.mp.20. experimental trial.mp.21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.22. (placebo or (placebo adj4 (trial or control))).mp.23. or/14-2224. 13 and 2325. nurs led.mp.26. (13 or 25) and 23

Database: HEEDData range: all years to July 31, 2012Results: 1081. advanced nursing practice.mp.2. advanced practice nurs.mp.3. apn.mp.4. clinical nurse specialist.mp.5. (CNS and nurse).mp.6. nurse practitioner.mp.7. (np and nurse).mp.8. nurse specialist.mp.9. specialist nurse.mp.10. nurse clinician.mp.11. nurse anaesthetist.mp.12. nurse anesthetist.mp.13. or/1-1214. randomi?ed controlled trial.mp.15. RCT.mp.16. random allocat.mp.17. randomi?ed experimental trial.mp.18. controlled clinical trial.mp.19. clinical trial.mp.20. experimental trial.mp.21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp.22. (placebo or (placebo adj4 (trial or control))).mp.23. or/14-2224. 13 and 2325. nurs led.mp.26. (13 or 25) and 23

Database: Web of ScienceIncludes: Science Citation Index, Social Sciences Index, Arts & Humanities Index, Conference Proceedings Citation Index (Science), Conference Proceedings Citation Index (Social Sciences and Humanities)Data range: all yearsResults: 1333#16  #15 AND #8#15  #14 OR #13 OR #12 OR #11 OR #9#14 TS = (placebo OR (placebo SAME (trial OR control)))#13  TS = ((singl or doubl or tripl or trebl) AND (blind OR mask))#12  TS = ((single or double or triple or trebl) SAME blind procedure)#11  TS = (clinical trial OR experimental trial)#10  TS = ((randomied controlled trial) OR rct)#9  TS = ((randomied controlled trial) OR rct OR randomiation OR random allocation OR randomied experimental trial)#8  #7 OR #6#7  TS =  (nurs lead)#6  #5 OR #4 OR #3 OR #2 OR #1#5  TS = (nurse anesthetist)#4  TS = (nurse practitioner)#3  TS = (nurse SAME specialist)#2  TS = ((apn or cns or np) AND nurse)#1  TS = (advanced SAME nurs SAME practice).

Conflict of Interests

The authors declare that there are no conflict of interests regarding the publication of this paper.

Acknowledgments

This systematic review was made possible through funding by the Ontario Ministry of Health and Long-Term Care, the Office of Nursing Policy in Health Canada, and the Canadian Foundation for Healthcare Improvement. The authors thank librarians, Laura Banfield and Jo-Anne Petropoulos, and staff, Julie Vohra, R. James McKinlay, Sara Kaffashian, and Rose Vonau. An international panel was very helpful in determining whether studies conducted outside of North America were evaluating advanced practice nursing roles. This panel included Melanie Rogers (UK), Petrie Roodbol (The Netherlands), Madrean Schober (Singapore), Barbara Sheer (USA), Helen Ward (UK), and Frances Wong (Hong Kong). The authors thank Brian Hutchison who provided input from a health policy perspective and Gordon Guyatt who provided methodological advice regarding the risk of bias. Special thanks go to our Advisory Board for their thoughtful feedback and suggestions: Theresa Agnew, Rachel Bard, Donna Denney, Barbara Foster, Elsabeth Jensen, Sandra MacDonald-Rencz, Lynne McVey, Robin Newhouse, and Linda Sawchenko. A special thank you goes to the many authors of papers included in our review for their research and for providing additional information. The views expressed in this paper are those of the authors and do not necessarily reflect those of the Ontario Ministry of Health and Long-Term Care, Office of Nursing Policy, Health Canada, or the Canadian Foundation for Healthcare Improvement.

References

  1. M. L. Delamaire and G. Lafortune, “Nurses in advanced roles: a description and evaluation of experiences in 12 developed countries,” OECD Health Working Paper 54, Organization for Economic Cooperation and Development, 2010, http://www.oecd-ilibrary.org/social-issues-migration-health/nurses-in-advanced-roles_5kmbrcfms5g7-en;jsessionid=20j38qefphtcs.x-oecd-live-01-01. View at Google Scholar
  2. International Council of Nurses, Nurse Practitioner/Advanced Practice Nurse: Definition and Characteristics, 2009, http://acnp.org.au/sites/default/files/33/definition_of_apn-np.pdf.
  3. F. Affara, ICN Framework of Competencies for the Nurse Specialist, International Council of Nurses, European Parkinson’s Disease Association, 2009, http://www.epda.eu.com/EasySiteWeb/GatewayLink.aspx?alId=7465.
  4. M. Laurant, M. Harmsen, H. Wollersheim, R. Grol, M. Faber, and B. Sibbald, “The impact of nonphysician clinicians: do they improve the quality and cost-effectiveness of health care services?” Medical Care Research and Review, vol. 66, no. 6, pp. 36S–89S, 2009. View at Publisher · View at Google Scholar · View at Scopus
  5. M. Laurant, D. Reeves, R. Hermens, J. Braspenning, R. Grol, and B. Sibbald, “Substitution of doctors by nurses in primary care,” Cochrane Database of Systematic Reviews, vol. 18, no. 2, Article ID CD001271, 2005. View at Google Scholar · View at Scopus
  6. G. M. Batchelor, W. O. Spitzer, A. E. Comley, and G. D. Anderson, “Nurse practitioners in primary care. IV. Impact of an interdisciplinary team on attitudes of a rural population,” Canadian Medical Association Journal, vol. 112, no. 12, pp. 1415–1420, 1975. View at Google Scholar · View at Scopus
  7. L. W. Chambers, P. Bruce-Lockhart, D. P. Black, E. Sampson, and M. Burke, “A controlled trial of the impact of the family practice nurse on volume, quality, and cost of rural health services,” Medical Care, vol. 15, no. 12, pp. 971–981, 1977. View at Publisher · View at Google Scholar · View at Scopus
  8. L. W. Chambers and A. E. West, “Assessment of the role of the family practice nurse in urban medical practices,” Canadian Journal of Public Health, vol. 69, no. 6, pp. 459–468, 1978. View at Google Scholar · View at Scopus
  9. L. W. Chambers and A. E. West, “The St John’s randomized trial of the family practice nurse: health outcomes of patients,” International Journal of Epidemiology, vol. 7, no. 2, pp. 153–161, 1978. View at Publisher · View at Google Scholar · View at Scopus
  10. D. L. Sackett, W. O. Spitzer, M. Gent, and R. S. Roberts, “The Burlington randomized trial of the nurse practitioner: health outcomes of patients,” Annals of Internal Medicine, vol. 80, no. 2, pp. 137–142, 1974. View at Publisher · View at Google Scholar · View at Scopus
  11. W. O. Spitzer, D. L. Sackett, J. C. Sibley et al., “The Burlington randomized trial of the nurse practitioner,” The New England Journal of Medicine, vol. 290, no. 5, pp. 251–256, 1974. View at Publisher · View at Google Scholar · View at Scopus
  12. W. O. Spitzer, D. J. Kergin, M. A. Yoshida, W. A. Russell, B. C. Hackett, and C. H. Goldsmith, “Nurse practitioners in primary care. III. The southern Ontario randomized trial,” Canadian Medical Association Journal, vol. 108, no. 8, p. 1005, 1973. View at Google Scholar · View at Scopus
  13. W. O. Spitzer, D. J. Kergin, M. A. Yoshida, W. A. Russell, B. C. Hackett, and C. H. Goldsmith, “Nurse practitioners in primary care: the southern Ontario randomized trial,” Health Care Dimensions, pp. 105–119, 1975. View at Google Scholar · View at Scopus
  14. W. O. Spitzer, R. S. Roberts, and T. Delmore, “Nurse practitioners in primary care. VI. Assessment of their deployment with the utilization and financial index,” Canadian Medical Association Journal, vol. 114, no. 12, pp. 1103–1108, 1976. View at Google Scholar · View at Scopus
  15. R. Burnip, R. Erickson, G. D. Barr, H. Shinefield, and E. J. Schoen, “Well-child care by pediatric nurse practitioners in a large group practice. A controlled study in 1,152 preschool children,” American Journal of Diseases of Children, vol. 130, no. 1, pp. 51–55, 1976. View at Publisher · View at Google Scholar · View at Scopus
  16. R. A. Hoekelman, “What constitutes adequate well baby care?” Pediatrics, vol. 55, no. 3, pp. 313–326, 1975. View at Google Scholar · View at Scopus
  17. K. Soghikian, “The role of nurse practitioners in hypertension care,” Clinical Science and Molecular Medicine Supplement, vol. 4, pp. 345s–358s, 1978. View at Google Scholar · View at Scopus
  18. L. W. Chambers, “Financial impact of family practice nurses on medical practice in Canada,” Inquiry, vol. 16, no. 4, pp. 339–349, 1979. View at Google Scholar · View at Scopus
  19. R. E. Lees, “Physician time-saving by employment of expanded-role nurses in family practice,” Canadian Medical Association Journal, vol. 108, no. 7, pp. 871–875, 1973. View at Google Scholar · View at Scopus
  20. W. O. Spitzer, W. A. Russell, and B. C. Hackett, “Financial consequences of employing a nurse practitioner,” Ontario Medical Review, vol. 29, pp. 96–100, 1973. View at Google Scholar
  21. U.S. Congress Office of Technology Assessment, Nurse Practitioners, Physician Assistants, and Certified Nurse-Midwives: A Policy Analysis, (Health Technology Case Study 37), OTA-HCS-37, Office of Technology Assessment, 1986, http://ota-cdn.fas.org/reports/8615.pdf.
  22. S. A. Brown and D. E. Grimes, “A meta-analysis of nurse practitioners and nurse midwives in primary care,” Nursing Research, vol. 44, no. 6, pp. 332–339, 1995. View at Google Scholar · View at Scopus
  23. M. Fry, “Literature review of the impact of nurse practitioners in critical care services,” Nursing in Critical Care, vol. 16, no. 2, pp. 58–66, 2011. View at Publisher · View at Google Scholar · View at Scopus
  24. S. Horrocks, E. Anderson, and C. Salisbury, “Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors,” British Medical Journal, vol. 324, no. 7341, pp. 819–823, 2002. View at Publisher · View at Google Scholar · View at Scopus
  25. R. P. Newhouse, J. Stanik-Hutt, K. M. White et al., “Advanced practice nurse outcomes 1990–2008: a systematic review,” Nursing Economics, vol. 29, no. 5, pp. 230–250, 2011. View at Google Scholar · View at Scopus
  26. M. W. Pozen, J. A. Stechmiller, W. Harris, S. Smith, D. D. Fried, and G. C. Voigt, “A nurse rehabilitator’s impact on patients with myocardial infarction,” Medical Care, vol. 15, no. 10, pp. 830–837, 1977. View at Publisher · View at Google Scholar · View at Scopus
  27. J. S. Fulton and K. Baldwin, “An annotated bibliography reflecting CNS practice and outcomes,” Clinical Nurse Specialist CNS, vol. 18, no. 1, pp. 21–39, 2004. View at Publisher · View at Google Scholar · View at Scopus
  28. F. Donald, D. Bryant-Lukosius, R. Martin-Misener et al., “Clinical nurse specialists and nurse practitioners: title confusion and lack of role clarity,” Nursing Leadership, vol. 23, special issue, pp. 189–201, 2010. View at Publisher · View at Google Scholar · View at Scopus
  29. J. P. T. Higgins and S. Green, Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0. [Updated], Cochrane, 2011, http://handbook.cochrane.org/.
  30. P. Jüni, D. G. Altman, and M. Egger, “Systematic reviews in health care: assessing the quality of controlled clinical trials,” The British Medical Journal, vol. 323, no. 7303, pp. 42–46, 2001. View at Publisher · View at Google Scholar · View at Scopus
  31. C. F. Chiou, J. W. Hay, J. F. Wallace et al., “Development and validation of a grading system for the quality of cost-effectiveness studies,” Medical Care, vol. 41, no. 1, pp. 32–44, 2003. View at Publisher · View at Google Scholar · View at Scopus
  32. J. J. Ofman, S. D. Sullivan, P. J. Neumann et al., “Examining the value and quality of health economic analyses: implications of utilizing the QHES,” Journal of Managed Care Pharmacy, vol. 9, no. 1, pp. 53–61, 2003. View at Google Scholar · View at Scopus
  33. L. E. Peterson, C. Goodman, E. K. Karnes, C. J. Chen, and J. A. Schwartz, “Assessment of the quality of cost analysis literature in physical therapy,” Physical Therapy, vol. 89, no. 8, pp. 733–755, 2009. View at Publisher · View at Google Scholar · View at Scopus
  34. G. Guyatt, A. D. Oxman, E. A. Akl et al., “GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings table,” Journal of Clinical Epidemiology, vol. 64, no. 4, pp. 383–394, 2011. View at Publisher · View at Google Scholar · View at Scopus
  35. G. H. Guyatt, A. D. Oxman, R. Kunz et al., “GRADE guidelines: 6. Rating the quality of evidence—imprecision,” Journal of Clinical Epidemiology, vol. 64, no. 12, pp. 1283–1293, 2011. View at Publisher · View at Google Scholar · View at Scopus
  36. F. Donald, A. DiCenso, D. Bryant-Lukosius et al., “A systematic review of the effectiveness and cost-effectiveness of nurse practitioners and clinical nurse specialists,” Final Report Grant 06514, Ontario Ministry of Health and Long-Term Care; The Office of Nursing Policy, Health Canada; and the Canadian Foundation for Healthcare Improvement, 2014. View at Google Scholar
  37. M. D. Naylor, L. H. Aiken, E. T. Kurtzman, D. M. Olds, and K. B. Hirschman, “The care span: the importance of transitional care in achieving health reform,” Health Affairs, vol. 30, no. 4, pp. 746–754, 2011. View at Publisher · View at Google Scholar · View at Scopus
  38. J. K. Allen, R. S. Blumenthal, S. Margolis, D. R. Young, E. R. Miller III, and K. Kelly, “Nurse case management of hypercholesterolemia in patients with coronary heart disease: Results of a randomized clinical trial,” American Heart Journal, vol. 144, no. 4, pp. 678–686, 2002. View at Publisher · View at Google Scholar · View at Scopus
  39. A. T. Dierick-van Daele, J. F. Metsemakers, E. W. Derckx, C. Spreeuwenberg, and H. J. M. Vrijhoef, “Nurse practitioners substituting for general practitioners: randomized controlled trial,” Journal of Advanced Nursing, vol. 65, no. 2, pp. 391–401, 2009. View at Publisher · View at Google Scholar · View at Scopus
  40. P. Kinnersley, E. Anderson, K. Parry et al., “Randomised controlled trial of nurse practitioner versus general practitioner care for patients requesting “same day” consultations in primary care,” The British Medical Journal, vol. 320, no. 7241, pp. 1043–1048, 2000. View at Publisher · View at Google Scholar · View at Scopus
  41. S. L. Krein, M. L. Klamerus, S. Vijan et al., “Case management for patients with poorly controlled diabetes: a randomized trial,” The American Journal of Medicine, vol. 116, no. 11, pp. 732–739, 2004. View at Publisher · View at Google Scholar · View at Scopus
  42. M. Limoges-Gonzalez, N. S. Mann, A. Al-Juburi, D. Tseng, J. Inadomi, and L. Rossaro, “Comparisons of screening colonoscopy performed by a nurse practitioner and gastroenterologists: a single-center randomized controlled trial,” Gastroenterology Nursing, vol. 34, no. 3, pp. 210–216, 2011. View at Publisher · View at Google Scholar · View at Scopus
  43. D. Litaker, L. C. Mion, L. Planavsky, C. Kippes, N. Mehta, and J. Frolkis, “Physician-nurse practitioner teams in chronic disease management: the impact on costs, clinical effectiveness, and patients’ perception of care,” Journal of Interprofessional Care, vol. 17, no. 3, pp. 223–237, 2003. View at Publisher · View at Google Scholar · View at Scopus
  44. M. O. Mundinger, R. L. Kane, E. R. Lenz et al., “Primary care outcomes in patients treated by nurse practitioners or physicians: a randomized trial,” The Journal of the American Medical Association, vol. 283, no. 1, pp. 59–68, 2000. View at Publisher · View at Google Scholar · View at Scopus
  45. E. W. Nelson, S. Van Cleve, M. K. Swartz, W. Kessen, and P. L. McCarthy, “Improving the use of early follow-up care after emergency department visits: a randomized trial,” American Journal of Diseases of Children, vol. 145, no. 4, pp. 440–444, 1991. View at Publisher · View at Google Scholar · View at Scopus
  46. M. L. A. Schuttelaar, K. M. Vermeulen, N. Drukker, and P. J. Coenraads, “A randomized controlled trial in children with eczema: nurse practitioner vs. dermatologist,” British Journal of Dermatology, vol. 162, no. 1, pp. 162–170, 2010. View at Publisher · View at Google Scholar · View at Scopus
  47. R. C. Smith, J. S. Lyles, J. C. Gardiner et al., “Primary care clinicians treat patients with medically unexplained symptoms: a randomized controlled trial,” Journal of General Internal Medicine, vol. 21, no. 7, pp. 671–677, 2006. View at Publisher · View at Google Scholar · View at Scopus
  48. P. Venning, A. Durie, M. Roland, C. Roberts, and B. Leese, “Randomised controlled trial comparing cost effectiveness of general practitioners and nurse practitioners in primary care,” British Medical Journal, vol. 320, no. 7241, pp. 1048–1053, 2000. View at Publisher · View at Google Scholar · View at Scopus
  49. E. A. Coleman, C. Parry, S. Chalmers, and S. J. Min, “The care transitions intervention: results of a randomized controlled trial,” Archives of Internal Medicine, vol. 166, no. 17, pp. 1822–1828, 2006. View at Publisher · View at Google Scholar
  50. A. Hollingsworth and S. Cohen, “Outcomes of early hospital discharge of women undergoing abdominal hysterectomy,” in Measuring Patient Outcomes, M. T. Nolan and V. Mock, Eds., pp. 155–167, Sage, London, UK, 2000. View at Google Scholar
  51. M. A. Kotowycz, T. L. Cosman, C. Tartaglia, R. Afzal, R. P. Syal, and M. K. Natarajan, “Safety and feasibility of early hospital discharge in ST-segment elevation myocardial infarction—a prospective and randomized trial in low-risk primary percutaneous coronary intervention patients (the Safe-Depart Trial),” American Heart Journal, vol. 159, no. 1, pp. 117.e1–117.e6, 2010. View at Publisher · View at Google Scholar · View at Scopus
  52. J. A. Nathan, L. Pearce, C. Field et al., “A randomized controlled trial of follow-up of patients discharged from the hospital following acute asthma: best performed by specialist nurse or doctor?” Chest, vol. 130, no. 1, pp. 51–57, 2006. View at Publisher · View at Google Scholar · View at Scopus
  53. S. M. Rawl, K. L. Easton, S. Kwiatkowski, D. Zemen, and B. Burczyk, “Effectiveness of a nurse-managed follow-up program for rehabilitation patients after discharge,” Rehabilitation Nursing Journal, vol. 23, no. 4, pp. 204–209, 1998. View at Publisher · View at Google Scholar · View at Scopus
  54. A. Mitchell-DiCenso, G. Guyatt, M. Marrin et al., “A controlled trial of nurse practitioners in neonatal intensive care,” Pediatrics, vol. 98, no. 6, part 1, pp. 1143–1148, 1996. View at Google Scholar · View at Scopus
  55. M. H. Pioro, C. S. Landefeld, P. F. Brennan et al., “Outcomes-based trial of an inpatient nurse practitioner service for general medical patients,” Journal of Evaluation in Clinical Practice, vol. 7, no. 1, pp. 21–33, 2001. View at Publisher · View at Google Scholar · View at Scopus
  56. J. S. Alexander, R. E. Younger, R. M. Cohen, and L. V. Crawford, “Effectiveness of a nurse-managed program for children with chronic asthma,” Journal of Pediatric Nursing, vol. 3, no. 5, pp. 312–317, 1988. View at Google Scholar · View at Scopus
  57. E. E. Arts, S. A. Landewe-Cleuren, N. C. Schaper, and H. J. Vrijhoef, “The cost-effectiveness of substituting physicians with diabetes nurse specialists: a randomized controlled trial with 2-year follow-up,” Journal of Advanced Nursing, vol. 68, no. 6, pp. 1224–1234, 2012. View at Publisher · View at Google Scholar · View at Scopus
  58. A. F. Brandon, J. B. Schuessler, K. J. Ellison, and R. B. Lazenby, “The effects of an advanced practice nurse led telephone intervention on outcomes of patients with heart failure,” Applied Nursing Research, vol. 22, no. 4, pp. e1–e7, 2009. View at Publisher · View at Google Scholar · View at Scopus
  59. D. Brooten, J. M. Youngblut, L. Brown, S. A. Finkler, D. F. Neff, and E. Madigan, “A randomized trial of nurse specialist home care for women with high-risk pregnancies: outcomes and costs. [Erratum appears in American Journal of Managed Care 2001 Sep;7(9):855],” The American Journal of Managed Care, vol. 7, no. 8, pp. 793–803, 2001. View at Google Scholar
  60. W. T. Chien, S. F. Leung, and C. Chu, “A nurse-led, needs-based psycho-education intervention for Chinese patients with first-onset mental illness,” Contemporary Nurse, vol. 40, no. 2, pp. 194–209, 2012. View at Publisher · View at Google Scholar · View at Scopus
  61. L. K. Evans, N. E. Strumpf, S. L. Allen-Taylor, E. Capezuti, G. Maislin, and B. Jacobsen, “A clinical trial to reduce restraints in nursing homes,” Journal of the American Geriatrics Society, vol. 45, no. 6, pp. 675–681, 1997. View at Google Scholar · View at Scopus
  62. S. Faithfull, J. Corner, L. Meyer, R. Huddart, and D. Dearnaley, “Evaluation of nurse-led follow up for patients undergoing pelvic radiotherapy,” British Journal of Cancer, vol. 85, no. 12, pp. 1853–1864, 2001. View at Publisher · View at Google Scholar · View at Scopus
  63. L. J. Ritz, M. J. Nissen, K. K. Swenson et al., “Effects of advanced nursing care on quality of life and cost outcomes of women diagnosed with breast cancer,” Oncology Nursing Forum, vol. 27, no. 6, pp. 923–932, 2000. View at Google Scholar · View at Scopus
  64. S. Ryan, A. B. Hassell, M. Lewis, and A. Farrell, “Impact of a rheumatology expert nurse on the wellbeing of patients attending a drug monitoring clinic,” Journal of Advanced Nursing, vol. 53, no. 3, pp. 277–286, 2006. View at Publisher · View at Google Scholar · View at Scopus
  65. R. W. Swindle, J. K. Rao, A. Helmy et al., “Integrating clinical nurse specialists into the treatment of primary care patients with depression,” International Journal of Psychiatry in Medicine, vol. 33, no. 1, pp. 17–37, 2003. View at Publisher · View at Google Scholar · View at Scopus
  66. G. J. Tijhuis, A. H. Zwinderman, J. M. W. Hazes, W. B. Van Den Hout, F. C. Breedveld, and T. P. M. Vliet Vlieland, “A randomized comparison of care provided by a clinical nurse specialist, an inpatient team, and a day patient team in rheumatoid arthritis,” Arthritis & Rheumatism, vol. 47, no. 5, pp. 525–531, 2002. View at Publisher · View at Google Scholar · View at Scopus
  67. D. Brooten, S. Kumar, L. P. Brown et al., “A randomized clinical trial of early hospital discharge and home follow-up of very-low-birth-weight infants,” The New England Journal of Medicine, vol. 315, no. 15, pp. 934–939, 1986. View at Publisher · View at Google Scholar · View at Scopus
  68. D. Brooten, M. Roncoli, S. Finkler, L. Arnold, A. Cohen, and M. Mennuti, “A randomized trial of early hospital discharge and home follow-up of women having cesarean birth,” Obstetrics and Gynecology, vol. 84, no. 5, pp. 832–838, 1994. View at Google Scholar · View at Scopus
  69. C. A. Dellasega and T. M. Zerbe, “A multimethod study of advanced practice nurse postdischarge care,” Clinical Excellence for Nurse Practitioners, vol. 4, no. 5, pp. 286–293, 2000. View at Google Scholar · View at Scopus
  70. L. Kennedy, S. Neidlinger, and K. Scroggins, “Effective comprehensive discharge planning for hospitalized elderly,” The Gerontologist, vol. 27, no. 5, pp. 577–580, 1987. View at Publisher · View at Google Scholar · View at Scopus
  71. A. S. Laramee, S. K. Levinsky, J. Sargent, R. Ross, and P. Callas, “Case management in a heterogeneous congestive heart failure population: a randomized controlled trial,” Archives of Internal Medicine, vol. 163, no. 7, pp. 809–817, 2003. View at Publisher · View at Google Scholar · View at Scopus
  72. R. McCorkle, N. E. Strumpf, I. F. Nuamah et al., “A specialized home care intervention improves survival among older post-surgical cancer patients,” Journal of the American Geriatrics Society, vol. 48, no. 12, pp. 1707–1713, 2000. View at Google Scholar · View at Scopus
  73. R. McCorkle, M. Dowd, E. Ercolano et al., “Effects of a nursing intervention on quality of life outcomes in post-surgical women with gynecological cancers,” Psycho-Oncology, vol. 18, no. 1, pp. 62–70, 2009. View at Publisher · View at Google Scholar · View at Scopus
  74. M. D. Naylor, “Comprehensive discharge planning for hospitalized elderly: a pilot study,” Nursing Research, vol. 39, no. 3, pp. 156–161, 1990. View at Google Scholar · View at Scopus
  75. M. Naylor, D. Brooten, R. Jones, R. Lavizzo-Mourey, M. Mezey, and M. Pauly, “Comprehensive discharge planning for the hospitalized elderly: a randomized clinical trial,” Annals of Internal Medicine, vol. 120, no. 12, pp. 999–1006, 1994. View at Publisher · View at Google Scholar · View at Scopus
  76. M. D. Naylor, D. Brooten, R. Campbell et al., “Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial,” Journal of the American Medical Association, vol. 281, no. 7, pp. 613–620, 1999. View at Publisher · View at Google Scholar · View at Scopus
  77. M. D. Naylor, D. A. Brooten, R. L. Campbell, G. Maislin, K. M. McCauley, and J. S. Schwartz, “Transitional care of older adults hospitalized with heart failure: a randomized, controlled trial,” Journal of the American Geriatrics Society, vol. 52, no. 5, pp. 675–684, 2004. View at Publisher · View at Google Scholar · View at Scopus
  78. D. R. Thompson, A. Roebuck, and S. Stewart, “Effects of a nurse-led, clinic and home-based intervention on recurrent hospital use in chronic heart failure,” European Journal of Heart Failure, vol. 7, no. 3, pp. 377–384, 2005. View at Publisher · View at Google Scholar · View at Scopus
  79. R. York, L. P. Brown, P. Samuels et al., “A randomized trial of early discharge and nurse specialist transitional follow-up care of high-risk childbearing women,” Nursing Research, vol. 46, no. 5, pp. 254–261, 1997. View at Publisher · View at Google Scholar · View at Scopus
  80. S. Talley, D. S. Davis, N. Goicoechea, L. Brown, and L. L. Barber, “Effect of psychiatric liaison nurse specialist consultation on the care of medical-surgical patients with sitters,” Archives of Psychiatric Nursing, vol. 4, no. 2, pp. 114–123, 1990. View at Publisher · View at Google Scholar · View at Scopus
  81. K. F. Schulz, D. G. Altman, and D. Moher, “CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials,” BMC Medicine, vol. 8, article 18, 2010. View at Publisher · View at Google Scholar · View at Scopus
  82. D. Moher, S. Hopewell, K. F. Schulz et al., “CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials,” Journal of Clinical Epidemiology, vol. 63, no. 8, pp. e1–e37, 2010. View at Google Scholar
  83. D. A. Richards and J. P. H. Hamers, “RCTs in complex nursing interventions and laboratory experimental studies,” International Journal of Nursing Studies, vol. 46, no. 4, pp. 588–592, 2009. View at Publisher · View at Google Scholar · View at Scopus
  84. J. J. Deeks, J. Dinnes, R. D’Amico et al., “Evaluating non-randomised intervention studies,” Health Technology Assessment, vol. 7, no. 27, pp. 1–173, 2003. View at Google Scholar · View at Scopus
  85. C. Y. Lu, “Observational studies: a review of study designs, challenges and strategies to reduce confounding,” International Journal of Clinical Practice, vol. 63, no. 5, pp. 691–697, 2009. View at Publisher · View at Google Scholar · View at Scopus
  86. J. Odgaard-Jensen, G. E. Vist, A. Timmer et al., Randomisation to Protect against Selection Bias in Healthcare Trials (Review), Cochrane Database of Systematic Reviews, 2011, http://apps.who.int/rhl/reviews/MR000012_standard.pdf.
  87. K. A. Paez and J. K. Allen, “Cost-effectiveness of nurse practitioner management of hypercholesterolemia following coronary revascularization,” Journal of the American Academy of Nurse Practitioners, vol. 18, no. 9, pp. 436–444, 2006. View at Publisher · View at Google Scholar · View at Scopus
  88. A. T. M. Dierick-van Daele, L. M. G. Steuten, J. F. M. Metsemakers, E. W. C. C. Derckx, C. Spreeuwenberg, and H. J. M. Vrijhoef, “Economic evaluation of nurse practitioners versus GPs in treating common conditions,” The British Journal of General Practice, vol. 60, no. 570, pp. e28–e35, 2010. View at Publisher · View at Google Scholar · View at Scopus
  89. E. R. Lenz, M. O. Mundinger, S. C. Hopkins, S. X. Lin, and J. L. Smolowitz, “Diabetes care processes and outcomes in patients treated by nurse practitioners or physicians,” Diabetes Educator, vol. 28, no. 4, pp. 590–598, 2002. View at Publisher · View at Google Scholar · View at Scopus
  90. E. R. Lenz, M. O. Mundinger, R. L. Kane, S. C. Hopkins, and S. X. Lin, “Primary care outcomes in patients treated by nurse practitioners or physicians: two-year follow-up,” Medical Care Research and Review, vol. 61, no. 3, pp. 332–351, 2004. View at Publisher · View at Google Scholar · View at Scopus
  91. M. L. Schuttelaar, K. M. Vermeulen, and P. J. Coenraads, “Costs and cost-effectiveness analysis of treatment in children with eczema by nurse practitioner vs. dermatologist: results of a randomized, controlled trial and a review of international costs,” British Journal of Dermatology, vol. 165, no. 3, pp. 600–611, 2011. View at Publisher · View at Google Scholar · View at Scopus
  92. J. S. Lyles, A. Hodges, C. Collins et al., “Using nurse practitioners to implement an intervention in primary care for high-utilizing patients with medically unexplained symptoms,” General Hospital Psychiatry, vol. 25, no. 2, pp. 63–73, 2003. View at Publisher · View at Google Scholar · View at Scopus
  93. Z. Luo, J. Goddeeris, J. C. Gardiner, and R. C. Smith, “Costs of an intervention for primary care patients with medically unexplained symptoms: a randomized controlled trial,” Psychiatric Services, vol. 58, no. 8, pp. 1079–1086, 2007. View at Publisher · View at Google Scholar · View at Scopus
  94. C. Parry, E. A. Coleman, J. D. Smith, J. Frank, and A. M. Kramer, “The care transitions intervention: a patient-centered approach to ensuring effective transfers between sites of geriatric care,” Home Health Care Services Quarterly, vol. 22, no. 3, pp. 1–17, 2003. View at Google Scholar · View at Scopus
  95. K. L. Easton, S. M. Rawl, and D. Zemen, “The effects of nursing follow-up on the coping strategies used by rehabilitation patients after discharge,” Rehabilitation Nursing Research, vol. 4, no. 4, pp. 119–127, 1995. View at Google Scholar
  96. N. E. Strumpf, L. K. Evans, J. Wagner, and J. Patterson, “Reducing physical restraints: developing an educational program,” Journal of Gerontological Nursing, vol. 18, no. 11, pp. 21–27, 1992. View at Google Scholar · View at Scopus
  97. J. E. Patterson, N. E. Strumpf, and L. K. Evans, “Nursing consultation to reduce restraints in a nursing home,” Clinical Nurse Specialist, vol. 9, no. 4, pp. 231–235, 1995. View at Publisher · View at Google Scholar · View at Scopus
  98. E. L. Siegler, E. Capezuti, G. Maislin, M. Baumgarten, L. Evans, and N. Strumpf, “Effects of a restraint reduction intervention and OBRA ’87 regulations on psychoactive drug use in nursing homes,” Journal of the American Geriatrics Society, vol. 45, no. 7, pp. 791–796, 1997. View at Google Scholar · View at Scopus
  99. E. Capezuti, N. E. Strumpf, L. K. Evans, J. A. Grisso, and G. Maislin, “The relationship between physical restraint removal and falls and injuries among nursing home residents,” Journals of Gerontology Series A: Biological Sciences and Medical Sciences, vol. 53, no. 1, pp. M47–M52, 1998. View at Google Scholar · View at Scopus
  100. G. J. Tijhuis, K. G. Kooiman, A. H. Zwinderman, J. M. W. Hazes, F. C. Breedveld, and T. P. M. Vliet Vlieland, “Validation of a novel satisfaction questionnaire for patients with rheumatoid arthritis receiving outpatient clinical nurse specialist care, inpatient care, or day patient team care,” Arthritis & Rheumatism, vol. 49, no. 2, pp. 193–199, 2003. View at Publisher · View at Google Scholar
  101. G. J. Tijhuis, A. H. Zwinderman, J. M. W. Hazes, F. C. Breedveld, and P. M. T. Vliet Vlieland, “Two-year follow-up of a randomized controlled trial of a clinical nurse specialist intervention, inpatient, and day patient team care in rheumatoid arthritis,” Journal of Advanced Nursing, vol. 41, no. 1, pp. 34–43, 2003. View at Publisher · View at Google Scholar · View at Scopus
  102. W. B. van den Hout, G. J. Tijhuis, J. M. Hazes, F. C. Breedveld, and T. P. M. Vliet Vlieland, “Cost effectiveness and cost utility analysis of multidisciplinary care in patients with rheumatoid arthritis: a randomised comparison of clinical nurse specialist care, inpatient team care, and day patient team care,” Annals of the Rheumatic Diseases, vol. 62, no. 4, pp. 308–315, 2003. View at Publisher · View at Google Scholar · View at Scopus
  103. C. Dellasega and T. M. Zerbe, “Caregivers of frail rural older adults. Effects of an advanced practice nursing intervention,” Journal of Gerontological Nursing, vol. 28, no. 10, pp. 40–49, 2002. View at Google Scholar · View at Scopus
  104. S. H. Neidlinger, K. Scroggins, and L. M. Kennedy, “Cost evaluation of discharge planning for hospitalized elderly,” Nursing Economics, vol. 5, no. 5, pp. 225–230, 1987. View at Google Scholar · View at Scopus
  105. C. Jepson, R. McCorkle, D. Adler, I. Nuamah, and E. Lusk, “Effects of home care on caregivers’ psychosocial status,” Journal of Nursing Scholarship, vol. 31, no. 2, pp. 115–120, 1999. View at Google Scholar · View at Scopus
  106. R. McCorkle, S. Jeon, E. Ercolano, and P. Schwartz, “Healthcare utilization in women after abdominal surgery for ovarian cancer,” Nursing Research, vol. 60, no. 1, pp. 47–57, 2011. View at Publisher · View at Google Scholar · View at Scopus
  107. M. D. Naylor and K. M. McCauley, “The effects of a discharge planning and home follow-up intervention on elders hospitalized with common medical and surgical cardiac conditions,” The Journal of Cardiovascular Nursing, vol. 14, no. 1, pp. 44–54, 1999. View at Publisher · View at Google Scholar · View at Scopus
  108. K. M. McCauley, M. B. Bixby, and M. D. Naylor, “Advanced practice nurse strategies to improve outcomes and reduce cost in elders with heart failure,” Disease Management, vol. 9, no. 5, pp. 302–310, 2006. View at Publisher · View at Google Scholar · View at Scopus
  109. D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and PRISMA group, “Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement,” British Medical Journal, vol. 339, p. b2535, 2009. View at Publisher · View at Google Scholar · View at Scopus