About this Journal Submit a Manuscript Table of Contents
ISRN Nursing
Volume 2013 (2013), Article ID 706054, 12 pages
http://dx.doi.org/10.1155/2013/706054
Research Article

Implementation Interventions Used in Nursing Homes and Hospitals: A Descriptive, Comparative Study between Austria, Germany, and The Netherlands

1Institute of Nursing Science, Medical University of Graz, Billrothgaße 6, 8010 Graz, Austria
2Department of Health Services Research Focusing on Chronic Care and Ageing, Section of Nursing Science, Maastricht University, P.O. Box 616, 6200 MD Maastricht, The Netherlands
3Department of Nursing and Management, Faculty of Business & Social Sciences, Hamburg University of Applied Sciences, Alexanderstraße 1, 20099 Hamburg, Germany
4Department of Nursing and Health Sciences, Protestant University of Applied Sciences, Zweifalltorweg 12, 64293 Darmstadt, Germany

Received 10 May 2013; Accepted 23 June 2013

Academic Editors: K. Clark, A. Green, B. Mandleco, K. G. Schepp, and D. Whitehead

Copyright © 2013 Helga E. Breimaier et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Translating guidelines into nursing practice remains a considerable challenge. Until now, little attention has been paid to which interventions are used in practice to implement guidelines on changing clinical nursing practice. This cross-sectional study determined the current ranges and rates of implementation-related interventions in Austria, Germany, and The Netherlands and explored possible differences between these countries. An online questionnaire based on the conceptual framework of implementation interventions (professional, organizational, financial, and regulatory) from the Cochrane Effective Practice and Organization of Care (EPOC) data collection checklist was used to gather data from nursing homes and hospitals. Provision of written materials is the most frequently used professional implementation intervention (85%), whereas changes in the patient record system rank foremost among organisational interventions (78%). Financial incentives for nurses are rarely used. More interventions were used in Austria and Germany than in The Netherlands (20.3/20.2/17.3). Professional interventions are used more frequently in Germany and financial interventions more frequently in The Netherlands. Implementation efforts focus mainly on professional and organisational interventions. Nurse managers and other responsible personnel should direct their focus to a broader array of implementation interventions using the four different categories of EPOC’s conceptual framework.

1. Introduction

Nurses are expected to deliver care that is regularly updated with research findings, and the volume of new scientific evidence for good clinical practice is growing quickly. This research-based knowledge should be used, for example, as a basis for decision making [1] and for providing evidence-based care to promote positive patient outcomes [2, 3]. Legal regulations commit governments to providing their citizens with healthcare based on the acknowledged state of scientific knowledge; for example, in Austria, there is the Health Care Quality Act (Section 3) [4]. Regulations also require nurses to update their knowledge and skills according to the latest scientific findings in nursing science. In Austria, this is governed by the Nursing Act (Section 4, paragraphs 1 and 2) [5]. However, numerous studies over the past decade have highlighted a failure to routinely translate research findings into daily practice [611]. A large gap between what is known from research and what happens in practice still exists [10, 12], and introducing change is far from straightforward [13]. As a consequence of this theory-practice gap, patients may suffer unnecessarily from inconveniences such as pain, pressure ulcers, and/or prolonged hospitalization [11, 14].

Much effort has been undertaken to implement new knowledge to change nursing practice. Clinical practice guidelines are seen as one means by which new knowledge can be instilled in practice [15] in order to change the process and the outcome of care provided by allied medical professionals [16]. However, knowledge distillation and the publication of research findings or guidelines do not ensure that practitioners will use them in their daily clinical practice [17, 18]. Translating research into healthcare decision making and practice remains a considerable challenge [19]. There are several reasons why research results are not implemented successfully in clinical practice, including, for example, characteristics of innovations [20], insufficient education/information, or inadequate organisational support [21]. Over the past decade, a growing number of articles have discussed interventions that implement new knowledge in healthcare [2226]. This is especially true for medical but also, to an increasing extent, for nursing care. These articles show, among other things, why the effectiveness of implementation interventions varies in healthcare [2730]. Alanen et al. revealed, for example, that the method of implementing a hypertension guideline in Finnish health centres varied widely. Health centres categorized as implementers used multiple strategies to promote the adoption of the guideline. Health centres categorised as disseminators did little to facilitate their adoption [31]. Earlier research disclosed that methods utilized to implement guidelines in primary healthcare were usually directive and passive [32]. According to Kortteisto et al., profession also has an effect on the intention to use clinical guidelines in patient care, and the authors recommended the use of different strategies when clinical guidelines are targeted at different professional groups [33].

There is a broad array of implementation interventions from which nurse managers, for example, may choose to implement new knowledge in their respective practical fields. The Cochrane Effective Practice and Organisation of Care Review Group compiled a checklist with a short description of each intervention, which was then grouped into four non-mutually exclusive categories: professional interventions (e.g., distribution of educational materials), organisational interventions (e.g., changes in medical records systems), financial interventions (e.g., fee-for-service), and regulatory interventions (e.g., changes in medical liability) [34, 35]. These implementation interventions are thought to facilitate the use of known knowledge and help to overcome barriers to its adoption in clinical settings [34]. In a systematic review, Medves et al. focussed on dissemination and implementation strategies of practice guidelines for healthcare teams and team-based practice. They identified 88 relevant studies that investigated 16 dissemination and implementation strategies. Four of these strategies are ranked as organisational interventions, one as patient-oriented, and one as structural interventions. Ten of these strategies were aimed at healthcare professionals and included the distribution of educational materials and educational meetings as the most common strategies [36]. In an explorative study, Meesterberends et al. found that interventions used in disseminating pressure-ulcer guidelines in nursing homes of six European countries differed in terms of number (2–4) and type. The most common implementation interventions were the use of the Internet and the presentation of the guideline at national or regional seminars or congresses [37]. According to the typology of EPOC, both interventions belong to the category of professional interventions [35]. However, the evidence supporting the use of the electronic retrieval of healthcare information (including the internet) by healthcare providers to improve practice and patient care was found to be insufficient [38]. Furthermore, attendance at seminars or congresses is a passive educational approach which is considered a less effective implementation intervention [39].

This indicates that interventions used are not in line with existing evidence regarding implementation and may be one of the reasons why innovations are not implemented sustainably. To resolve this inconsistency, the effectiveness of implementation interventions aimed at instilling new knowledge into nursing practice must be understood. To this end, this study sets out to discover which implementation strategies are actually used in nursing practice and whether the most frequently used implementation strategies are indeed effective ones. This can help nurse managers to find the best methods for implementing new knowledge and/or guidelines. To date, however, only little is known about which interventions are actively used to implement guidelines in nursing—be it in hospitals or in nursing homes.

To shed more light on this issue, an international research group from Austria, Germany and The Netherlands set itself the objective of (1) determining the current range and rates of the interventions used for implementation purposes regarding guidelines in nursing in these countries and (2) exploring whether there are any differences in the interventions employed between the three countries. For this purpose, four care problems were chosen on an exemplary basis—discharge management (in hospitals), urinary incontinence (in nursing homes), and malnutrition and pressure ulcers (in both settings)—as these are among the more common care problems in hospitals and nursing homes [4042].

2. Methods

2.1. Design

Using a cross-sectional design, data were gathered from nursing homes and hospitals in Austria (A), Germany (D), and The Netherlands (NL).

2.2. Sample

Austrian, German, and Dutch hospitals and nursing homes were invited to participate in the study via email. In Austria, all nursing homes (692) and hospitals (268) nationwide were asked to participate. In Germany, for purposes of convenience, the survey was limited to the two federal states of Berlin and Hesse but included all nursing homes (Berlin: 228 and Hesse: 908) and hospitals (47 and 166, resp.). In The Netherlands, all 210 nursing homes and 70 hospitals which participated in the yearly National Prevalence Measurement of Care Problems (Landelijke Prevalentiemeting Zorgproblemen, LPZ) were invited to participate. The ethical principles of good scientific practice according to the Declaration of Helsinki [43] were observed throughout the entire research process. In a cover letter, participating institutions were informed about the study. Anonymity and confidentiality were assured, and voluntary participation was guaranteed. Contact information for further questions was made available for all three institutions carrying out this project. Approval for filling out the questionnaire was given by the participating institutions. Consent was assumed on the basis of a returned and completed questionnaire.

2.3. Procedure and Data Collection

The management and nursing directors of nursing homes and hospitals received an email notification about the intention to investigate interventions used to implement guidelines in nursing practice. The contact details of the institutions in Austria were acquired through an extensive internet search and from lists of nursing homes published by the Federal Ministry of Labour, Social Affairs and Consumer Protection. Contact details in Germany were acquired from lists published by the Ministry of Health, by the Public Inspecting Authorities for Nursing Homes of Hesse, and by searching the internet. In The Netherlands, all participants of the National Prevalence Measurement of Care Problems (LPZ) [41] were contacted. Two weeks later, all managers and nursing directors received detailed information about the purpose of the proposed study, the researchers’ intention to disseminate the results, the data collection procedure, the return of the questionnaire, and the time requirements (45 minutes) for completing the questionnaire. A link to the online questionnaire was attached. The email contact and collection of the completed questionnaires were carried out by an external organisation (Flycatcher, Internet Research, Maastricht). Two reminders were sent to all managers and nursing directors, and the deadline was extended twice to increase the response rate. After completing the data collection, the anonymized final data file was forwarded to authors 1, 2, and 3 for analysis. Thus, anonymity was ensured for all participants. The managers and nursing directors were asked to forward the questionnaire to the nurse responsible for implementing guidelines in their institution and to return it by June 2008.

2.3.1. Instrument

For comparison purposes, the modified typology of implementation interventions of the Cochrane Effective Practice and Organisation of Care (EPOC) Review Group [34, 35] was used for this study. The questionnaire included a total of four sets of questions based on the conceptual framework of four implementation interventions, namely, professional interventions, organisational interventions, financial interventions, and regulatory interventions [34, 35]. For each category of intervention, single interventions varying from bottom-up approaches (through participation) to top-down approaches (through instructions) were presented. Each intervention could be answered with yes, no, or I do not know. All questions were asked in relation to specific clinical domains. They were chosen on the basis of their prevalence in the three countries and the specific setting. In nursing homes, all questions relating to prevention of pressure ulcers, urinary incontinence, and malnutrition were asked. In hospitals, the clinical domain of urinary incontinence is less prevalent [44, 45] and was arbitrarily replaced by discharge management; its importance for hospitals has been pointed out previously [4648]. The questionnaire consisted of 22 pages all together.

2.3.2. Translation

The intervention part of the English EPOC-Checklist [35] was translated into German by authors 3, 5, and 6. Based on this translation, a German version of the instrument, addressing implementation issues of particular concern to nursing, was developed. This version was revised for Austria in order to take into account the localized nuances of Austrian German. In The Netherlands, the intervention part of the English EPOC-Checklist [35] was translated into Dutch by authors 2 and 4. The Dutch version was then translated into German by a professional translator. Both German versions were cross-checked, and unclear items were discussed within the research group.

2.3.3. Content Validation

The questionnaire was developed by the researchers, and a content validation procedure was performed in all three countries. Three experts in the field (nursing and medicine) from each country were asked to indicate whether the included items were relevant to implementation issues in hospitals and nursing homes. Possible answers ranged from agree completely, partly agree/partly disagree, to disagree completely. The instrument was amended using the feedback from the content validation procedure. In this course, the existence of nursing guidelines and expert standards was highlighted in the questionnaire, and some items concerning financial interventions were modified or removed completely.

2.3.4. Pretest

After the process of content validation, an electronic web-based version of the questionnaire was developed. The questionnaire was piloted in three institutions per country to ensure that it was practical as well as easy to understand and read. The comments received were used to write the final electronic version of the questionnaire.

2.4. Analysis

Data were analyzed with SPSS 17.0. The percentage of each intervention was calculated by adding the answers of both the settings and the three care problems per intervention and dividing the result by the total number of institutions and then by three (care problems). The mean total number of each individual category of interventions and of the total interventions per setting and per care problem was calculated by adding the percentages of the respective individual items and dividing the result by 100. Next, the mean number of interventions of each category of intervention was calculated for each clinical domain and country. Differences between the three countries regarding the implementation interventions used were tested separately for each setting with one-way ANOVA. Due to the numerous tests, a value of 0.01 was used for significance. The basis for the analysis was the net-response rates with .

3. Results

The response rates varied from 12.8% to 55.7%. In all countries, the response rate of hospitals was higher than nursing homes (between 8.3% and 28.6%). However, not all questionnaires were fully completed, so the net response of completed questionnaires ( ) was lower (8.7%–47.1%).

3.1. Implementation Interventions

In the three countries, various interventions were used during implementation, with a mean total of 19.6 interventions per setting per care problem. An overview of the interventions is shown in Table 1. The percentages presented are mean percentages calculated over the three care problems and over the institutions, which means that, for example, in Austria, professional internal written materials are used in 90% of the cases when implementing a guideline. The most frequently used implementation intervention is from the category of professional interventions, namely, providing written materials. On average this is used in 85% of cases. Other commonly used professional interventions (63%–66%) are internal consultations, internal audits and internal lectures. The organisational interventions used mostly (65%–78%) are changes in the patient record system, individual patient participation in treatment decisions, efforts to increase nurses’ work satisfaction, regular use of assessment instruments, changes in structure and facilities of the institution, and management of patient complaints.

tab1
Table 1: Percentage of implementation interventions used per setting per care problem in Austria, Germany and the Netherlands.

External electronic training and the use of telemedicine are the professional and organisational interventions which are least used (≤12%). Financial interventions for nurses are rarely used (10%), while an average of 1.3 regulatory interventions is used per setting per care problem.

3.2. Implementation Interventions per Country

When looking at the mean total of interventions used per country, it becomes apparent that settings in Austria and Germany used slightly more interventions than those in The Netherlands (resp., 20.3, 20.2, and 17.3; ) (Table 1). This is also true for the mean total of professional (A: 10.6, D: 10.3, NL: 8.2; ) and organisational (A: 8.5, D: 8.3, NL: 7.0; ) interventions. Regulatory interventions are, however, used less in Austria (A: 1.0, D: 1.4, NL: 1.5; ).

The most frequently used professional interventions with significant differences between the countries are internal audits, internal lectures, regular team meetings about a clinical topic, local opinion leaders, and written/manual reminders (Table 1). Internal written materials, internal consultations and external lectures are used less in The Netherlands. The least frequently used professional intervention with a significant difference is external conferences. With regard to organisational interventions, the results show differences between the regular use of assessment instruments, efforts to improve nurses’ work satisfaction, case management, changes in right to access documentation, and changes in number of staff. Compared to Austria and Germany, financial incentives for nurses are used more frequently in The Netherlands. Within the regulatory interventions, inspecting authorities and insurance-related regulations differ significantly (Table 1).

The use of internal written materials ranks among the most frequently used professional interventions in all three countries, followed by internal audits and internal lectures in Germany, and internal consultations, external conferences as well as internal audits and internal lectures in Austria. In The Netherlands, the use of internal written materials is preceded by regular team meetings about a clinical topic and manual reminders. The three countries differ regarding the rank order of the most frequently used organisational interventions: in Austria these are efforts to increase nurses’ work satisfaction, changes in patient record systems and individual patient participation in treatment decisions. In Germany the regular use of assessment instruments ranks first, whereas in The Netherlands additional staff qualifications is the third most frequent organisational intervention. The most often used regulatory intervention in Austria and The Netherlands is inspecting authorities, whereas in Germany insurance-related regulations is used most often (Table 1). An overview of the five most frequently used implementation interventions per country is provided in Table 2. In Austria and Germany, these are two professional and three organisational interventions. In The Netherlands, these are one regulatory, two professional, and two organisational interventions.

tab2
Table 2: Overview of the five most used implementation interventions per country.
3.3. Implementation Interventions per Care Problem and Clinical Setting

These results are presented for each individual category of intervention.

3.3.1. Professional Interventions

The number of professional interventions used in nursing homes and hospitals does not differ between the countries regarding the care problems of pressure ulcers and malnutrition (Table 3). However, there are significant differences regarding the care problems of incontinence in nursing homes and discharge management in hospitals. In both cases, the number of professional interventions is less in The Netherlands. Regarding incontinence, an average of 7.4 professional interventions was used to implement guidelines in Dutch nursing homes compared to 10.9 in Austria and 10.1 in Germany ( ). In terms of discharge management, an average of 2.1 professional interventions was used to implement guidelines in hospitals in The Netherlands compared to 9.6 in Austria and 9.3 in Germany ( ).

tab3
Table 3: Mean number of professional, organisational, financial and regulatory implementation interventions.
3.3.2. Organisational Interventions

The evaluation of the organisational interventions yields a similar picture regarding differences between the settings and care problems in the three countries (Table 3). Regarding incontinence, an average of 6.3 organisational interventions were used to implement guidelines in Dutch nursing homes compared to 8.6 in Austria and 8.1 in Germany ( ). Regarding discharge management, an average of 3.2 organisational interventions was used to implement guidelines in hospitals in The Netherlands compared to 8.2 in Austria and 7.9 in Germany ( ).

3.3.3. Financial Intervention

Financial incentives for the implementation of guidelines regarding care problems are available in only a few organizations (Table 3). In Dutch nursing homes they are used in total significantly more often (19.7%) than in Austrian (6.1%) and German (5.1%; ) ones. This also holds true for the care problem of pressure ulcers (28.2%/6.5%/7.1%; ). In hospitals, financial interventions are used less in Austria (6.7%) compared to Germany (17.0%) and The Netherlands (23.3%; ) with the exception of discharge management ( ).

3.3.4. Regulatory Interventions

The mean number of regulatory interventions is significantly lower in Austrian nursing homes (1.1) than in German (1.5) and Dutch ones (1.8; ) (Table 3). The same is true for the individual care problem of pressure ulcers (1.2/1.5/1.9; ) and malnutrition (1.0/1.6/1.8; ). There is no significant difference between hospitals in the three countries, neither in total nor for the individual care problems.

4. Discussion

Until now, “studies on the implementation of evidence based practice in nursing care make a very small contribution to the overall evidence on effective implementation” [49, p. 1163]. This study sought to gain an insight into ranges and rates of interventions used to implement nursing guidelines in Austria, Germany, and The Netherlands.

4.1. Ranges and Rates of Implementation Interventions

The results describing implementation interventions revealed that, on average, a broad array of implementation interventions was used. The most frequent ones were found within professional and organisational strategies. Financial and regulatory interventions were rarely used. This is in line with findings from the literature [37, 50] and, although indirectly, comparable with findings of a systematic review about the effectiveness of the dissemination and implementation interventions of practice guidelines for (multidisciplinary) healthcare teams. In this review, Medves et al. revealed that of the 16 interventions (classified according to the EPOC taxonomy), ten were professional and five were organisational/structural interventions [36]. Robertson and Jochelson conclude from their review that there is less research into organisational-level interventions than those targeted at individuals [51].

Among the five most frequently used interventions, the distribution of written materials as a professional intervention ranks first. This may be due to the advantage that printed educational materials can be distributed relatively cheaply and easily to a large number of professionals. Furthermore, its dissemination aims at improving the awareness, knowledge, attitudes, skills, and professional practice of nurses as well as patient outcomes [52]. This intervention is also among the five most frequently used interventions of a study investigating described implementation strategies in published abstracts [50], and it is the most commonly investigated one [36]. Although the majority of reviewed studies reported significant findings, it was not possible to attribute this intervention to the described effect. Grimshaw et al. concluded from their review that the distribution of written materials is a less effective method and that the addition of educational materials to other interventions did not seem to increase effectiveness [27]. This is supported by Farmer et al., who found a slightly beneficial effect on process outcomes but not on patient outcomes [52].

Four organisational interventions are ranked next: changes in the patient record system, individual patient participation in treatment decisions, efforts to increase nurses’ work satisfaction, and regular use of assessment instruments. These measures do not force nurses to actively react to recommendations of an implemented guideline. Nonetheless, these measures seem to be a necessary basis which makes their use possible, in that, for example, it should be possible to record recommended care once it is carried out. Work satisfaction, which is associated with autonomy, good collaboration between nurses and physicians along with reduced job stress [53], was found to be an important individual characteristic of research utilization [54]. Out of these four most often used organisational interventions, changes in the patient record system was the only one included in Medves et al.’s work [36]. No conclusions could be drawn as to their effectiveness. In a former systematic review, however, Urquhart et al. found some limited evidence of effects on practice attributable to changes in nursing record systems [55]. A systematic review of reviews about organisational interventions found limited evidence in organisational interventions. None of the interventions produced consistent effects [56].

Additional frequently used interventions directed to professionals include internal consultations, internal audits and internal lectures. In these internally directed interventions, nursing managers may see a possible advantage in tailoring the specific content to the necessities of the respective setting in which they are delivered and/or a cost effective intervention possibility to introduce change. Often, audits are found to be investigated together with feedback as an implementation intervention. In general, their effect is consistently described as small to moderate for health practitioners [27, 28, 51, 5760], yet not mandatory [61], and their effect could be increased where baseline adherence to recommended practice is low and when feedback is delivered more intensively [28]. An investigation by Dulko et al. in acute nursing care also showed their effectiveness [25].

Internal lectures are part of educational interventions which are usually regarded to be of limited [62] or no effect when used as a single intervention [39, 51, 57]. Simpson and Doig unexpectedly found that in-servicing (didactic lectures) was ranked highly, that is, as an effective practice change intervention, by participants [63]. In general, it can be said that educational meetings which include courses, conferences, lectures, workshops, seminars, and symposia can have a small effect on improving professional practice and healthcare outcomes [27, 39]; however, interactive educational strategies such as small-scale interactive meetings and workshops appear to be more successful in changing practice [51] and were consistently reported as effective [57, 64]. Yet, workshops, on average, were offered in less than half of the participating settings. Studies which reveal the effects of different educational implementation interventions showed only modest or moderate effects on processes or outcomes of care [27, 29, 39].

In general, financial and regulatory interventions were rarely used. Financial interventions, especially financial incentives, are seldom used to implement innovations in hospitals and nursing homes. The industrial sector, on the other hand, makes widespread use of financial incentives to encourage employees to achieve certain goals in their individual fields of responsibility. In the healthcare sector, however, this seems to be a rare measure to foster and implement quality improvement policies—at least in nursing. Actually, the use of financial incentives seems almost impossible considering that a separate budget for innovation implementation, which might be used for providing financial incentives, is very rare in the healthcare sector [16, 37]. Nevertheless, the effect of financial incentives was found to be inconclusive and provided no evidence that the magnitude of the incentive influenced compliance [57]. Flodgren et al. concluded that financial incentives may be effective in changing healthcare professional practice but not in improving patient outcomes [65]. Based on sparse observational studies found regarding regulatory interventions, Robertson and Jochelson summarized that regulation is associated with improvements in healthcare quality [51].

4.2. Comparison between the Three Countries

A count of the amount of implementation interventions shows that more interventions were used in Austria and Germany than in The Netherlands. However, when evaluating implementation interventions, limiting the perspective to mere numbers is not necessarily useful. Studies have revealed no relationship between the number of component interventions and the effects of multifaceted interventions on improvements in care [27]. It is still not clear how many and which combination of strategies are required to be successful [36].

For this reason, the authors of this study have focused their evaluation on the most common implementation interventions. In Austria and Germany, the five implementation interventions used most frequently are part of professional and organisational interventions with a ratio of 2 : 3. In The Netherlands they are part of professional, organizational, and regulatory interventions with a ratio of 3 : 1 : 1. The configuration of the used categories of implementation interventions shows a broader approach of implementation interventions in Dutch nursing homes and hospitals than in Germany and Austria. Moreover, in The Netherlands, interventions are also used which seem to focus on the ward-level, such as regular team meetings about a clinical topic and written/manual reminders, in addition to interventions applied on an institutional level. This might address nurses more directly and force them to react in a short time frame. According to Prior et al., the use of reminders consistently resulted in significant practical improvements, and computer-delivered reminders had a slightly greater effect than manual- or paper-based reminders [57]. Furthermore, in all but one of the five most used interventions per country, there was a significant difference between the countries. This exception is individual patient participation in treatment decisions, although ranked differently in the top-five list. Robertson and Jochelson found some evidence that providing educational materials to patients can help in the implementation of guidelines [51]. Inspecting authorities was among the five most used interventions in Dutch settings and is significantly more applied there than in Austrian and German settings. Inspection in its function of external oversight refers to surveillance and enforcement to ensure minimum standards are met [51]. The differences between the countries regarding these top-five interventions might be due to a higher awareness of the relevance of the process and steps of implementation in Dutch hospitals and nursing homes than in German and Austrian settings because of the longer history of nursing science in The Netherlands. Furthermore, The Netherlands started to develop national guidelines, for example, for pressure-ulcer prevention and treatment, as early as 1985 [66], whereas, in Germany, this process was started in 1999 with the development of expert standards (a type of guideline) [67], and education as the recommended implementation intervention [68]. In Austria, there exists no equivalent institution on a national level which is responsible for the development of national guidelines for relevant care problems in nursing. The difference regarding inspecting authorities might be due to the fact that, in The Netherlands, the Health Care Inspectorate regularly controls whether healthcare institutions meet their standards. Dutch healthcare settings are obliged to collect data about the so-called “norms of safe care”, for example, in the form of the annual prevalence survey, which was launched in 1998 [69].

4.3. Implementation Interventions per Care Problem

With regard to the amount of used professional and organisational interventions, no differences were found for pressure ulcers and malnutrition in the three countries. A possible explanation may be that these problems occur to a similar extent and that the amount of intervention reflects the magnitude of the care problems. This applies to malnutrition where prevalence rates are similarly high in nursing homes and hospitals: 25.7%/25.1% (A), 20.2%/22.8% (NL), and 23.0% in German nursing homes. In contrast, the prevalence rates of pressure ulcers are found to be highest in Dutch hospitals (10.2%) and lowest in German nursing homes (4.0%) [45]. In the field of incontinence care in nursing homes and discharge management in hospitals, there was a remarkably lower amount of professional and organisational implementation interventions in The Netherlands. It might be that these topics are regarded as less essential in Dutch nursing care. Pressure ulcers, malnutrition, and incontinence seem to be equally important areas in nursing care in Austria and Germany, and this might be due to their epidemiological relevance. Furthermore, since 2008 it has been mandatory for German nursing homes to implement what are known as expert standards [70], a kind of guideline focusing, among other things, on these care problems. Therefore, the high number of implementation interventions used in relation to those problems seems reasonable. Financial and regulatory interventions do not seem to play an important role regarding specific care problems, although they are used more frequently in The Netherlands relating to pressure ulcers.

Finally one should consider that the majority of the cited literature did not focus on the nursing profession but on healthcare professionals and medical doctors. Yet, interventions to implement guidelines should be tailored to different groups of stakeholders [15]. For this reason we should know the effectiveness of the used interventions.

4.4. Limitations

The results of this study might be limited due to low response rates—especially from nursing homes, which made comparison between hospitals and nursing homes difficult. Although low response rates are rather common nowadays [71, 72], several aspects could have had a negative influence, such as the use of a web-based questionnaire. Even though the questionnaire was developed in a very user-friendly way and tested for applicability, it could have been a problem for some respondents to fill in a web-based questionnaire. Also, the persons contacted might have considered the completion time for this task to be too long. Proposed participants were, however, notified beforehand of the forthcoming study. One further reason for low response rates may be that the questionnaire was distributed by a Dutch company, thus influencing possible participants from Austria and Germany. Furthermore, a high percentage of questionnaires was not fully completed; that is, the respondents stopped after the first few questions. In those cases, we decided to only use completely filled-out questionnaires.

Another limitation might lie in the fact that neither the characteristics of the person who filled out the questionnaire nor their role in the nursing team of each respective institution was identified. It is, therefore, possible that nurses in different positions with a variety of views about and responsibilities regarding implementation interventions filled in the questionnaire and this may, in turn, have influenced the results.

5. Conclusions

This study suggests that the most widely used implementation interventions to translate guidelines into nursing practice focus on professionals and organisations. There are only a few marked differences between the three countries, a fact which may be attributed to different familiarity with the field of nursing science and a different significance of problems in nursing care. After investigating the effectiveness of the most frequently used single interventions in the literature, it can be concluded that these are either of little or modest to moderate effect or that their effectiveness still lacks evidence. Furthermore, one should consider that the majority of the cited literature did not focus on the nursing profession.

Nurses responsible for implementing nursing guidelines in practice should not focus mainly on a few implementation interventions with modest or even unknown effectiveness. On the contrary, at the beginning of an implementation process it might be worth considering what exactly has to be achieved and, based on this, selecting appropriate implementation strategies from all four different implementation categories. Yet to date, it is unknown how many implementation interventions, in which combination and from which EPOC-framework category [34, 35], would be effective in implementing a guideline into nursing practice. More research is necessary to explore whether a wide variety of implementation interventions is more effective than focussing on professional and organisational interventions alone.

Acknowledgments

Upon request, a list of all nursing homes in Hesse was kindly made available by the Public Inspecting Authorities for Nursing Homes (Heimaufsicht) of the Regional Administrative Authority of the Federal State of Hesse in Gießen (Regierungspräsidium in Gießen). The Regional Administrative Authority in Gießen also very kindly supported the survey in Hesse by conveying greetings to all nursing homes as part of the introduction to the electronic questionnaire encouraging nursing homes to participate in the study.

References

  1. D. J. McCloskey, “Nurses' perceptions of research utilization in a corporate health care system,” Journal of Nursing Scholarship, vol. 40, no. 1, pp. 39–45, 2008. View at Publisher · View at Google Scholar · View at Scopus
  2. P. Hart, L. A. Eaton, M. Buckner et al., “Effectiveness of a computer-based educational program on nurses' knowledge, attitude, and skill level related to evidence-based practice,” Worldviews on Evidence-Based Nursing, vol. 5, no. 2, pp. 75–84, 2008. View at Publisher · View at Google Scholar · View at Scopus
  3. M. Kirshbaum, “Translation to practice: a randomised, controlled study of an evidence-based booklet for breast-care nurses in the United Kingdom,” Worldviews on Evidence-Based Nursing, vol. 5, no. 2, pp. 60–74, 2008. View at Publisher · View at Google Scholar · View at Scopus
  4. “Federal Act on the Quality of Health Services (Health-Care Quality Act—Gesundheitsqualitätsgesetz—GQG),” Excerpt from the 2005 Health Reform Act, Fed. Law Gaz. I no. 179/2004, 2004, http://www.bmg.gv.at/cms/home/attachments/3/4/1/CH1063/CMS1043931577060/health_care_quality_act.pdf.
  5. G. Flemmich and W. Nöstlinger, Eds., GesUndheits-Und Krankenpflegegesetz, Gesetze und Kommentare NR 180, Verlag des Österreichischen Gewerkschaftsbundes GmbH, Vienna, Austria, 2004.
  6. R. Grol, “Successes and failures in the implementation of evidence-based guidelines for clinical practice,” Medical Care, vol. 39, no. 8, pp. II46–II54, 2001. View at Scopus
  7. R. Grol and J. Grimshaw, “From best evidence to best practice: effective implementation of change in patients' care,” The Lancet, vol. 362, no. 9391, pp. 1225–1230, 2003. View at Publisher · View at Google Scholar · View at Scopus
  8. J. Grimshaw, M. Eccles, and J. Tetroe, “Implementing clinical guidelines: current evidence and future implications,” The Journal of Continuing Education in the Health Professions, vol. 24, supplement 1, pp. S31–S37, 2004. View at Scopus
  9. D. S. Thompson, C. A. Estabrooks, S. Scott-Findlay, K. Moore, and L. Wallin, “Interventions aimed at increasing research use in nursing: a systematic review,” Implementation Science, vol. 2, article 1, 2007. View at Publisher · View at Google Scholar · View at Scopus
  10. T. van Achterberg, L. Schoonhoven, and R. Grol, “Nursing implementation science: how evidence-based nursing requires evidence-based implementation,” Journal of Nursing Scholarship, vol. 40, no. 4, pp. 302–310, 2008. View at Publisher · View at Google Scholar · View at Scopus
  11. E. Meesterberends, R. J. G. Halfens, C. Lohrmann, J. M. G. A. Schols, and R. de Wit, “Evaluation of the dissemination and implementation of pressure ulcer guidelines in Dutch nursing homes,” Journal of Evaluation in Clinical Practice, vol. 17, no. 4, pp. 705–712, 2011. View at Publisher · View at Google Scholar · View at Scopus
  12. M. L. Koehn and K. Lehman, “Nurses' perceptions of evidence-based nursing practice,” Journal of Advanced Nursing, vol. 62, no. 2, pp. 209–215, 2008. View at Publisher · View at Google Scholar · View at Scopus
  13. K. Gerrish, “Evidence-based practice,” in The Research Process in Nursing, K. Gerrish and A. Lacey, Eds., pp. 491–505, Blackwell Publishing, Oxford, UK, 5th edition, 2007.
  14. L. Wallin, “Knowledge translation and implementation research in nursing,” International Journal of Nursing Studies, vol. 46, no. 4, pp. 576–587, 2009. View at Publisher · View at Google Scholar · View at Scopus
  15. J. Ploeg, B. Davies, N. Edwards, W. Gifford, and P. E. Miller, “Factors influencing best-practice guideline implementation: lessons learned from administrators, nursing staff, and project leaders,” Worldviews on Evidence-Based Nursing, vol. 4, no. 4, pp. 210–219, 2007. View at Publisher · View at Google Scholar · View at Scopus
  16. L. H. Thomas, N. A. Cullum, E. McColl, N. Rousseau, J. Soutter, and N. Steen, “Guidelines in professions allied to medicine,” Cochrane Database of Systematic Reviews, no. 2, article CD000349, 2000. View at Scopus
  17. S. Campbell, H. Hancock, and H. Lloyd, “Implementing evidence-based practice,” in The Research Process in Nursing, K. Gerrish and A. Lacey, Eds., pp. 506–520, Blackwell Publishing, Oxford, UK, 5th edition, 2007.
  18. S. E. Straus, J. Tetroe, and I. Graham, “Defining knowledge translation,” The Canadian Medical Association Journal, vol. 181, no. 3-4, pp. 165–168, 2009. View at Publisher · View at Google Scholar · View at Scopus
  19. K. Seers, K. Cox, N. J. Crichton et al., “FIRE (Facilitating Implementation of Research Evidence): a study protocol,” Implementation Science, vol. 7, article 25, 2012. View at Publisher · View at Google Scholar · View at Scopus
  20. R. Grol and M. Wensing, “Characteristics of successful innovations,” in Improving Patient Care. The Implementation of Change in Clinical Practice, R. Grol, M. Wensing, and M. Eccles, Eds., pp. 60–70, Elsevier, Edinburgh, UK, 2005.
  21. H. E. Breimaier, R. J. G. Halfens, and C. Lohrmann, “Nurses' wishes, knowledge, attitudes and perceived barriers on implementing research findings into practice among graduate nurses in Austria,” Journal of Clinical Nursing, vol. 20, no. 11-12, pp. 1744–1756, 2011. View at Publisher · View at Google Scholar · View at Scopus
  22. M. Eccles, E. McColl, N. Steen et al., “Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial,” The British Medical Journal, vol. 325, no. 7370, pp. 941–944, 2002. View at Scopus
  23. J. Rashotte, M. Thomas, D. Grégoire, and S. Ledoux, “Implementation of a two-part unit-based multiple intervention: moving evidence-based practice into action,” The Canadian Journal of Nursing Research, vol. 40, no. 2, pp. 94–114, 2008. View at Scopus
  24. M. Dobbins, P. Robeson, D. Ciliska et al., “A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies,” Implementation Science, vol. 4, article 23, 2009. View at Publisher · View at Google Scholar · View at Scopus
  25. D. Dulko, E. Hertz, J. Julien, S. Beck, and K. Mooney, “Implementation of cancer pain guidelines by acute care nurse practitioners using an audit and feedback strategy,” Journal of the American Academy of Nurse Practitioners, vol. 22, no. 1, pp. 45–55, 2010. View at Publisher · View at Google Scholar · View at Scopus
  26. E. Fineout-Overholt, K. M. Williamson, B. Kent, and A. M. Hutchinson, “Teaching EBP: strategies for achieving sustainable organizational change toward evidence-based practice,” Worldviews on Evidence-Based Nursing, vol. 7, no. 1, pp. 51–53, 2010. View at Publisher · View at Google Scholar · View at Scopus
  27. J. M. Grimshaw, R. E. Thomas, G. MacLennan et al., “Effectiveness and efficiency of guideline dissemination and implementation strategies,” Health Technology Assessment, vol. 8, no. 6, iii–iv, pp. 1–72, 2004. View at Scopus
  28. G. Jamtvedt, J. M. Young, D. T. Kristoffersen, M. A. O'Brien, and A. D. Oxman, “Audit and feedback: effects on professional practice and health care outcomes,” Cochrane Dtabase of Systematic Reviews, vol. 2, article CD000259, 2006. View at Scopus
  29. M. A. O'Brien, S. Rogers, G. Jamtvedt et al., “Educational outreach visits: effects on professional practice and health care outcomes,” Cochrane Database of Systematic Reviews, vol. 4, article CD000409, 2007. View at Scopus
  30. K. G. Shojania, A. Jennings, A. Mayhew, C. R. Ramsay, M. P. Eccles, and J. Grimshaw, “The effects of on-screen, point of care computer reminders on processes and outcomes of care,” Cochrane Database of Systematic Reviews, vol. 3, article CD001096, 2009. View at Scopus
  31. S. I. Alanen, R. Johannala-Kemppainen, J. J. Ijäs et al., “Evaluation of current care effectiveness: a survey of hypertension guideline implementation in Finnish health centres,” Scandinavian Journal of Primary Health Care, vol. 25, no. 4, pp. 232–236, 2007. View at Publisher · View at Google Scholar · View at Scopus
  32. S. Miilunpalo, E. Toropainen, and P. Moisio, “Implementation of guidelines in primary health care: a challenge for the municipal health centres in Finland,” Scandinavian Journal of Primary Health Care, vol. 19, no. 4, pp. 227–231, 2001. View at Publisher · View at Google Scholar · View at Scopus
  33. T. Kortteisto, M. Kaila, J. Komulainen, T. Mäntyranta, and P. Rissanen, “Healthcare professionals' intentions to use clinical guidelines: a survey using the theory of planned behaviour,” Implementation Science, vol. 5, article 51, 2010. View at Publisher · View at Google Scholar · View at Scopus
  34. T. Thorsen and M. Mäkelä, “A framework for guidelines implementation studies,” in Changing Professional Practice. Theory and Practice of Clinical Guidelines Implementation, T. Thorsen and M. Mäkelä, Eds., pp. 23–53, Danish Institute for Health Services Research, Copenhagen, Denmark, 1999.
  35. Cochrane Effective Practice and Organisation of Care Review Group (EPOC), “Data collection checklist,” 2002, http://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/datacollectionchecklist.pdf.
  36. J. Medves, C. Godfrey, C. Turner et al., “Systematic review of practice guideline dissemination and implementation strategies for healthcare teams and team-based practice,” International Journal of Evidence-Based Healthcare, vol. 8, no. 2, pp. 79–89, 2010. View at Scopus
  37. E. Meesterberends, R. Halfens, C. Lohrmann, and R. de Wit, “Pressure ulcer guideline development and dissemination in Europe,” Journal of Clinical Nursing, vol. 19, no. 11-12, pp. 1495–1503, 2010. View at Publisher · View at Google Scholar · View at Scopus
  38. J. L. McGowan, R. Grad, P. Pluye et al., “Electronic retrieval of health information by healthcare providers to improve practice and patient care,” Cochrane Database of Systematic Reviews, vol. 3, article CD004749, 2009. View at Scopus
  39. L. Forsetlund, A. Bjørndal, A. Rashidian et al., “Continuing education meetings and workshops: effects on professional practice and health care outcomes,” Cochrane Database of Systematic Reviews, vol. 2, CD003030, 2009. View at Scopus
  40. T. Dassen, Ed., Bundesweite Erhebung Zu Pflegeproblemen 2009, Studie aus Pflegeheimen und Krankenhäusern, Charité-Universitätsmedizin Berlin, Centrum 1 Human- und Gesundheitswissenschaften, Institut für Medizin-, Pflegepädagogik und Pflegewissenschaft, Berlin, Germany, 2009.
  41. R. J. G. Halfens, J. M. M. Meijers, J. C. L. Neyens, and J. M. G. A. Schols, “Rapportage resultaten. Landelijke Prevalentiemeting Zorgproblemen 2009,” Report of the National Prevalence Measurement, Universiteit Maastricht, Onderzoeksinstituut Caphri, Department of Health Care and Nursing Science, Maastricht, The Netherlands, 2009.
  42. C. Lohrmann, Ed., Prävalenzerhebung 07. April 2009, Medizinische Universität Graz, Institut für Pflegewissenschaft, Graz, Austria, 2009.
  43. WMA, (World Medical Association), “Declaration of Helsinki—Ethical principles for Medical Research involving Human Subjects,” 2008, http://www.wma.net/en/30publications/10policies/b3/.
  44. G. Schmitz, “Inkontinenz,” in Pflegeprobleme in Deutschland. Ergebnisse Von 10 Jahren Forschung in Pflegeheimen Und Kliniken 2001–2010, T. Dassen, Ed., pp. 51–57, Charité-Universitätsmedizin Berlin, Berlin, Germany, 2010.
  45. R. J. G. Halfens, J. M. G. A. Schols, S. Bartholomeyczik et al., International Prevalence Measurement of Care Problems (LPZ): 2009-2010. At a glance, Datawyse, Maastricht, 2011.
  46. S. Bartholomeyczik, “Reparaturbetrieb Krankenhaus. DRGs und ihre Auswirkungen aus Sicht der Pflege,” Langfassung des Artikels aus Dr. med. Mabuse Nr. 166, März/April 2007, http://www.mabuse-verlag.de/chameleon/outbox/public/4/166_Bartholomeyczik.pdf.
  47. B. Braun, P. Buhr, and R. Müller, “Pflegearbeit im Krankenhaus. Ergebnisse einer wiederholten Pflegekräftebefragung und einer Längsschnittanalyse von GEK-Routinedaten,” GEK-Edition, Schriftenreihe zur Gesundheitsanalyse, Band 60, Asgard-Verlag, St. Augustin, Germany, 2008, http://www.barmer-gek.de/barmer/web/Portale/Presseportal/Subportal/Infothek/Studien-und-Reports/GEK-Schriftenreihe-Gesundheitsanalyse/Berufsleben-Gesundheit/PDF-Pflegekraefte-60, property=Data.pdf.
  48. Deutsches Netzwerk für Qualitätsentwicklung in der Pflege (DNQP), Ed., Expertenstandard Entlassungsmanagement in Der Pflege, Aktualisierung, University of Applied Sciences Osnabrück, Osnabrück, Germany, 2009, http://www.dnqp.de/ExpertenstandardEntlassungsmanagement_Atk.pdf.
  49. T. van Achterberg and A. E. Sales, “Implementation Science for nursing: evidence needed! Call for papers for a special issue,” International Journal of Nursing Studies, vol. 48, no. 10, pp. 1163–1164, 2011. View at Publisher · View at Google Scholar · View at Scopus
  50. D. Mazza, P. Bairstow, H. Buchan et al., “Refining a taxonomy for guideline implementation: results of an exercise in abstract classification,” vol. 8, article 32, 2013. View at Publisher · View at Google Scholar
  51. R. Robertson and K. Jochelson, Interventions That Change Clinician Behaviour: Mapping the Literature, NICE, London, UK, 2006, http://www.nice.org.uk/media/AF1/42/HowToGuideKingsFundLiteratureReview.pdf.
  52. A. P. Farmer, F. Légaré, L. Turcot et al., “Printed educational materials: effects on professional practice and health care outcomes,” Cochrane Database of Systematic Reviews, vol. 3, article CD004398, 2008. View at Scopus
  53. G. A. Zangaro and K. L. Soeken, “A meta-analysis of studies of nurses' job satisfaction,” Research in Nursing and Health, vol. 30, no. 4, pp. 445–458, 2007. View at Publisher · View at Google Scholar · View at Scopus
  54. J. E. Squires, C. A. Estabrooks, P. Gustavsson, and L. Wallin, “Individual determinants of research utilization by nurses: a systematic review update,” Implementation Science, vol. 6, article 1, 2011. View at Publisher · View at Google Scholar · View at Scopus
  55. C. Urquhart, R. Currell, M. J. Grant, and N. R. Hardiker, “Nursing record systems: effects on nursing practice and healthcare outcomes,” Cochrane Database of Systematic Reviews, vol. 1, article CD002099, 2009. View at Scopus
  56. M. Wensing, H. Wollersheim, and R. Grol, “Organizational interventions to implement improvements in patient care: a structured review of reviews,” Implementation Science, vol. 1, 1, article 2, 2006. View at Publisher · View at Google Scholar · View at Scopus
  57. M. Prior, M. Guerin, and K. Grimmer-Somers, “The effectiveness of clinical guideline implementation strategies—a synthesis of systematic review findings,” Journal of Evaluation in Clinical Practice, vol. 14, no. 5, pp. 888–897, 2008. View at Publisher · View at Google Scholar · View at Scopus
  58. B. Gardner, C. Whittington, J. McAteer, M. P. Eccles, and S. Michie, “Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback,” Social Science and Medicine, vol. 70, no. 10, pp. 1618–1625, 2010. View at Publisher · View at Google Scholar · View at Scopus
  59. A. Boaz, J. Baeza, A. Fraser, and The European Implementation Science Collaborative Group (EIS), “Effective implementation of research into practice: an overview of systematic reviews of the health literature,” BMC Research Notes, vol. 4, article 212, 2011. View at Publisher · View at Google Scholar
  60. T. Sinuff, N. E. Cahill, R. Dhaliwal, M. Wang, A. G. Day, and D. K. Heyland, “The value of ‘audit and feedback reports’ in improving nutritional therapy in the intensive care unit: a multicentre observational study,” American Society for Parenteral and Enteral Nutrition ASPEN, s.a., http://www.nutritioncare.org/Index.aspx?id=4568.
  61. P. Sudsawad, Knowledge Translation: Introduction To Models, Strategies, and Measures, Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research, Austin, TX, USA, 2007, http://www.ncddr.org/kt/products/ktintro/ktintro.pdf.
  62. N. Opiyo and M. English, “In-service training for health professionals to improve care of the seriously ill newborn or child in low and middle-income countries,” Cochrane Database of Systematic Reviews, vol. 4, article CD007071, 2010.
  63. F. Simpson and G. S. Doig, “The relative effectiveness of practice change interventions in overcoming common barriers to change: a survey of 14 hospitals with experience implementing evidence-based guidelines,” Journal of Evaluation in Clinical Practice, vol. 13, no. 5, pp. 709–715, 2007. View at Publisher · View at Google Scholar · View at Scopus
  64. S. C. Kim, C. E. Brown, W. Fields, and J. F. Stichler, “Evidence-based practice-focused interactive teaching strategy: a controlled study,” Journal of Advanced Nursing, vol. 65, no. 6, pp. 1218–1227, 2009. View at Publisher · View at Google Scholar · View at Scopus
  65. G. Flodgren, M. P. Eccles, S. Shepperd, A. Scott, E. Parmelli, and F. R. Beyer, “An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes,” Cochrane Database of Systematic Reviews, vol. 7, article CD009255, 2011. View at Scopus
  66. I. C. Buss, R. J. G. Halfens, H. H. Abu-Saad, and G. Kok, “Pressure ulcer prevention in nursing homes: views and beliefs of enrolled nurses and other health care workers,” Journal of Clinical Nursing, vol. 13, no. 6, pp. 668–676, 2004. View at Publisher · View at Google Scholar · View at Scopus
  67. Deutsches Netzwerk für Qualitätsentwicklung in der Pflege (DNQP), “Expertenstandards in der Pflege: Zur Entwicklung wissenschaftsbasierter Expertenstandards,” s.a., http://www.dnqp.de/Projekte.htm.
  68. Deutsches Netzwerk für Qualitätsentwicklung in der Pflege (DNQP), Ed., Methodisches Vorgehen Zur Entwicklung, Einführung Und Aktualisierung Von Expertenstandards in Der Pflege, University of Applied Sciences, Osnabrück, Germany, 2011.
  69. “LPZ—Résumé,” CAPHRI School for Public Health and Primary Care, Maastricht University, s.a., http://www.lpz-um.eu/eng/home.
  70. “Bundesgesetzblatt, Gesetz zur strukturellen Weiterentwicklung der Pflegeversicherung (Pflege-Weiterentwicklungsgesetz) vom 28. Mai 2008,” Bundesgesetzblatt Jahrgang, Teil I Nr. 20, ausgegeben zu Bonn am 30, 113a, 3, 2008, http://www.mdk-qualitaetsberichte.de/files/080528-pfwg-mit-aenderungen.pdf.
  71. M. B. Hamilton, “Online survey response rates and times. Background and Guidance for Industry,” 2009, http://www.supersurvey.com/papers/supersurvey_white_paper_response_rates.pdf.
  72. D. F. Polit and C. T. Beck, Nursing Research. Generating and Assessing Evidence For Nursing Practice, Wolters Kluwer Health, Lippincott Williams and Wilkins, Philadelphia, Pa, USA, 9th edition, 2012.