- About this Journal ·
- Abstracting and Indexing ·
- Advance Access ·
- Aims and Scope ·
- Article Processing Charges ·
- Articles in Press ·
- Author Guidelines ·
- Bibliographic Information ·
- Citations to this Journal ·
- Contact Information ·
- Editorial Board ·
- Editorial Workflow ·
- Free eTOC Alerts ·
- Publication Ethics ·
- Reviewers Acknowledgment ·
- Submit a Manuscript ·
- Subscription Information ·
- Table of Contents
Nursing Research and Practice
Volume 2013 (2013), Article ID 814360, 7 pages
Adopting Best Practices from Team Science in a Healthcare Improvement Research Network: The Impact on Dissemination and Implementation
Academic Center for Evidence-Based Practice, School of Nursing, University of Texas Health Science Center San Antonio, MSC 7949, 7703 Floyd Curl Drive, San Antonio, TX 78229, USA
Received 7 December 2012; Revised 1 February 2013; Accepted 4 February 2013
Academic Editor: Deborah Vincent
Copyright © 2013 Frank Puga et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Healthcare is a complex adaptive system, and efforts to improve through the implementation of best practice are well served by various interacting disciplines within the system. As a transdisciplinary model is new to clinicians, an infrastructure that creates academic-practice partnerships and builds capacity for scientific collaboration is necessary to test, spread, and implement improvement strategies. This paper describes the adoption of best practices from the science of team science in a healthcare improvement research network and the impact on conducting a large-scale network study. Key components of the research network infrastructure were mapped to a team science framework and evaluated in terms of their effectiveness and impact on a national study of nursing operations. Results from this study revealed an effective integration of the team science principles which facilitated the rapid collection of a large dataset. Implications of this study support a collaborative model for improvement research and stress a need for future research and funding to further evaluate the impact on dissemination and implementation.
Healthcare has been described as a complex adaptive system (CAS) that involves multiple, interdependent entities and organizational levels [1–3]. Such complexity poses a challenge for transformative change as relationships within a CAS are nonlinear and unpredictable [4, 5]. This challenge is echoed by leaders in the field of quality improvement who have identified shortcomings, such as a lack of rigorous research methods, a failure to study contextual variables, and weak evaluation designs [6–8]. These shortcomings are an indication that the interwoven processes of healthcare delivery are difficult for a single investigator to tease apart. Thus, a collaborative model that integrates multiple perspectives from several disciplines may help advance the field of improvement science and facilitate dissemination and implementation strategies.
Transdisciplinary collaboration is a potentially effective model as it brings together a diverse group of individuals who fully integrate theories, methodologies, and frameworks from their respective fields to work as a cohesive unit on complex issues . This differs from multidisciplinary and interdisciplinary collaboration where multiple individuals work together but remain grounded in their respective ideologies [9–11]. Recently, a few studies have addressed the potential of transdisciplinary collaboration to develop effective interventions in healthcare [12–14]. Despite these efforts, research in healthcare improvement has typically involved a single researcher studying an intervention at a single hospital or clinical unit. This approach is ineffective as the complexity of organizational change and contextual influence can impact the implementation and effectiveness of an intervention [7, 15].
The current consensus in the field of improvement science is that new methodologies are needed to address complex issues associated with healthcare delivery [6, 16]. Specifically, researchers and clinicians need to adopt strategies that redefine relationships and establish new ways of communicating [6, 16–18]. A transdisciplinary model may be the solution. In fact, implementation science frameworks suggest that transdisciplinary interaction promotes effective and sustainable intervention programs .
The shift from a single investigator/single site model to a transdisciplinary investigative team/multisite model calls for unique interactions between clinicians and researchers. Currently, there are few competencies and models for team performance; especially in the context of building transdisciplinary teams for improvement . This paper describes the adoption of a transdisciplinary model in a healthcare improvement research network and the impact of collaboration on the conduct of a national improvement study. The results from this study show promise for enhancing research on improvement and uptake of best practices using transdisciplinary collaboration.
2. Theoretical Framework
The guiding framework to implement and evaluate a transdisciplinary collaborative model in this project was grounded in principles from the science of team science (SciTS). SciTS is a hybrid scientific field that incorporates factors that facilitate and hinder scientific collaboration . This field has generated the evidence to support the claim that transdisciplinary collaboration spurs innovation and accelerates scientific discovery [22–24]. Additionally, SciTS is founded on evidence from human factors engineering and offers guidance on how investigative teams can be effective in studying healthcare improvement.
The SciTS framework for this study was adapted from a previous review of factors contributing to collaborative success . Concepts were adopted by a healthcare research network based on the number of times they were referenced, how often they were used for an evaluation of collaborative work, and their relevance to long-distance collaboration. The four concepts in the framework are as follows: (1) readiness for collaboration, (2) creating a shared mental model, (3) management and planning, and (4) virtual readiness. These concepts are briefly described below.
2.1. Readiness for Collaboration
In order for a collaboration to be successful, scientific teams need to be ready to collaborate on an individual, group, and organizational level . This readiness comes in the form of one’s adaptability and flexibility, openness to diverse perspectives, communication skills, conflict resolution, respect for others, institutional support, and availability of reliable technology [27, 28]. Each of these qualities is essential for collaborative processes and group productivity .
2.2. Shared Mental Model
A shared mental model is the organized knowledge that people use to interpret, explain, analyze, and predict what is happening around them . A shared mental model helps collaborating researchers to coordinate with teammates, form accurate expectations about tasks, and understand and anticipate each other’s actions and needs . Teams make fewer mistakes than individuals, especially when each team member knows his or her responsibilities, as well as the responsibilities of other team members . Organizational researchers have observed that when team members have a shared mental model, it increases the overall team engagement and performance .
2.3. Management and Planning
The success of a project is dependent on the way the work is organized and carried out in a scientific team . Structuring and monitoring are necessary to maintain scientific rigor and protocol fidelity especially in multisite studies. Without proper management, the project may not meet its objectives, or results generated from a study may not be reliable due to variations in study implementation and data collection.
2.4. Virtual Readiness
Virtual collaboration requires a high quality, well-functioning technical infrastructure that is designed to fit the nature of the work [22, 27]. Research suggests that user-centered technology, such as access to email, web space for sharing documents, and a centralized databases, is essential features for long distance collaboration [25, 33]. Without a stable technical infrastructure to support virtual collaboration, readiness to collaborate, shared mental models, and management planning are ineffective in multisite, transdisciplinary teams.
The main research questions for this study were (1) does an SciTS framework facilitate transdisciplinary collaboration in a healthcare improvement research network? and (2) what impact does a transdisciplinary model have on the conduct of a national, multisite improvement study? Mixed methods were used to collect and analyze data. This project was approved by the University of Texas Health Science Center San Antonio (UTHSCSA) Institutional Review Board (IRB).
3.2. Sample and Setting
The objectives of this project focused on transdisciplinary collaboration within a multisite study conducted by the Improvement Science Research Network (ISRN). The ISRN is a national hospital-based research network comprised of approximately 200 partners from academic and practice settings across the nation that have an interest in studying quality improvement. Demographic information for the ISRN membership at the time of this study is presented in Table 1. The majority of the membership consisted of nurses working in various fields in acute care settings or at academic institutions. Doctorate prepared members represented a variety of fields including nursing research, health services research, public health, translational science, quality improvement, and implementation science.
From the ISRN membership, research partners are invited to join Research Collaboratives for ISRN studies. Each network study is supported by a research infrastructure modeled after practice-based research networks (PBRNs) and multisite clinical trials. The research infrastructure supports a Virtual Collaboratory, or center without walls, that allows a team of scientists to work on common problems regardless of location [25, 34]. The cornerstone of this infrastructure is a set of national research priorities that serve as a common rallying point to attract diverse perspectives and integrate paradigms from multiple disciplines in order to study improvement. These priorities were developed through a national stakeholder survey and include transitions of care, high-performing clinical systems and microsystems approaches to improvement, evidence-based quality improvement and best practice, and learning organizations and culture of quality and safety.
ISRN studies are supported through a cyber infrastructure equipped with appropriate communications technology (virtual meeting platforms and teleconference lines), a dedicated web space for sharing information and a central database for data entry. This technology platform is backed with sufficient bandwidth, electronic networking capabilities, and technical support. Adequate capacity for data security, integrity, privacy, rapid retrieval, and long-term archiving are also integrated into the ISRN infrastructure to support data entry and storage.
For this study, data were collected from 14 collaborating hospitals. The average hospital size was 425 beds (range: 120–665) with an average daily census of 74.7% (range: 60%–100%). Each site had at least one principal investigator (PI) and one research coordinator. Both PIs and research coordinators had nursing backgrounds with educational experiences that ranged from Bachelor’s to Doctorate levels. Although the collaborative team also consisted of two study PIs (a doctorate-prepared nurse and a physician, both with extensive research backgrounds) and two research scientists (both PhD prepared with backgrounds in psychology and physiology), they were not included in the data collection due to their proximity to the evaluation and interpretation of results.
SciTS principles were used to develop the key ISRN resources to facilitate the conduct of network studies. These resources included training meetings to build readiness and a shared mental model, a protocol implementation kit to facilitate study management, and a robust technical infrastructure to support long distance collaboration. Sites on the investigative team were required to complete capacity building exercises, including a training session on participating in a Research Collaborative. This 2-hour training session consisted of an overview of the ISRN mission and research priorities, a review of the study objectives, an introduction to the protocol, an explanation of data entry procedures, and a presentation of how to work as a collaborative team. Sites were also asked to review recorded ISRN presentations from experts in the fields of team science and virtual collaboration. Concepts from these presentations were discussed during study meetings. Finally, sites were given a resource guidebook on building successful research collaboration. This guidebook synthesized essential qualities for succeeding in research collaboratives for healthcare improvement .
To facilitate the study implementation and ensure the protocol fidelity, sites were given a protocol implementation kit (PIK). The PIK was designed with two goals in mind: (1) systematic implementation of the study protocol across multiple sites to yield analyzable/reliable data and valid study outcomes; (2) guidance for site PIs to facilitate the conduct of the study. The PIK provided a structured overview of various topics related to the study including: forming the project team, preparing for IRB submission, establishing project timelines, identifying participating staff members, using data collection materials, submitting data to the ISRN, and understanding results from the study.
Finally, sites were given access to a variety of technical resources, including a shared web space, access to conference lines, and a centralized database. A SharePoint site was developed to allow for the easy exchange of study documents. GoToMeeting (Citrix Systems, Inc.; Santa Barbara, CA) and a teleconferencing line were used for real-time meetings and presentations. Lastly, a centralized database was created and maintained by the UTHSCSA Department of Epidemiology and Biostatistics using the Informatics Data Exchange and Acquisition System (IDEAS), a robust web-based human studies research informatics data management framework.
3.4. Data Collection and Analysis
Variables that best represented measures of collaboration readiness, study management, and the presence of a strong mental model were selected for analysis. These variables included (1) the number of protocol deviations (study management), (2) meeting attendance percentage (shared mental model), (3) number of study timeline adjustments (readiness), and (4) number of questions received from PIs and coordinators regarding study objectives (study management, shared mental model, and readiness) which were all collected from regulatory documents (e.g., protocol deviation logs), study meeting notes, and email communication.
Additionally, site PIs () and research coordinators () were surveyed on ISRN coordinating center services and resources. The survey was designed in Survey Monkey (SurveyMonkey.com, LLC; Palo Alto, CA), and a link to access the survey was emailed to Site PIs and research coordinators. Respondents were asked to rate the importance and quality of ISRN features and services on 6-point Likert scale. Dillman’s best practices for internet surveys were adopted for data collection . Results on key resources and services for a collaborative environment are reported. Frequencies and percentages were used to summarize raw data.
4.1. Study Document Review
The number of protocol deviations, number of email correspondence, number of timeline adjustments, and meeting attendance percentage is broken down by site in Table 2. On average, collaborating sites reported six protocol deviations. Examples of deviations reported were study instruments turned in after data collection period ended and data collected from unconsented participants. A review of the study notes revealed that a total of ten sites (70%) successfully adhered to study timelines. Study timeline delays were due to late IRB approvals, accreditation visits, or a change in health information technology systems. The average number of email correspondences per site was 55 emails (range: 20–101 emails). The majority of questions from site PIs and coordinators pertained to the conduct of the study, including IRB regulations, participant recruitment, data collection, and data entry. Few questions (<1%) were directed towards the study’s goals or the function of the collaboration. Finally, the attendance on conference calls averaged 94% with sites being absent due to scheduling conflicts or unexpected emergencies.
4.2. ISRN Coordinating Center Effectiveness Survey
A survey on the quality and importance of ISRN coordinating center services and resources was sent to a total 30 site PIs and coordinators. A total of 17 responses were received for response rate of 57%. Demographic information from survey participants is presented in Table 3. Each respondent had previous experience participating in studies as a part of a formal investigative team (mean: 6 studies per respondent, Range: 1–20 studies). On average, half of these studies involved professions other than nursing (mean: 3 studies per respondent, range: 0–15 studies). Results from survey items specific to providing a collaborative environment are presented in Table 4. Most services and resources were positively rated in terms of quality, but those related to organizational structure, study objectives, and communication were rated highest by respondents (>90%).
The present study described the adoption of a transdisciplinary collaborative model in a multisite improvement study. Resources developed using best practices from SciTS facilitated the study’s conduct as evident by consistent protocol implementation, active engagement, and focused task completion. The impact on productivity could have implications on identifying effective improvement strategies that lead to rapid uptake and spread. Improvement scientists and clinicians in the field need to engage in systems change to support team science and adopt resources, such as research collaborative guidebooks and training modules on SciTS principles, to build capacity for transdisciplinary collaboration.
In the context of collaboration readiness, individuals who demonstrate high levels of readiness are less likely to make mistakes, communicate effectively, and complete objectives in a timely manner . This level of readiness can create a fertile environment for multiple disciplines to come together, blend knowledge, and create a new intellectual space . Both researchers and clinicians must raise their readiness capacity in order to spur transdisciplinary initiatives in the field. The importance of this concept is further validated by the development of standardized evaluations that directly assess readiness in scientific teams, including the National Cancer Institute’s Transdisciplinary Research on Energetics and Cancer (TREC) survey and the Collaboration Success Wizard [29, 37].
Additionally, resources are needed to cultivate shared mental models in the field. Currently resources do exist such as national research priorities in improvement science, frameworks for implementation science, and national methodology conferences [6, 15, 38–41]; yet, more is needed to help build transdisciplinary scientific teams for research. For example, experts have called for the establishment of professional organization for improvement and implementation science . Attention has also been called to taxonomy development for an overarching language in hybrid fields . In the present study, the ISRN research priorities and mission were intended to help create a shared mental model that kept research partners engaged in conducting the study. Without a mental model, the observed level of engagement would have decreased and impacted the quality of study outcomes.
The present study also demonstrated the effectiveness of the ISRN research infrastructure, in particular, the management of a multisite study with standardized approaches. A protocol implementation kit was shown to be an effective tool for ensuring protocol fidelity. The use of such a tool helps meet a need in the field for rigorous research and standardized implementation methods [6, 16, 39]. A failure to ensure consistent implementation and protocol fidelity can decrease the reliability of research findings and thus impact translation into multiple clinical settings.
The move from research to clinical practice requires scaling-up quality improvement initiatives to large-scale network studies. Doing so may yield more effective improvement and implementation strategies, improve dissemination of knowledge, and ultimately change policy. An example of transdisciplinary collaboration is in the numbers associated with the ISRN study evaluated for this paper. The dataset consisted of 24,014 data points reported by 716 acute care medical-surgical nurses across 14 hospitals. This represented 2,452 day shifts and 1,447 night shifts. A team-based approach enabled this particular study to capture a representative national sample to enhance the quality of research and raise scientific rigor. Such scale up of an improvement study will hopefully affect spread and generalizability.
A weakness of the present study is that team science concepts were measured indirectly. This is an indication of a need for established evaluation tools for transdisciplinary collaboration in improvement and implementation research. In fact, there is a nonhealthcare specific tool that is designed to identify potential barriers to collaboration and provide recommendations for improvement. The Collaboration Success Wizard (CSW) was developed to evaluate virtual collaborations based on 15 years of evidence that has identified factors that predict collaborative success . An external evaluation of the same ISRN study described in the present paper revealed that the project was well positioned for successful collaboration using the CSW .
The present study was also limited in its ability to measure the impact of transdisciplinary collaboration on dissemination and implementation. This is mostly due to the fact that this type of evaluation requires time. Thus, more research and funding are needed to demonstrate the impact of transdisciplinary research on spread and uptake of evidence-based quality improvement uptake. Given the magnitude of data collected from improvement studies using this approach, it can be inferred that rigors methods and generalizable results may speed uptake and spread; however, data are still needed to justify this claim.
This project demonstrated the effectiveness of a transdisciplinary model for academic and clinical scientists that are interested in studying improvement. Indications are that national improvement studies are positively assisted by the guidance gleaned from SciTS. Further research is needed on the causal relationship between team-based research and dissemination and implementation of evidence-based quality improvement. With advances in team research and increased funding opportunities to investigate dissemination, implementation, and improvement strategies in healthcare, the gap between what works and what is actually practiced will narrow to raise the quality of care delivery.
This work was supported by the National Institutes of Health/National Institute of Nursing Research Grant no. RC2NR011946 to K. Stevens, and by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant 8UL1TR000149 to R. Clark. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
- J. W. Begun, B. Zimmerman, and K. Dooley, “Health care organizations as complex adaptive systems,” in Advances in Health Care Organization Theory, S. M. Mick and M. Wyttenbach, Eds., pp. 253–288, Jossey-Bass, San Francisco, Calif, USA, 2003.
- R. R. McDaniel and D. J. Driebe, “Complexity science and health care management,” Advances in Health Care Management, vol. 2, pp. 11–36, 2001.
- W. B. Rouse, “Health care as a complex adaptive system: implications for design and management,” The Bridge, vol. 38, no. 1, pp. 17–25, 2008.
- H. J. Lanham, R. R. Mcdaniel, B. F. Crabtree et al., “How improving practice relationships among clinicians and nonclinicians can improve quality in primary care,” Joint Commission Journal on Quality and Patient Safety, vol. 35, no. 9, pp. 457–466, 2009.
- P. E. Plsek and T. Greenhalgh, “The challenge of complexity in health care,” British Medical Journal, vol. 323, no. 7313, pp. 625–628, 2001.
- K. R. Stevens, “Delivering on the promise of EBP,” Nursing Management, vol. 43, no. 4, pp. 19–21, 2012.
- J. Ovretveit, “Understanding the conditions for improvement: research to discover which context influences affect improvement success,” BMJ Quality and Safety, vol. 20, supplement 1, pp. i18–i23, 2011.
- R. E. Glasgow and K. M. Emmons, “How can we increase translation of research into practice? Types of evidence needed,” Annual Review of Public Health, vol. 28, pp. 413–433, 2007.
- P. L. Rosenfield, “The potential of transdisciplinary research for sustaining and extending linkages between the health and social sciences,” Social Science and Medicine, vol. 35, no. 11, pp. 1343–1357, 1992.
- K. Börner, N. Contractor, H. J. Falk-Krzesinski et al., “A multi-level systems perspective for the science of team science,” Science Translational Medicine, vol. 2, no. 49, p. 49cm24, 2010.
- D. Stokols, J. Fuqua, J. Gress et al., “Evaluating transdisciplinary science,” Nicotine and Tobacco Research, vol. 5, no. 1, pp. S21–S39, 2003.
- S. Gehlert, A. Murray, D. Sohmer, M. McClintock, S. Conzen, and O. Olopade, “The importance of transdisciplinary collaborations for understanding and resolving health disparities,” Social Work in Public Health, vol. 25, no. 3-4, pp. 408–422, 2010.
- K. M. Emmons, K. Viswanath, and G. A. Colditz, “The role of transdisciplinary collaboration in translating and disseminating health research: lessons learned and exemplars of success,” American Journal of Preventive Medicine, vol. 35, no. 2, supplement, pp. S204–S210, 2008.
- D. B. Abrams, “Applying transdisciplinary research strategies to understanding and eliminating health disparities,” Health Education and Behavior, vol. 33, no. 4, pp. 515–531, 2006.
- L. J. Damschroder, D. C. Aron, R. E. Keith, S. R. Kirsh, J. A. Alexander, and J. C. Lowery, “Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science,” Implementation Science, vol. 4, no. 1, article 50, 2009.
- D. M. Berwick, “The science of improvement,” JAMA, vol. 299, no. 10, pp. 1182–1184, 2008.
- W. M. Trochim, S. E. Marcus, L. C. Mâsse, R. P. Moser, and P. C. Weld, “The evaluation of large research initiatives: a participatory integrative mixed-methods approach,” American Journal of Evaluation, vol. 29, no. 1, pp. 8–28, 2008.
- B. Boushon, L. Provost, J. Gagnon, and P. Carver, “Using a virtual breakthrough series collaborative to improve access in primary care,” Joint Commission Journal on Quality and Patient Safety, vol. 32, no. 10, pp. 573–584, 2006.
- R. E. Glasgow, L. W. Green, and M. V. Taylor, “An evidence integration triangle for aligning science with policy and practice,” American Journal of Preventive Medicine, vol. 42, no. 6, pp. 646–654, 2012.
- L. Neuhauser, D. Richardson, S. Mackenzie, and M. Minkler, “Advancing transdisciplinary and translational research practice: issues and models of doctoral education in public health,” Journal of Research Practice, vol. 3, no. 2, article 19, 2007.
- H. J. Falk-Krzesinski, K. Börner, N. Contractor et al., “Advancing the science of team science,” Clinical and Translational Science, vol. 3, no. 5, pp. 263–266, 2010.
- G. M. Olson, A. Zimmerman, and N. Bos, Scientific Collaboration on the Internet. Acting with Technology, MIT Press, Cambridge, Mass, USA, 2008.
- D. Stokols, R. Harvey, J. Gress, J. Fuqua, and K. Phillips, “In vivo studies of transdisciplinary scientific collaboration: lessons learned and implications for active living research,” American Journal of Preventive Medicine, vol. 28, no. 2, supplement 2, pp. 202–213, 2005.
- K. L. Hall, A. X. Feng, R. P. Moser, D. Stokols, and B. K. Taylor, “Moving the science of team science forward: collaboration and creativity,” American Journal of Preventive Medicine, vol. 35, no. 2, supplement, pp. S243–S249, 2008.
- J. S. Olson, E. Hofer, N. Bos et al., “A theory of remote scientific collaboration (TORSC),” in Scientific Collaboration on the Internet, G. M. Olson, A. Zimmerman, and N. Bos, Eds., MIT Press, Cambridge, Mass, USA, 2008.
- B. Gray, “Enhancing transdisciplinary research through collaborative leadership,” American Journal of Preventive Medicine, vol. 35, no. 2, supplement, pp. S124–S132, 2008.
- G. M. Olson and J. S. Olson, “Distance matters,” Human-Computer Interaction, vol. 15, no. 2-3, pp. 139–178, 2000.
- D. Stokols, K. L. Hall, B. K. Taylor, and R. P. Moser, “The science of team science: overview of the field and introduction to the supplement,” American Journal of Preventive Medicine, vol. 35, no. 2, supplement, pp. S77–S89, 2008.
- K. L. Hall, D. Stokols, R. P. Moser et al., “The collaboration readiness of transdisciplinary research teams and centers: findings from the National Cancer Institute's TREC Year-One evaluation study,” American Journal of Preventive Medicine, vol. 35, no. 2, supplement, pp. S161–S172, 2008.
- J. E. Mathieu, G. F. Goodwin, T. S. Heffner, E. Salas, and J. A. Cannon-Bowers, “The influence of shared mental models on team process and performance,” Journal of Applied Psychology, vol. 85, no. 2, pp. 273–283, 2000.
- B. D. Edwards, E. A. Day, W. Arthur, and S. T. Bell, “Relationships among team ability composition, team mental models, and team performance,” Journal of Applied Psychology, vol. 91, no. 3, pp. 727–736, 2006.
- E. Salas, D. DiazGranados, S. J. Weaver, and H. King, “Does team training work? Principles for health care,” Academic Emergency Medicine, vol. 15, no. 11, pp. 1002–1009, 2008.
- H. Beyer and K. Holtzblatt, Contextual Design: 4 Customer-Centered Approach to Systems Designs, Morgan Kaufmann Series in Interactive Technologies, Morgan Kaufmann, 1997.
- W. A. Wulf, “The collaboratory opportunity,” Science, vol. 261, no. 5123, pp. 854–855, 1993.
- K. R. Stevens, F. Puga, and D. Patel, Building Successful Research Collaboratives for Healthcare Improvement, Academic Center for Evidence-Based Practice, UT Health Science Center San Antonio, San Antonio, Tex, USA, 2012.
- D. A. Dillman, J. D. Symyth, and L. M. Christian, Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, John Wiley & Sons, New York, NY, USA, 3rd edition, 2008.
- G. M. Olson and J. S. Olson, Collaboration Success Wizard (National Science Foundation under Grant No. 1025769 and U.S. Army Research Institute under contract number W91WAW-07-C-0060), 2012, http://hana.ics.uci.edu/wizard/.
- Improvement Science Research Network. Research priorities, 2010, http://www.isrn.net/research.
- R. E. Glasgow, C. Vinson, D. Chambers, M. J. Khoury, R. M. Kaplan, and C. Hunter, “National institutes of health approaches to dissemination and implementation science: current and future directions,” American Journal of Public Health, vol. 102, no. 7, pp. 1274–1281, 2012.
- E. K. Proctor, J. Landsverk, G. Aarons, D. Chambers, C. Glisson, and B. Mittman, “Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges,” Administration and Policy in Mental Health and Mental Health Services Research, vol. 36, no. 1, pp. 24–34, 2009.
- R. E. Glasgow, E. Lichtenstein, and A. C. Marcus, “Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition,” American Journal of Public Health, vol. 93, no. 8, pp. 1261–1267, 2003.
- M. Wensing, J. M. Grimshaw, and M. P. Eccles, “Does the world need a scientific society for research on how to improve healthcare?” Implementation Science, vol. 7, no. 1, article 10, 2012.
- S. Michie, D. Fixsen, J. M. Grimshaw, and M. P. Eccles, “Specifying and reporting complex behaviour change interventions: the need for a scientific method,” Implementation Science, vol. 4, no. 1, article 40, 2009.
- M. J. Bietz, S. Abrams, D. M. Cooper et al., “Improving the odds through the Collaboration Success Wizard,” Translational Behavioral Medicine, vol. 2, no. 4, pp. 480–486, 2012.