Reflecting & Evaluating

Though less attention has been paid historically to the need for group and personal reflection, more recent literature is acknowledging its key role in strong teaming and team-building (Edmondson, 2012). Dedicating time for reflecting or debriefing before, during, and after implementation is one way to promote shared learning and improvements (Edmondson et al., 2001). The original CFIR asserted that these times of reflection help foster a learning climate – one in which a successful implementation can be ingrained into institutional memory and help improve the odds for future implementations (Edmondson et al., 2001; Simpson & Dansereau, 2007). Even failures, when reflected upon in an effective way, can lead to future success when the root causes are uncovered with psychologically safety (Klein & Sorra, 1996); failure is key for strengthening learning within organizations (Lapré & Nembhard, 2011).

Timely availability to data for monitoring, evaluation, and process improvement is important (Dy et al., 2015). The original CFIR highlighted that data to support reflection and evaluation includes quantitative and qualitative feedback about the experience, progress, and quality of implementation efforts. Evaluation includes traditional forms of feedback, such as reports and graphs, as well as qualitative feedback and anecdotal stories of success (US DHHS-National Cancer Institute, 2005). Feedback on progress toward those goals or objectives is a key behavior change technique in many individual behavior change theories and models (Carey et al., 2018; US DHHS-National Cancer Institute, 2005) and has strong to moderate evidence supporting implementation at an organizational level (Greenhalgh, Robert, et al., 2004). One review found that effects of using audit and feedback mechanisms to improve practices can lead to small to moderate effects (Jamtvedt et al., 2006), a finding confirmed by a more recent review (Ivers et al., 2012). It is hard for people to continue the work without sufficient feedback that is tightly coupled to goals that are important to them (Hysong et al., 2006). Reviewing progress toward goals allows people to assess whether the innovation is creating value (Jamtvedt et al., 2006).

Qualitative coding guidelines that are aligned with the Updated CFIR will be added in the future.

Regarding quantitative measurement of this construct: In a systematic review of quantitative measures related to implementation, Dorsey et al. identified five measures (Dorsey et al., 2021). Using PAPERS criteria of measurement quality with an aggregate scale ranging from -9 to +36 (Lewis, Mettert, Stanick, et al., 2021), three were assessed with score ranging from 2 to 8. The Community Leader Survey (Prevention Activity Progress Subscale) (Valente et al., 2007) had the highest score of eight. Results indicate the need for continued development of high-quality measures.

As we become aware of measures, we will post them here. Please contact us with updates.

Carey, R. N., Connell, L. E., Johnston, M., Rothman, A. J., de Bruin, M., Kelly, M. P., & Michie, S. (2018). Behavior Change Techniques and Their Mechanisms of Action: A Synthesis of Links Described in Published Intervention Literature. Annals of Behavioral Medicine. https://doi.org/10.1093/abm/kay078.

Dorsey, C. N., Mettert, K. D., Puspitasari, A. J., Damschroder, L. J., & Lewis, C. C. (2021). A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence. Implementation Research and Practice, 2, 263348952110024.

Dy, S. M., Ashok, M., Wines, R. C., & Rojas Smith, L. (2015). A Framework to Guide Implementation Research for Care Transitions Interventions: Journal for Healthcare Quality, 37(1), 41–54. https://doi.org/10.1097/01.JHQ.0000460121.06309.f9.

Edmondson, A. C., Bohmer, R. M., & Pisana, G. P. (2001). Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly, 46(4), 685–716.

Edmondson, A. C. (2012). Teaming: How organizations learn, innovate, and compete in the knowledge economy. Jossey-Bass.

Klein, K. J., & Sorra, J. S. (1996). The Challenge of Innovation Implementation. The Academy of Management Review, 21(4), 1055–1080.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Q, 82(4), 581–629.

Hysong, S. J., Best, R. G., & Pugh, J. A. (2006). Audit and feedback and clinical practice guideline adherence: Making feedback actionable. Implementation Science, 1(1), 9. https://doi.org/10.1186/1748-5908-1-9.

Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O’Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012). Audit and feedback: Effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.CD000259.pub3.

Jamtvedt, G., Young, J. M., Kristoffersen, D. T., O’Brien, M. A., & Oxman, A. D. (2006). Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Quality and Safety in Health Care, 15(6), 433–436. https://doi.org/10.1136/qshc.2006.018549.

Lapré, M. A., & Nembhard, I. M. (2011). Inside the Organizational Learning Curve: Understanding the Organizational Learning Process. Foundations and Trends® in Technology, Information and Operations Management, 4(1), 1–103. https://doi.org/10.1561/0200000023.

Lewis, C. C., Mettert, K. D., Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., & Weiner, B. J. (2021). The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implementation Research and Practice, 2, 263348952110373. https://doi.org/10.1177/26334895211037391.

Simpson, D. D., & Dansereau, D. F. (2007). Assessing Organizational Functioning as a Step Toward Innovation. NIDA Science & Practice Perspectives, 3(2), 20–28.

US DHHS-National Cancer Institute. (2005). Theory at a Glance: A guide for health promotion practice 2nd Edition. https://cancercontrol.cancer.gov/sites/default/files/2020-06/theory.pdf.

Valente, T. W., Chou, C. P., & Pentz, M. A. (2007). Community Coalitions as a System: Effects of Network Change on Adoption of Evidence-Based Substance Abuse Prevention. American Journal of Public Health, 97(5), 880–886. https://doi.org/10.2105/AJPH.2005.063644.