A Formative Evaluation of a Crucial Conversations™ Program: A Case Study

Medical errors account for approximately 200,000 deaths per year Andel et al. [1]. There are alarming trends showing increasing risks for patient harm Downey et al. [2]. There is evidence that medical errors are underreported, and possibly the third leading cause of death in the United States Makey & Daniel [3]. Doweney and colleagues further contend that multiple analyses of patient safety and quality improvement programs demonstrate at best mixed results. One problem is a reliance on methodologies that focus on performance metrics emphasizing easily measurable but inconsequential factors Bilimoria & Barnard [4]. The formative program evaluation is a comprehensive assessment methodology for considering eclectic factors affecting continuous improvement within the organization Owen [5]. This case study describes a formative program evaluation conducted to evaluate a patient safety and quality continuous improvement program within a large medical center Trinidad [6].

The Center for Disease Control (2012) defined a program as any set of organized activities supported by a set of resources to achieve a specific and intended result. Program evaluation applies pragmatic methodologies to assess the value of an organized program effort while gaining knowledge Owen [5]. The formative program evaluation is “defined as a rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts” Stetler et al. [7]. More so, the formative program evaluation is an effective and robust methodology for evaluating patient safety and quality programs Farley & Battles [8]. Research suggests that there are five specific domains guiding the conceptual framework for a formative program evaluation of patient safety and quality program. The domains used in this case study are

A. Evaluating progress in patient safety

B. Translating evidence into practice

C. Assessing and improving culture

D. Identifying and mitigating hazards

E. Evaluating the association of organizational characteristics and outcomes Pronovost et al, [9].

1. Process Theory and Program Logic Model

Mohr [10] argued that describing organizational performance is either quantitative measurement of variance or qualitative assessments through observation. Mohr described two theories

A. Variance theory, which quantitatively explains organizational performance through variances of exogenous and endogenous variables, and

B. Process theory, which explains evaluating organizational activities through observation and analyzing probable explanations for outcomes.

Mohr [10] proposed that the inapplicable emphasis on quantitative variance theory explains the failure to capture various human interactions and responses as influential organizational factors. The program logic model might be the instrument for balancing these assessment approaches. The program logic model is a structure and a tool for clearly communicating the results of a formative program evaluation to multiple audiences Rossi et al. [11]. The program logic model can facilitate integrating the results derived from various evaluation methodologies that includes both quantitative and qualitative approaches Cooksy et al. [12].

The program logic model provides the outline defining a program’s specific elements for planning, executing, and evaluating continuous improvement programs Knowlton & Phillips [13]. Program logic modeling is particularly suited for evaluating patient safety and quality programs by clearly structuring resources and interventions into a logical sequence of inputs, activities, outputs, results, and goals Goeschel et al. [14]. Further, the program logic model provides a structured methodology for assessing program implementation fidelity McCabe et al. [15].

The challenges of patient safety and quality involves understanding human dynamics working within a system as a cause for failures Leape & Berwick [16]. Effective patient safety and quality program interventions are difficult to design and evaluate because patient safety and quality issues involve human complexities working within complicated systems Kaplan et al. [17]. Kaplan suggested that evaluating patient safety and quality programs involves multiple interactive factors requiring different approaches outside of currently accepted linear scientific methodologies. A non-linear approach that identifies patterns, such as the formative program evaluation, might be the better methodology to improve patient safety and quality programs. Non-linear approaches are particularly effective for understanding systems failures Weinberg [18]. Patient safety and quality catastrophes often originate from humans acting within failures in a system, and prevention requires finding systems problems rather than pronouncing blame for individual shortcomings Leape & Berwick [16].

Very frequently organizations only look at superficial problems, and often respond to sentinel events impetuously by implementing simple checklists or safety tools to counter systemic human failures that risk patient safety Maxfield et al. [19]. He further proposed that communication failures within a system are complex, and checklists alone do not prevent communications failures that repeatedly result in unintended patient harm. Complex problems involving human interactions are often unpredictable, and understanding is typically constrained by retrospection Snowden [20]. Furthermore, the failure to understand and address the variation in human responses and communications within a system is a major contributor to the lack of progress in improving patient safety and quality Wachter [21]. It is clear that understanding human interactions requires non-linear thinking outside of testing isolated factors within experimental random control trials Kaplan et al. [17], Higgins [22]. This evidence suggests that a retrospective formative program evaluation analyzing a communications improvement program directed at systemic failures might yield results for understanding human interaction affecting patient safety and quality.

Patterson et al. [23] theorize that the root cause of certain adverse outcomes are a direct result of dysfunctional human interactions and communication stemming from escalating disagreements. Medical errors are more likely to occur because of conflict that occurs related to differences of opinion in interactions involving high stakes and highly emotional situations. VitalSmarts® Crucial Conversations™ is a formalized program to address communication issues in healthcare organizations. Silence Kills™ is an application of Crucial Conversations theory specifically to address patient safety and quality by proposing seven specific sources of human conflict. The identifiable sources of conflict affecting patient safety and quality are: “Broken Rules; Mistakes; Lack of Support; Incompetence; Poor Teamwork; Disrespect; and Micromanagement” Maxfield et al. [24]. This represents an organization wide opportunity to improve patient safety and quality by resolving these explicit sources of conflict through a Crucial Conversations™ program. This case study demonstrates evaluating an existing patient safety and quality program, Crucial Conversations™, by applying the program logic model as a framework for communicating continuous improvement. In this case study, changing human interactions and communication between healthcare professionals are explored that might improve overall patient safety and quality.

The first aim of this project was to construct a prototypical program logic model specifying the VitalSmarts® Crucial Conversations™ program by identifying program theory and constructing program elements. However, a review of the literature did not indicate a well-established logic model representation for the program. Table 1 illustrates the basis for deriving general program theory and synthesizing logic model elements for determining program fidelity. The logic model shown in Table 1 depicts individual elements specified by multiple sources used for a comprehensive formative evaluation of the program. The VitalSmarts® Crucial Conversations™ program teaches people to communicate more effectively, especially in situations with a high risk of long-term harm Maxfield et al. [24].

Table 1:Exemplar VitalSmarts® Crucial Conversations™ Program logic model element sources.

The logic model derived from VitalSmarts® Crucial Conversations™ general program theory used well-founded organizational theory principles. One important aspect is the concept of the linking construct connecting planned interventions with the intended and measured program outputs McDavid & Hawthorn [25]. This is the basis for demonstrating Crucial Conversations™ communication theory as applied to an organizational continuous improvement program. The source for linking the program outputs to designed overall organizational improved outcomes were derived from theoretical concepts found in the literature. Human interactions can involve highly emotional and opinionated conflict, and these conversations can result in disruptive behaviors rather than resolution Patterson et al. [26]. Disruptive behaviors can even escalate into physical violence Grenny [27].

Reducing conflict and engaging in meaningful dialogue requires early recognition of interactions that risk escalating emotions. A productive dialogue requires understanding how opposing opinions and heightened emotions lead to dysfunctional behaviors, and how specific conversational techniques avoids escalating disruptive behaviors related to confrontational issues Patterson et al. [23]. Individuals must first self-recognize susceptibility for disorderly conduct when faced with disturbing conflict and training oneself to finding shared meaning rather than fixating on differences Patterson et al. [23]. Educational techniques can teach the individual to engage in meaningful conversation centered on collective values that connect wide chasms in opinions while reducing the risks of escalation to disruptive behaviors. Greeny [28] proposes that a synergy of productive energy from organizational wide Crucial Conversations™ leads to better overall organizational communication and outcomes, particularly vertical communication through the hierarchy. These theoretical concepts provide a foundation for improving organizational communication and eventually improving overall results.

The identifiable performance goals for this organizational wide program is training people to participate in VitalSmarts® Crucial Conversations™ and using coaching effectively to improve areas of importance. VitalSmarts® Crucial Conversations™ theory is used to realize outcomes through the Impact Cloud™, the tool designed to track results. Models for a prescribed, wide-ranging organizationwide program operation requires employees and managers to engage in the training together with an immediate implementation objective of measuring numbers trained and valuing the application of the program within the organization Greeny [27], VitalSmarts [29], VitalSmarts (n.d.) [30]. Table 2 lists the training components of the program. The training program’s main goal is teaching people to engage in conversations where passion and risks are highest when values are in extreme contrast VitalSmarts [31].

Table 2:Exemplar VitalSmarts® Crucial Conversations™ training program logic model.

In a structured format of viewing a vignette showing poor interactions, engaging in an instructor-led discussion, and then viewing a Crucial Conversation™ with the same situation evolving into a better outcome. Students are encouraged to present real life vignettes VitalSmarts (n.d.) [30]. Nine systematized lessons cover a wide range of typically contentious confrontations. The course begins with a focus focusing on self-control and redirecting one’s own unconstructive energy towards meaningful resolutions, and then transitions to spotting and diffusing situations with a risk of unruly or even violent behaviors VitalSmarts, n.d. [30] Classes consist of a structured two-day seminar held throughout the United States with costs advertised at $1195 per student VitalSmarts [29].

This case study presents the methodology and data from a Doctor of Nursing project that conducted a comprehensive formative program evaluation Trinidad [6]. After constructing an exemplary logic model of the VitalSmarts® Crucial Conversations™ program by classifying program theory into program logic model components, Trinidad [6] used the information gained through interviews about the medical center’s implementation of the program and evaluated the medical center’s application of the Crucial Conversations™ program fidelity to the program logic model. The Chief Research Officer [CRO] was interviewed first to provide historical data for operationalizing the Crucial Conversations™ program. The CRO also exchanged information with the Director of Employee Development, who provided specific training data for the medical center. The data collection process consisted of multiple personal interviews and email exchanges Trinidad [6].

The results for a formative evaluation includes describing the activities in addition to findings. First, the process of evaluating Crucial Conversations™ program theory and constructing a program logic model are discussed. The results of these activities structured the foundation for a formative evaluation of the program. Next, the process for developing elements from a program in progress at a Major Medical Center and organizing that information within a program logic model are described, Finally, results comparing the VitalSmarts and the Major Medical Center’s Crucial Conversations™ programs’ distinct elements are described.

Table 1 shows the literature referenced for the logic model components used to compose the general program logic model. Table 2 organizes these components into a logic model describing the VitalSmarts® Crucial Conversations™ general training program. This program logic model is the basis for comparing and evaluating program fidelity.

The formative program evaluation of the Medical Center’s implementation of Crucial Conversations™ was conducted through formal interviews and email exchanges. Trinidad [6] developed questions specifying program elements organized by focus area (Table 3). The answers to these questions formulated the program logic model exemplified in Table 4. The VitalSmarts® Crucial Conversations™ program began as the result of a strategy to focus on reported unresolved issues between employees and management. The goal was to introduce newly hired supervisors, managers, and directors to Crucial Conversations™ principles to reduce the number of reported unresolved issues. The motivation for implementing Crucial Conversations™ theory began in 2008 with exposure to the book Crucial Conversations: Tools for Talking when the Stakes are High Patterson et al. [26].

A presentation to executive management with materials from a workshop led senior management to endorse a change in orientation training. Orientation training would include addressing conflicts and promoting positive discourse rather than implicitly enabling a worsening culture of avoiding discussing unresolved differences and viewpoints. The immediate program goal was to reveal to management a possible solution to the problem with the number of unresolved issues between employees and management. The following dynamics influenced this assessment. Multiple reports of rude and unruly behaviors happening during management and employee interactions and of medical staff confrontations with professional healthcare workers. Increasing provocative activities were suspected of inhibiting patient quality and safety reporting.

Table 3:Specific areas of inquiry organized within patient safety and quality domains.

Table 4:Major medical center’s historical crucial conversations training program logic model.

The Southeast Medical Center’s training program was modified to consist a four-hour exploratory lesson during orientation. The training materials included the book Crucial Conversations: Tools for Talking when the Stakes are High Patterson et al. [26], and brochures summarizing key points of the book. The handouts guided participants through types of conflicts that are likely to deteriorate into hostility and inappropriate behaviors.

There were 32 of potentially 308 managers trained from a population of 5,774 employees during April 2009 to August 2013.

Program measurements included the count of unresolved issues, the amount of new unresolved issues reported to executives, and the quantity of reportable employee incidents. Longer-term measurements affecting executive level metrics were not tracked; however, it was reported that a shorter-term reduction in employee incidents would lead to a longer-term improvement in other areas.

The elements from the VitalSmarts® Crucial Conversations™ General Program Logic Model were compared with like elements from Major Medical Center Historical Crucial Conversations™ program logic model for program fidelity (Table 5). The comparison revealed that the Major Medical Center Historical Crucial Conversations™ program implementation does not demonstrate program fidelity and did not implement the program as designed. In particular, three areas lacked program fidelity: the number and type of employees trained, the resources used for training, and metrics to measure program success.

Table 5:Exemplar VitalSmarts® Crucial Conversations™ general program logic model and major medical center’s Crucial Conversations™ historical logic model program fidelity comparison matrix.

This project demonstrated the application of process theory, formative program evaluation methodologies, and the program logic model to evaluate a patient safety and quality program. This enables both evaluating theoretical outcomes as well as comparing actual outcomes for analyzing lessons learned and inputs into causal analysis. Furthermore, the program logic model provides a means for tying a patient safety and quality program’s outcomes into the organization’s holistic short term and long-term goals outside of the continuous improvement program Maxfield [32].

The formative evaluation process in itself can be methodology for probing internal factors constraining patient safety and quality improvement efforts, and the basis for understanding requirements for continuous improvement. For example, VitalSmarts® Crucial Conversations™ general program theory proposes training all employees, but the formative evaluation demonstrated how resources constrained scope to only management personnel. Though the formative evaluation demonstrated a lack of fidelity between VitalSmarts® Crucial Conversations™ general program theory and the organization’s program, there are still many extensible lessons learned. One outcome of the process might suggest that organizations will individually compromise rigorous program adoption, program fidelity, and program strength as factors constrained by a program scope, resources, and time. Medical Center for disease control [33] only trained a limited number of select managers rather than every employee to control the program’s overall costs.

The case study demonstrated how the program logic model provides a tool for systematically evaluating these factors as a part of an organization’s continuous improvement process [34]. The application of a formative program evaluation and program logic models are of significance to healthcare leadership implementing translational research and organizational continuous improvement as it allows for a comprehensive evaluation of outcomes. Using program logic models to frame a formative program review of complex problems might be useful for addressing other complex problems affecting patient safety and quality.

1. Andel C, Davidow SL, Hollander M, Moreno DA (2012) The economics of health care quality and medical errors. J Health Care Finance 39(1): 39-50.
2. Downey JR, Hernandez BT, Banka G, Morton JM (2012) Is patient safety improving? National trends in patient safety indicators: 1998-2007. Health Serv Res 47(1 Pt 2): 414-430.
3. Makary M, Daniel M (2016) Medical error-the third leading cause of death in the US. BMJ 353.
4. Bilimoria KY, Barnard C (2016) The new CMS hospital quality star ratings: The stars are not aligned. Send to JAMA 316(17): 1761-1762.
5. Owen J (2006) Program evaluation: Forms and approaches (3rd edn), Allen & Unwin Crow’s Nest, Australia.
6. Trinidad DR (2013) A formative evaluation of the crucial conversationsTM program. (Doctorial practice inquiry).
7. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, et al. (2006) The Role of Formative Evaluation in Implementation Research and the QUERI Experience. J Gen Intern Med 21(Suppl 2): S1-S8.
8. Farley DO, Battles JB (2009) Evaluation of the AHRQ patient safety initiative: framework and approach. Health Serv Res 44(2 Pt 2): 628- 645.
9. Pronovost PJ, Goeschel CA, Marsteller JA, Sexton JB, Pham JC, et al. (2009) Framework for patient safety research and improvement. Circulation 119(2): 330-337.
10. Mohr LB (1982) Explaining organizational behavior. San Francisco, Jossey-Bass, Canada.
11. Rossi PH, Lipsey MW, Freeman HE (2003) Evaluation: A Systematic Approach. (7th edn), SAGE Publications, New York, USA.
12. Cooksy LJ, Gill P, Kelly PA (2001) The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning 24(2): 119-128.
13. Knowlton LW, Phillips CC (2013) The logic model guidebook: Better strategies for great results. (2nd edn), Sage Publications, Washington DC, USA.
14. Goeschel CA, Weiss WM, Pronovost PJ (2012) Using a logic model to design and evaluate quality and patient safety improvement programs. Int J Qual Health Care 24(4): 330-337.
15. McCabe BK, Potash D, Omohundro E, Taylor CR (2012) Design and implementation of an integrated, continuous evaluation, and quality improvement system for a state-based home-visiting program. Matern Child Health J 16(7): 1385-1400.
16. Leape LL, Berwick DM (2005) Five years after to Err is human: What have we learned? JAMA 293(19): 2384-2390.
17. Kaplan G, Bo-Linn G, Carayon P, Pronovost P, Rouse W, et al. (2013) Bringing a systems approach to health: Discussion paper
18. Weinberg G (2011) An introduction to general systems thinking.
19. Maxfield DG, Lyndon AA, Kennedy HP, Zlatnik MD (2013) Confronting safety gaps across labor and delivery teams. Am J Obstet Gynecol 209(5): 402-408.
20. Snowden D (2002) Complex acts of knowing: Paradox and descriptive self-awareness. Journal of Knowledge Management, 6(2): 100-111.
21. Wachter RM (2010) Patient safety at ten: Unmistakable progress, troubling gaps. Health Aff (Millwood) 29(1): 165-173.
22. Higgins JP (2002) Nonlinear systems in medicine. Yale J Biol Med 75(5- 6): 247-260.
23. Patterson K, Grenny J, McMillan R, Switzler A (2011) Crucial Conversations: Tools for talking when the stakes are high (2nd edn), McGraw-Hill, New York, USA.
24. Maxfield D, Grenny J, McMillian R, Patterson K, Switzler A (2005) Silence kills: The seven crucial conversations for healthcare.
25. McDavid JD, Hawthorn LR (2006) Program evaluation and performance measurement: An introduction to practice. Thousand Oaks, Sage, Canada.
26. Patterson K, Grenny J, Maxfield D, McMillan R, Switzler A (2013) Crucial accountability: Tools for resolving violated expectations, broken commitments, and bad behavior (2nd edn), McGraw-Hill, New York, USA.
27. Grenny J (2009) Crucial conversations: The most potent force for eliminating disruptive behavior. Crit Care Nurs Q 32(1): 58-61.
28. Grenny J (2006) Knowing no boundaries: Five crucial conversations for influencing administration. Physician Executive Journal 12-15.
29. VitalSmarts (2013) Crucial Conversations™ Training.
30. VitalSmarts Training course details.
31. VitalSmarts (2013) Impact cloud.
32. Maxfield LD, Grenny J, Lavandero R, Groah L (2011) The silent treatment: Why safety tools and checklists aren’t enough
33. Centers for Disease Control (2012) Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
34. VitalSmarts (2012) Training and certification.