CU Management Approaches for Managing Specific Patient Safety Events Essay

Topic: Managing Patient Safety Events

Consider the complexity in managing specific patient safety events, such as falls or medication errors. Research and discuss accepted management approaches for a specific patient safety event. Be sure to include leadership styles that create a culture of safety.

2/4/23, 5:11 PM
Graduate Discussion Participation Scoring Guide
Print
Graduate Discussion Participation Scoring Guide
Due Date: weekly
Percentage of Course Grade: 30%.
Graduate Discussion Participation Grading Rubric
Criteria
Non-performance
Basic
Proficient
Distinguished
Apply relevant course concepts,
theories, or materials correctly.
25%
Does not explain relevant
course concepts, theories, or
materials.
Explains relevant course concepts,
theories, or materials.
Applies relevant course
concepts, theories, or materials
correctly.
Analyzes course concepts, theories, or
materials correctly, using examples or
supporting evidence.
Collaborate with fellow
learners, relating the discussion
to relevant course concepts.
25%
Does not collaborate with
fellow learners.
Collaborates with fellow learners
without relating the discussion to
the relevant course concepts.
Collaborates with fellow
learners, relating the
discussion to relevant course
concepts.
Collaborates with fellow learners,
relating the discussion to relevant
course concepts and extending the
dialogue.
Apply relevant professional,
personal, or other real-world
experiences.
25%
Does not contribute
professional, personal, or
other real-world experiences.
Contributes professional, personal,
or other real-world experiences,
but contributions lack relevance.
Applies relevant professional,
personal, or other real-world
experiences.
Applies relevant professional, personal,
or other real-world experiences to
extend the dialogue.
Support position with
applicable knowledge.
25%
Does not establish relevant
position.
Establishes relevant position.
Supports position with
applicable knowledge.
Validates position with applicable
knowledge.
Participation Guidelines
Actively participate in discussions. To do this you should create a substantive post for each of the discussion topics.
Each post should demonstrate your achievement of the participation criteria. In addition, you should respond to the
posts of at least two of your fellow learners for each discussion question—unless the discussion instructions state
otherwise. These responses to other learners should be substantive posts that contribute to the conversation by
asking questions, respectfully debating positions, and presenting supporting information relevant to the topic. Also,
respond to any follow-up questions the instructor directs to you in the discussion area.
To allow other learners time to respond, you are encouraged to post your initial responses in the discussion area by
midweek. Comments to other learners’ posts are due by Sunday at 11:59 p.m. (Central time zone).
https://courserooma.capella.edu/bbcswebdav/institution/DHA/DHA8044/190100/Scoring_Guides/discussion_participation_scoring_guide.html
1/1
The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0952-6862.htm
IJHCQA
25,8
Patient safety analysis linking
claims and administrative data
Pietro Giorgio Lovaglio
698
Received 11 June 2009
Revised 15 September 2009
18 November 2009
9 April 2010
16 March 2011
Accepted 16 March 2011
Department of Quantitative Methods, University of Bicocca-Milan, Milan, Italy
Abstract
Purpose – The purpose of this paper is to provide international data on the occurrence (and rates) of
clinical errors, identified by type and consequence in the Lombardy region, and to assess empirically
the association between hospital accreditation-type measures and clinical error rates by merging
hospital discharge records and medical malpractice claim data in the Lombardy region (Italy).
Design/methodology/approach – Data were drawn from the regional database collecting claims
and demands for reimbursement declared by patients hospitalized in regional healthcare structures
and regional archives collecting hospital discharge records. To model the variability of clinical errors
rates, binomial negative regression models were applied. For improved interpretation of the results, a
regression tree methodology was used.
Findings – The results demonstrated that the rate of readmission for the same major diagnostic
category and the rate of discharges against medical advice significantly affect the incidence of errors
causing patient death, whereas the rate of unscheduled surgical readmission in the operating room
significantly affects the rate of surgical error.
Research limitations/implications – The findings confirm that claims data is problematic in
nature because of the limited number of claims generally emerging from administrative sources. The
article proposes using proper regression models for count data, taking into account over-dispersion
and excess zeroes and classification tree methods for a better interpretation of empirical evidence.
Practical implications – Health structures where quality outcomes have a significant impact on
clinical error rates should be monitored in depth, investigating the medical charts of involved patients
to identify quality problems and problematic areas.
Originality/value – As a risk management strategy, the combined use of claims data and clinical
administrative data is proposed to shed light on the more problematic, error-prone areas, allowing
regional stakeholders to receive relevant, highly cost-effective and timely information and an in-depth
understanding of the problematic areas in the assessment of risk.
Keywords Patient safety, Clinical risk management, Adverse events, Medical management,
Quality control
Paper type Research paper
Introduction
Within the framework of health structure evaluation, one of the more recently
investigated dimensions is the monitoring of adverse events, defined as unintentional
injuries or complications caused by healthcare management, resulting in disability,
death or prolonged hospital stay for hospitalized patients (Harvard Medical Practice
Study Investigators, 1990).
The interest in monitoring adverse events is essentially motivated primarily by the
International Journal of Health Care
Quality Assurance
priority of reducing the incidence of medical errors in the health sector (Vincent, 1997,
Vol. 25 No. 8, 2012
2001; Kohn et al., 2000). The seminal US report “To Err is Human: Building a Safer
pp. 698-711
q Emerald Group Publishing Limited Health System” (IOM 2000) as cited by (Kohn et al., 2000), dramatically influenced the
0952-6862
debate over clinical errors in healthcare, both in the US and internationally,
DOI 10.1108/09526861211270640
establishing healthcare safety as a fundamental issue for health stakeholders and
public opinion. While the drive to reduce medical errors has been strengthened by
public pressure and research, the monitoring of adverse events in hospitals has an
ulterior priority motivated by the financial pressures of risk and liability insurance
costs and reimbursements for damages to patients.
Patient safety
analysis
National context
In Italy, the Italian Association of Insurance Companies (ANIA, 2005) reported an
increase in claims against health structures in the period of 1994-2004, indicating a
notable increment in the trend towards physician responsibility (þ148 percent, from
3,150 to 7,800 claims). Furthermore, although the ratio between compensation to
damaged patients and premiums was reduced from 300 percent to 150 percent during the
study period (due to an increase in premium costs), the absolute value of compensations
rapidly increased over time. In 2004, these compensations were quantified as 450 million
euros, whereas the amount of paid premiums exceeded 300 million euros (ANIA, 2005).
Considering the difficulty in collecting reliable data on adverse events and claims, the
insurance market has introduced increasingly stringent practices regarding structural
and managerial risk. These considerations indicate the need to develop instruments to
drastically reduce the occurrence of errors of an organizational/medical nature, even in
advanced health systems. To this regard, the most imperative need is the availability of
data on clinical errors. Clinical errors are consequences of either the failure of a planned
action to be completed as intended or the use of an incorrect medical treatment and in
both cases may cause the patient’s death, significant disability or alteration in the
patient’s homeostasis or damage to the general state of health.
699
Reporting systems
The principal approach to patient safety in the UK, the US, and many other countries
has been to establish local and national reporting systems. However, national/regional
/hospital reporting systems cannot be used as safety measurement systems for various
reasons. First, measuring safety in health care is more problematic than in other
sectors, where mistakes and injuries occur much less frequently, are less varied in
nature, and can be more clearly defined. Second, numerous studies have shown that
reporting systems do not effectively detect adverse events when compared with events
based on systematic reviews of patient records (Vincent et al., 2008; Sari et al., 2007).
Unfortunately, no consolidated systems of voluntary incident reporting have been
established in hospital structures in Italy to allow anonymous voluntary reporting of
medical errors. The few attempts made in this direction have not provided satisfactory
results, mainly because of a prevalent atmosphere of distrust and fear among the
operators regarding retribution and punishment from the structure for those who
report events of malpractice. This is due to a general lack of awareness of the problem,
exacerbated by the lack of legislative procedures in force to de-penalize and protect
operators who report adverse events. Various international studies (Leape, 2002;
Vincent et al., 2008) have provided evidence that such initiatives are, in fact, effective,
when developed in a favorable environment, characterized by a cultural orientation
which shifts the focus from a sense of individual guilt to the promotion a culture which
encourages the reporting and analysis of hospital errors as a valuable contribution to
quality improvement in medical processes.
IJHCQA
25,8
700
In addition to the need for behavioral modification at a cultural level, a uniform
organizational model to manage the risk of clinical errors in hospitals is also an
essential requirement, in order to ensure systematic reduction of malpractice events.
This can be achieved through experimentation of diverse error reporting systems/tools
while simultaneously collecting data to obtain relevant information on high-risk
procedures and error frequency at hospital level. in situations where international data
on adverse events is not available, the utilization of medical malpractice claim files as a
source of information has recently stimulated interest in malpractice studies (Andrews
et al., 1997; Rolph et al., 1991). Wilson et al. (2000) and Vincent at al. (2008), however,
suggest that claims data is problematic in nature, because of the limited number of
claims generally emerging and the lack of evaluation of the severity of incidents. As a
result, neither systematic chart reviews or observational studies which are not easily
replicable on a large scale, have been able to collect sufficient data on large enough
numbers to discern underlying patterns and recommend intervention strategies
(Iezzoni, 1997), thus one alternative could be the utilization of a combination of claims
data and clinical administrative data.
However, the debate on the use of clinical administrative data (by means of
processes or accreditation-type measures used as proxies of quality assessment) to
furnish useful information on quality assessment remains unproven. According to
(Vincent et al., 2008) administrative data does not provide a suitably transparent
perspective on quality or improvement, with (Iezzoni, 1997) suggesting that limited
clinical content and questionable data quality may compromise its utility for this
purpose, posing serious caveats against drawing definitive conclusions. On the
other hand, major consensus exists on the use of administrative data as a useful
screening tool for identifying quality problems and targeting areas in which
quality should be investigated in greater depth by using detailed clinical
information (Iezzoni, 1997).
Therefore, even though claims data cannot be considered a sufficient safety
measurement system, in the case of structures where adverse events are rare,
investigation of the empirical association between the occurrence of clinical errors and
other processes or accreditation-type measures may be helpful (Vincent et al., 2008).
The strategy of combining claims data and clinical administrative data appears
promising, both as an effective means of gaining a more comprehensive understanding
of the health sector processes exhibiting the highest susceptibility to errors, as well as
being effective in identifying areas in need of intervention either in hospital processes
or within specific disciplines working within hospitals (Mitchell et al., 1994; Glynn and
Buring, 1996; Rolph et al., 1991).
The Lombardy Directorate of Healthcare financed research projects with the
purpose of “extracting” appropriate performance indicators from the regional HDR, to
utilize in further analyses for benchmarking within regional health structures
(Lombardy Region Healthcare Directorate, 2004).
This paper has two main goals:
(1) to furnish international data on the occurrence of clinical errors (by type and
consequence to patients) in the Lombardy region (Italy); and
(2) to use claims data and clinical administrative data as a risk management
strategy to highlight problematic, error-prone health structures in the region.
The paper is structured as follows: the next two sections describe the adopted
statistical methodology and the existing Lombardy regional archives regarding patient
claims and demands for reimbursement from regional Hospitals. The penultimate
section provides an empirical application, based on closed claims in the triennium
2004-2006, focused on the evaluation of the associations between the occurrence of
clinical errors and accreditation-type measures and hospital characteristics. The final
section offers pertinent conclusions.
Patient safety
analysis
701
Methodology
The central methodological aim of this paper is to evaluate the relationships between
clinical error occurrence in the health structures and accreditation-type measures
(context outcomes) and other health structure characteristics. In the merged dataset
(HDR and claims data), we chose specialties (responsible for clinical errors and where
the patient was charged in a specific year) belonging to a particular hospital (from here
on referred to as health structures) as a unit of analysis. However, since the relatively
small intersection period of both data sources and the underestimation of claims in
more recent years have prevented drawing robust indications in a longitudinal
perspective, the statistical analyses were performed considering the entire triennium
2004-2006 as the reference period.
Methodologically, count data regression techniques have become popular options for
the examination of the association and effects of several explicative variables (in our
case, the accreditation-type measures and health structure characteristics) with outcome
variables expressed as rates or percentages. The most widely used regression model for
multivariate count data is the Poisson regression model (PR, Cameron and Trivedi, 1996).
However, PR has often been criticized for its restrictive assumptions resulting in an
underestimation of the outcome variability (Lambert, 1992; Cameron and Trivedi, 1996).
In fact, in real-life applications, count data often exhibits over-dispersion (Lambert, 1992):
Table I demonstrates that the variance of surgical error distribution and that of errors
causing lesions distribution are about four times the mean, implying the possibility of
over-dispersion. One classic cause of over-dispersion is the presence of the excess of
zeroes in the analyzed outcome distribution (for example, when many health structures
are not responsible of clinical errors).
Ignoring over-dispersion seriously compromise the goodness-of-fit of the model, also
leading to an overestimation of the statistical significance of the explicative variables.
Although adjustments have been provided by specifying an over-dispersion parameter
in the PR model, other count models, such as the Negative Binomial Regression model
(NBR, Cameron and Trivedi, 1996) appear to be more flexible. NBR is able to model count
Clinical errors by type and consequence
Sum
Mean
Variance
Max
90-Pct
% zero
Infections
Prevention errors
Anaesthesia errors
Therapeutic errors
Surgical errors
Lesions
Death
114
120
250
254
808
1,416
165
0.28
0.30
0.62
0.63
2.00
3.50
0.41
0.66
1.17
1.72
1.28
7.57
13.66
0.72
8
17
14
7
23
39
4
1
1
2
3
4
7
2
84.9
84.4
70.0
67.6
39.8
7.9
76.2
Table I.
Clinical error counts by
type and consequence
(2004-2006)
IJHCQA
25,8
702
data with over-dispersion, because NBR is the extension of PR with a more liberal
variance assumption, modeled by means of a dispersion parameter (“alpha”). Other
methodologies, called Zero-Inflated regression (Lambert, 1992) such as Zero-Inflated
Poisson regression (ZIP, Cameron and Trivedi, 1996), address the issue of excess zeroes
in their own right, explicitly modeling the production of zero counts. Zero-inflated
models are based on the idea that there are two different processes generating the counts
of the outcome: the overall population is partitioned into two groups: one producing zero
counts (e.g. zero errors) and one producing positive counts. To compare competitive
regression models, three goodness-of-fit statistics are helpful: the deviance, essentially
measuring the difference between the observed and expected counts of errors across
health structures, and two similar measures (AIC and SBC) which monitor the sample
size and the addition of ulterior explicative variables in the model.
Additionally, in order to assess the associations between clinical error percentages
and explicative variables (offering a better interpretation of count data regression
results), we propose regression tree-based techniques as additional methodological
instruments. These techniques have been employed with increasing frequency in a
variety of medical studies (Rolph et al., 1991). Regression tree techniques are able to
capture complex interactions between explicative variables used to organize
observations (health structures) in homogenous groups according to the outcome
variable of interest (occurrence of errors). Specifically, we used CART (Breiman et al.,
1984), a recursive-partitioning algorithm. In the first step, CART identifies the single
explicative variable or hospital characteristic that most strongly differentiates health
structures across amounts of clinical errors, thus dividing the entire population of
health structures into two sub-groups, defined by the presence or absence of this
characteristic (or by the levels of this characteristics), each being more homogeneous in
its amount of clinical errors than the overall population. Further, each sub-group is
then divided using another explicative variable that best discriminates the amount of
clinical errors for those subgroups. The process continues iteratively until the last
subgroup candidate either consists of only one health structure or is too small to be
split further.
Claims data in Lombardy
The Lombardy Region (Circular 46/SAN/2004) defined the operating strategy,
activities and modalities for the implementation of a regional risk management
strategy with the aims to reduce medical errors, improve safety for patients and
minimize financial loss for hospitals and doctors (Lombardy Region, 2004). Clinical
errors are classified by the following typologies (RVA, 2006):
.
diagnostic error (such as misdiagnosis leading to an incorrect choice of therapy,
failure to use an indicated diagnostic test, misinterpretation of test results and
failure to act on abnormal results);
.
surgical error (involving surgery, care related to surgery, surgical procedures
related to an operation, or an alleged failure to provide a timely and appropriate
surgical procedure);
.
therapeutic error (as the concomitant use of medications known to interact
toxically, the use of a treatment known to be contraindicated in a specific
condition or failure to adequately monitor a treatment, collateral effects, drug
.
.
.
.
.
dosage prescription or administration, lack of communication between hospital
personnel);
invasive procedure error (caused by diagnostic or therapeutic invasive
procedures);
anaesthesia error (events occurring under the anesthesia/intubation phase);
prevention error (event as a result of omitted procedures of patient prevention
and not considered in the previous types of errors);
infection (such as nosocomial and post-surgical wound infections); and
equipment failure error.
Patient safety
analysis
703
A regional database (Monitoring incidents for Public Liability and Employer’s
Liability of Regional Health System (RCT/O) was established to collect, organize and
monitor claims by patients (or their legal representatives), hospitalized in regional
healthcare structures. Claims were defined as written demands for compensation for
medical injury as a result of an incident. The above (clinical errors) classification
reflects the definitions contained in the Lombardy Region laws aimed at implementing
the Regional Healthcare Risk Management Plan (Lombardy Region, 2004).
Results: the RCT/O database
In total, the RCT/O database contains more than 26,000 denunciations, referring to
events occurring in Lombardy Region Health structures from 1973 to 2006. Out of
11,084 events causing damage to patients, 75 percent of are related to clinical errors.
Despite the large amount of available records, more than 60 percent of all records
collected are still “open claims”, referring to incidents where the responsibility has not
been legally assessed. Thus, only 2,279 closed claims referring to clinical errors
constitute the “cases” for purposes of subsequent analyses. Table II outlines the
percentage of clinical errors by consequence to patients, occurring in different Areas in
the period 1973-2006.
Nearly 6 percent (5.6 percent) of clinical errors caused patient deaths with 94.6
percent attributable to patient injury. Clinical errors differed within Specialties:
Surgical (53.4 percent), Emergency (13.8 percent), Obstetric/Gynecology (12.0 percent),
Hospital General Services (11.0 percent) – including Anesthesia, Laboratory Analysis,
Radiology and Blood transfusion – are responsible for 90.2 percent of clinical errors.
Death
Lesions
Total
Area
n
%
n
%
n
%
Surgical
Emergency unit
Obstetric and gynecology
Hospital general services
Medical
Pediatrics
Intensive Care Unit
Psychiatric
Missing
Total
52
27
15
7
18
2
6
1
1
129
40.3
20.9
11.6
5.4
14.0
1.6
4.7
0.8
0.8
100
1,165
287
258
244
156
24
11
4
1
2,150
54.2
13.3
12.0
11.3
7.3
1.1
0.5
0.2
0.0
100
1,217
314
273
251
174
26
17
5
2
2,279
53.4
13.8
12.0
11.0
7.6
1.1
0.7
0.2
0.1
100
Table II.
Clinical errors by area
and consequence
IJHCQA
25,8
704
Of the 1,217 clinical errors recorded in the surgical area, almost 70 percent of errors
causing death occurred in General Surgery (48.1 percent), Orthopedics-Trauma (21.2
percent), whereas more than 80 percent of errors causing lesions are concentrated in
Orthopedics-Trauma (41.7 percent), General Surgery (24.5 percent), Otolaryngology
(9.5 percent) and Ophthalmology (5.9 percent). The distribution of clinical errors by
type and clinical consequence (outlined in Table III) indicates that 41 percent of total
lesions are attributable to surgical errors, 21 percent to diagnostic errors, 10 percent to
therapeutic errors, 10 percent to anaesthesia errors, 9 percent to invasive procedure
errors, 4 percent to infections, 4 percent to prevention errors and 1 percent to
equipment failure errors. Instead, errors causing deaths are mainly attributable to
diagnostic errors (40 percent), surgical errors (26 percent), therapeutic errors (14
percent) and prevention errors (11 percent), and in a lesser extent to other errors
(anaesthesia errors 3 percent, invasive procedure errors 5 percent and infections 1
percent).
Results: the merged database
As a result of the merging process (HDR and RCT/O database), the database collected
404 health structures (39 types of Specialties in 53 different Hospitals), measured in the
period 2004-2006. Despite the HDR dataset collects 4,464 records (1,697 health
structures – measured in different years – referred to 39 types of Specialties of 203
Hospitals) the relatively small intersection period of both sources (triennium
2004-2006) is essentially due to scarce data quality in HDR records before 2004 and to a
high incidence of open claims in the RCT/O database after 2004.
Table IV shows total counts and rates (calculated on 1,000 discharges at risk) of
clinical errors by type, consequence for patients and year of occurrence.
In the analyzed triennium, 1,581 clinical errors were detected: 51 percent (808 errors)
attributable to surgical errors, 16 percent (254 errors) to therapeutic errors, 16 percent
(250 errors) to anaesthesia errors, 8 percent (120 errors) to prevention errors, 7 percent
(114 errors) to infections, 1 percent to equipment failure errors (24 errors), whereas 11
claims did not have an identifiable type of clinical error attributable to medical care.
Clinical errors caused 165 patient’s deaths and 1,416 lesions. 49 percent of errors
having caused the patient’s death are concentrated in four specialties: Orthopedic,
Cardiology, Obstetric-Gynecology and General Surgery (causing 23, 20, 20 and 18
deaths, respectively in the triennium).
Death
Table III.
Clinical errors by type
and consequence
Lesions
Total
Type of clinical error
n
%
n
%
n
%
Surgical error
Diagnostic errors
Therapeutic error
Anaesthesia errors
Invasive procedure error
Infections
Prevention error
Equipment failure
Total
33
52
18
4
6
2
14
0
129
25.6
40.3
14.0
3.1
4.7
1.6
10.9
0.0
100
888
457
219
207
192
91
76
20
2,150
41.3
21.3
10.2
9.6
8.9
4.2
3.5
0.9
100
921
509
237
211
198
93
90
20
2,279
40.4
22.3
10.4
9.3
8.7
4.1
3.9
0.9
100
The data has provided evidence of two serious problems. First, the declining trend over
time in clinical errors, as well as the particularly low counts in 2006, can be attributed
to the underestimation of claims in more recent years, due to the temporal gaps
existing between the date of the event and the date for demands for compensation. This
is confirmed by the analysis of the temporal gap, whose mean value is 609 days for
therapeutic errors, 705 days for surgical errors, 1,065 days for prevention errors and
3,010 days for infections.
Second, the yearly error rates are relatively rare events (e.g. in 2004 the mean value
of surgical errors is 1.5 per 1,000 discharges), shedding doubt on the robustness of
statistical analyses, if based on this data. Since the above-mentioned factors prevent
drawing robust indications in a longitudinal perspective, the analysis was performed
considering the triennium 2004-2006 as the reference period, summing the clinical
errors occurring each year over the three-year span.
Table I shows the descriptive statistics based on the distribution of clinical errors
by type and consequence in the triennium 2004-2006, across 404 health structures; the
columns of Table I report the total number of occurred errors (sum) and the mean
number of errors (mean) across 404 health structures, the variance of error distribution,
the highest number errors occurring in a health structure (max) and the 90-th percentile
(90-Pct), describing the minimum number of errors occurring in the top 10 percent of
health structures (showing more errors), respectively. The final column ( percent zero)
shows the percentage of health structures not responsible for errors.
For example, the 165 fatal events were provoked by only 24 percent of health
structures. One health structure caused four deaths, the mean value of errors across all
involved structures is 0.41 and the top 10 percent of health structures with largest fatal
errors provoked more than two fatalities. This data demonstrates two different situations:
an extreme sparseness of errors across the involved structures for surgical errors and
errors causing lesions, and a high concentration of errors in isolated units of care.
Patient safety
analysis
705
Modelling error occurrence: variables and results
In the empirical investigation of the association between the occurrence of clinical
errors and Hospital accreditation-type measures, we consider the three following
outcomes as dependent variables to be modeled (occurrence of clinical errors):
(1) the percentage of surgical errors to the total surgical discharges in the
triennium (y1);
Year Discharges
Anaesthesia
error
Surgical
error
Prevention
error
Therapeutic
error
Infection Lesion Death
2004
267,226
85
0.3
388
1.5
57
0.2
108
0.4
57
0.2
634
2.4
79
0.3
2005
262,518
103
0.4
307
1.2
35
0.1
101
0.4
43
0.2
540
2.1
61
0.2
2006
180,084
62
0.3
113
0.6
28
0.2
45
0.2
14
0.1
242
1.3
25
0.1
Total
709,828
250
808
120
254
114
1,416
165
Table IV.
Counts and rates (per
1,000 discharges) of
clinical errors, by year,
type and consequence for
patients
IJHCQA
25,8
706
(2) the percentage of clinical errors resulting in patient death to the total discharges
in the triennium (y2); and
(3) the percentage of clinical errors causing lesions to the total discharges in the
triennium (y3).
Other clinical error typologies proved too rare to be modeled.
Furthermore, as accreditation-type measures to be utilized as explicative variables,
following international evidence (Ibrahim et al., 2007; IQIP, 2004; AHRQ, 2002), we
selected a set of “context outcomes” which could indicate malpractice, such as the total
number of readmissions for the same Major Diagnostic Category (MDC, groups of
diagnoses corresponding to a specific organ system) within a year per 1,000 discharged
patients (T_readmiss_MDC), unscheduled surgical returns to the operating room
within 48 hours (T_readmiss_Room), per 1,000 discharged patients at risk (referring
procedures with ICD-IX surgical code) and the total number of discharges against
medical advice per 1,000 discharged patients (T_discharge_AMA). For these
indicators, infants under two years, day hospital and day surgery discharges,
long-stay patients and patients in rehabilitation were excluded.
Moreover, as additional explicative variables, we selected the following health
structure characteristics to summarize the case-mix of hospitalized patients:
.
the percentage of discharged males (%Male);
.
the mean age (Mean_age);
.
the mean length of stay (Mean_LOS);
.
the percentage of patients with urgency admission (%Urgency);
.
the mean number of comorbidities of involved patients (Mean_Comorb);
.
the percentage of patients with oncological diagnosis (%Onco); and
.
the percentage of patients with cardiologic diagnosis (%Cardio).
Results
The first empirical analysis considers y1 (the percentage of surgical errors to total
surgical discharges) as the outcome of interest. Results suggest that the variation in
surgical error percentages (among health structures) in the triennium 2004-2006 is
well explained by the NBR model. The goodness-of-fit measures (Deviance ¼ 347.7,
df ¼ 324, p-value ¼ 0.175; AIC ¼ 1202; SBC ¼ 1243) show that the NBR model fits
the data very well and almost no over-dispersion is seen. Both the estimated
dispersion parameter (alpha ¼ 0.5766, 95 percent CI: 0.366-0.787) and the test
(Chi-square ¼ 143.2 df ¼ 1), that compares the fit of NBR with PR, are highly
significant, confirming that the NBR is preferred over PR. Furthermore, the ZIP
model has demonstrated poorer fit than the NBR model ðAIC ¼ 1309; BIC ¼ 1372Þ;
suggesting that NBR accounts for over-dispersion in the analyzed data, excluding the
necessity of explicitly modeling the presence of zeroes. Graphically, the fit of PR,
NBR and ZIP can be evaluated by comparing the observed percentages of surgical
errors across all health structures and the average predicted count probability,
provided by three competitive models.
This is shown in Figure 1, where predicted count probabilities have been computed
for health structures having surgical errors 0 through 9 (constituting the 98 percent of
Patient safety
analysis
707
Figure 1.
Probability of surgical
error occurrences modeled
by PR, NBR and ZIP
total surgical errors). The figure illustrates that the PR clearly underestimates the
proportion of zero errors, while the other two models are quite accurate at zero
(adequately fitting the observed proportion of health structures having 0 surgical
errors). Furthermore, the NBR adequately fits the observed proportion of errors for
health structures responsible for high number of errors.
To evaluate the association between the specified explicative variables and the
percentage of surgical errors, Table V shows the NBR parameters estimates (b), their
variability (Std. Error), their confidence interval (95 percent CI) and their significance
(Sign).
Table V demonstrates that the rate of unscheduled readmissions to surgery is one of
the most significant explicative variables. The coefficient (0.0162) illustrates that for each
additional unscheduled return to the operating room for 1,000 patients in the triennium,
the surgical error percentage increases by a factor of 1.016 ð¼ expð0:0162ÞÞ: Instead, the
rate of discharges against medical advice is negatively associated with the occurrence of
surgical errors, whereas the rate of readmissions for the same MDC is not significantly
associated with the occurrence of surgical errors. Looking at case-mix (explicative)
Explicative variables
b
St error
Intercept
T_discharge_AMA
T_readmiss_Room
%Male
Mean_age
%Urgency
Mean_LOS
%Cardio
%Onco
Alpha (dispersion)
2 7.5330
2 0.0163
0.0162
0.0141
0.0191
2 0.0445
2 0.0777
2 0.0106
2 0.0116
0.5766
0.3412
0.0075
0.0054
0.0039
0.0074
0.0161
0.0323
0.0041
0.0062
0.1075
95% CI
28.2017
20.0311
0.0055
0.0065
0.0046
20.0762
20.1409
20.0186
20.0238
0.3660
26.8643
20.0016
0.0269
0.0218
0.0336
20.0129
20.0145
20.0026
0.0006
0.7873
Chi-square
Sign.
487.48
4.71
8.88
13.00
6.65
7.62
5.80
6.79
3.50
, .0001
0.0300
0.0029
0.0003
0.0099
0.0058
0.0160
0.0092
0.0614
Table V.
Negative binomial
regression model for
surgical errors rate
IJHCQA
25,8
708
variables, surgical errors occur frequently in non-specialized health structures with low
percentages of patients with of oncologic or cardiologic diagnosis. Furthermore, planned
admissions and short length of stay are other significant risk factors for surgical errors,
whereas the number of comorbidities is not a significant risk factor.
Using CART (with ten-fold cross-validation to obtain a smaller, statistically stable
tree) to discriminate health structures according to the amount of surgical errors, the rate
of unscheduled readmissions to surgery resulted as being the most significant
explicative variable. It identified two important groups: the first “problematic” group is
composed of 226 health structures (characterized by T_readmiss_Room $ 3.8 per 1,000
discharges) registering a high risk of surgical errors. In this group, 79 percent of the
health structures caused at least one surgical error, whereas in the total population this
percentage was 61.2 percent (as shown in the last column of Table I). The second group
(characterized by T_readmiss_Room , 3.8 per 1,000 discharges and mean length of stay
higher than five days) is composed of 34 health structures presenting low risk of surgical
errors: in this group, 95 percent of the health structures were not responsible for surgical
errors, whereas in the total population this percentage was only 39.8 percent.
In the second analysis, which considers y2 (the percentage of clinical errors
resulting in patient death to the total discharges in the triennium) as the outcome of
interest, the NBR model fits the data more satisfactorily than the PR and ZIP models.
With regard to the context outcomes, the rate of readmissions for the same MDC is
significant ðb ¼ 0:005; p ¼ 0:0002Þ; indicating a strong positive association with the
risk of fatal error. Additionally, the rate of discharge against medical advice is
significant, but to a lesser extent ðb ¼ 0:016; p ¼ 0:0354Þ: By contrast, the rate of
unscheduled readmissions to the operating room is not significant ðp ¼ 0:5713Þ:
Among case-mix explicative variables, increasing percentages of cardiologic patients
ðb ¼ 1:7694; p ¼ 0:0026Þ and urgency admissions ðb ¼ 0:083; p , 0.0001) are highly
significant risk factors for errors causing death. Other case-mix variables are not
statistically significant at the 0.05 level.
In the CART analysis, only health structure characteristics discriminate health
structures according to the amount of errors causing patient death, whereas context
outcomes do not. The most problematic group is composed of 32 health structures
hosting more than 40 percent of cardiology patients: in this group, 55 percent of health
structures are responsible for at least one fatal error (23.8 percent in the total
population). Another problematic group is composed of 72 health structures hosting
less than 40 percent of cardiology patients and having more than 5 percent of urgency
admissions: in this group, 33 percent of health structures caused at least one fatal error
(23.8 percent in the total population).
In the final analysis, aimed at modeling y3 (the percentage of errors causing lesions
to total discharges in the triennium), all three selected count models (PR, NBR and ZIP)
demonstrated poor fit and context outcome results, which were statistically not
significant at the 0.10 level. Among case-mix variables, only the percentage of
cardiology patients is a significant decreasing risk factor for lesions, indicating that
lesions typically do not occur in Cardiology-specialized health structures. Similarly, the
CART analysis demonstrates that context outcomes do not discriminate health
structures. CART identified one group of 32 health structures (structures having more
than 40 percent of cardiology patients) with low risk of lesions: 37 percent of them were
not responsible for lesions (only 7.9 percent in the total population).
Conclusion
The main objective of this article was to shed light on the potential utility of claims
data.
To this end, for the first time, international data and information on occurrence (and
rates) of clinical errors by type and consequence were collected in the Lombardy
region. We then explored the feasibility of simultaneously analyzing clinical
administrative records and claims data in order to empirically investigate the
association between the occurrence of errors and context indicators. The paper has
furnished two important empirical findings. First, empirical evidence supports the
connection between context indicators (extracted from HDR) and the incidence of
clinical errors. Particularly, both the NBR and CART analysis found that the rate of
unscheduled readmissions to the operating room significantly affects the occurrence of
surgical errors. Second, the rate of readmissions for the same MDC and the rate of
discharges against medical advice significantly affect the incidence of errors causing
patient death (not confirmed by the CART analysis).
The empirical exploration of relationships between hospital characteristics and
types of clinical errors has suggested that:
.
surgical errors occur more frequently in unspecialized health structures hosting
low percentages of patients with oncologic or cardiologic diagnosis with planned
admissions and short periods of hospitalization; and
.
increasing percentages of cardiology patients and emergency admissions are
significant risk factors for errors causing death.
Two practical implications arise from these empirical findings. First, we found little
evidence that claims related to surgical type or errors resulting in patient lesion were
sufficiently concentrated to permit negligence reduction strategies targeted at health
structures (60 percent of involved health structures were responsible for surgical
errors, 82 percent of them causing at least one lesion). In fact, when clinical errors are
more diffusely distributed, targeting individual structures may be a less efficient
strategy than investigating the clinical processes in which many physicians are
involved. This confirms the need to improve clinical processes in the spirit of
continuous quality improvement, emphasizing careful examination, statistical
analysis, and adjustment of clinical processes. To this end, specific adjustments are
recommended to overcome the technical difficulties inherent to the merging process
(e.g. the adoption of uniform coding schemes to identify hospitals, health agencies and
specialties, according to the type of clinical error definition, established by
international institutions or scientific societies).
As a secondary implication, the evidence of significant causal relationships between
error occurrence and accreditation-type measures suggests that available
administrative data may be effective in prospectively determining situations in
which clinical errors typically occur in the regional context, consequently aiding in the
identification of priority intervention areas.
More specifically, we suggest the strategy of monitoring specific context outcomes
(quickly and easily extracted from administrative archives) which are significantly
associated with the occurrence of clinical errors, in order to identify the health
structures, processes and systems of care responsible for not only quality problems,
but also for injury to patients.
Patient safety
analysis
709
IJHCQA
25,8
710
These advantages notwithstanding, although the integration of HDR and RCT/O
administrative archives undoubtedly holds promise for the future, current applications
remain limited. In fact, the presented data has confirmed that claims data is
problematic in nature, given the limited number of claims generally emerging from
administrative sources (underreporting, underestimation due to gaps, or lack of close
calls or near misses/errors that do not result in injury), and the lack of information on
the causes of medical errors causing injury to patients (e.g. processes and systems of
care that may be responsible).
In conclusion, although much remains to be improved in the systematic
measurement of patient safety, in regard to both routine monitoring and the
detection of errors, the approach in the present paper provides an extremely
cost-effective method of receiving timely, relevant information, offering regional
stakeholders the opportunity to gain a deeper understanding of the problematic areas
in clinical risk assessment.
References
Agency for Healthcare Research and Quality (2002), AHRQ Quality Indicators – Patient Safety
Indicators: Software Documentation, Agency for Healthcare Research and Quality,
Rockville, MD.
Andrews, L.B., Stocking, C., Krizek, T., Gottlieb, L., Krizek, C., Vargish, T. and Siegler, M. (1997),
“An alternative strategy for studying adverse events in medical care”, Lancet, Vol. 349,
pp. 309-31.
ANIA (2005), ANIA trends Newsletter, Ufficio Studi, Ania Press, Rome.
Breiman, L., Friedman, J.H., Olshen, R.A. and Stone, C.J. (1984), Classification and Regression
Trees, Wadsworth International Group, Belmont, CA.
Cameron, A.C. and Trivedi, P.K. (1996), “Count data models for financial data”, in Maddala, G.S.
and Rao, C.R. (Eds), Handbook of Statistics, XIV, Statistical Methods in Finance,
North-Holland, Amsterdam, pp. 363-92.
Glynn, R.J. and Buring, J.E. (1996), “Ways of measuring rates of recurrent events”, British
Medical Journal, Vol. 312, pp. 364-7.
Harvard Medical Practice Study Investigators (1990), “Patients, doctors and lawyers: medical
injury malpractice litigation and patient compensation in New York”, report of the
Harvard Medical Practice Study to the State of New York, Harvard University, Boston,
MA.
Ibrahim, S.A., Kwoh, C.K. and Krishnan, E. (2007), “Factors associated with patients who leave
acute-care hospitals against medical advice”, American Journal of Public Health, Vol. 97,
pp. 2204-8.
Iezzoni, L.I. (1997), “Assessing quality using administrative data”, Annals of Internal Medicine,
Vol. 127, pp. 666-74.
International Quality Indicator Project (2004), available at: www.internationalqip.com
Kohn, L., Corrigan, J. and Donaldson, M. (2000), To Err Is Human: Building a Safer Health
System, Committee on Quality of Health Care in America, Institute of Medicine, National
Academy Press, Washington, DC.
Lambert, D. (1992), “Zero-inflated Poisson regression with an application to defects in
manufacturing”, Technometrics, Vol. 34, pp. 1-14.
Leape, L.L. (2002), “Reporting of adverse events”, New England Journal of Medicine, Vol. 347,
pp. 1633-8.
Lombardy Region (2004), Regional Community Health Plan 2002-2004, Regione Lombardia
Press, Milan.
Lombardy Region Healthcare Directorate (2004), Triennial Program for the Implementation of an
Evaluation System of Health Structures and Management of Health Structures, Regione
Lombardia Press, Milan.
Mitchell, J., Bubolz, T., Paul, J., Pashos, C., Escarce, J., Muhlbaier, L., Wiesman, J., Young, W.,
Epstein, R. and Javitt, J. (1994), “Using Medicare claims for outcomes research”, Medical
Care, Vol. 32, pp. JS38-JS51.
Rolph, J.E., Kravitz, R. and McGuigan, K. (1991), “Malpractice claims data as a quality
improvement tool. Is targeting effective?”, Journal of American Medical Association,
Vol. 266, pp. 2093-7.
RVA (2006), Mappatura dei sinistri di RCT/O del Sistema Sanitario Regionale: II Edition Report,
Regione Lombardia Press, Milan.
Sari, A.B.A., Sheldon, T.A., Cracknell, A. and Turnbull, A. (2007), “Sensitivity of routine systems
for reporting patient safety incidents in an NHS hospital: retrospective patient case note
review”, British Medical Journal, Vol. 334, pp. 79-82.
Vincent, C. (1997), “Risk, safety and the dark side of quality”, British Medical Journal, No. 314,
pp. 1775-6.
Vincent, C. (2001), Clinical Risk Management: Enhancing Patient Safety, British Medical Journal
Books, London.
Vincent, C., Aylin, P., Franklin, B., Holmes, A., Iskander, S., Jacklin, A. and Moorthy, K. (2008),
“Is health care getting safer?”, British Medical Journal, Vol. 337, pp. 1205-7.
Wilson, L. and Fulton, M. (2000), “Risk management: how doctors, hospitals and MDOs can limit
the costs of malpractice litigation”, Medical Journal of Australia, Vol. 172, pp. 77-80.
About the author
Pietro Giorgio Lovaglio has since 2005 been an Associate Professor in the Department of
Quantitative Methods, Faculty of Statistics, University of Milan-Bicocca (Italy). From 2001 to
2004 he was a Researcher in the Department of Statistics, Faculty of Statistics, University of
Milan-Bicocca. Since 2002 he has been a Senior (Statistician) Researcher of CRISP
(Interuniversitary Research Center on the Service of Public Utility to the Person) at the
University of Bicocca-Milan. CRISP aims to promote and develop research programs on the
service of public utility to the person (effectiveness, efficiency, equity, customer satisfaction).
Since 2002, CRISP has collaborated with the Health Directorate of the Lombardy Region in the
“Regional Observatory on the Quality of Health Service”, particularly in the areas of measuring
effectiveness and quality of health structures and patient satisfaction. Pietro Giorgio Lovaglio’s
recent publications refer to latent variable modeling and methods of evaluation applied in the
context of the services to the person (effectiveness and customer satisfaction in health, human
capital in labor market; impact analysis). Pietro Giorgio Lovaglio can be contacted at:
piergiorgio.lovaglio@unimib.it
To purchase reprints of this article please e-mail: reprints@emeraldinsight.com
Or visit our web site for further details: www.emeraldinsight.com/reprints
Patient safety
analysis
711
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Health Services Research
© Health Research and Educational Trust
DOI: 10.1111/1475-6773.12625
PATIENT SAFETY & MEDICAL LIABILITY
Progress at the Intersection of Patient
Safety and Medical Liability: Insights
from the AHRQ Patient Safety and
Medical Liability Demonstration Program
M. Susan Ridgely, Michael D. Greenberg, Michelle B. Pillen, and
James Bell
Objective. To identify lessons learned from the experience of the Agency for Healthcare Research and Quality (AHRQ) Patient Safety and Medical Liability (PSML)
Demonstration Program.
Data Sources/Study Setting. On September 9, 2009, President Obama directed the
Secretary of Health and Human Services to authorize demonstration projects that put
“patient safety first” with the intent of reducing preventable adverse outcomes and
stemming liability costs. Seven demonstration projects received 3 years of funding
from AHRQ in the summer of 2010, and the program formally came to a close in June
2015.
Study Design. The seven grantees implemented complex, broad-ranging innovations
addressing both patient safety and medical liability in “real-world” contexts. Some projects featured novel approaches, while others implemented adaptations of existing
models. Each project was funded by AHRQ to collect data on the impact of its interventions. In addition, AHRQ funded a cross-cutting qualitative evaluation focused on
lessons learned in implementing PSML interventions.
Data Collection/Extraction Methods. Site visits and follow-up interviews supplemented with material abstracted from formal project reports to AHRQ.
Principal Findings. The PSML demonstration projects focused on three broad
approaches: (1) improving communication around adverse events through disclosure
and resolution programs; (2) preventing harm through implementation of clinical “best
practices”; and (3) exploring alternative methods of settling claims. Although the
demonstration contributed to accumulating evidence that these kinds of interventions
can positively affect outcomes, there is also evidence to suggest that these interventions
can be difficult to scale.
Conclusions. In addition to producing at least preliminary positive outcomes, the
demonstration also lends credence to the idea that targeted interventions that improve
some aspect of patient safety or malpractice performance may also contribute more
broadly to institutional culture and the alignment of all parties around reducing risk
2414
PSML Demonstration Cross-Site Evaluation
2415
and preventing harm. However, more empirical work needs to be carried out to quantify the effect of such interventions.
Key Words. Patient safety, adverse events, medical error, medical malpractice,
claims, disclosure, offer
Under the Patient Safety and Medical Liability Demonstration Program,
AHRQ funded a diverse set of health care and legal system interventions,
aimed at improving some aspect of patient safety and malpractice outcomes,
and (more fundamentally) the medical–legal environment in which those outcomes occur (Battles, Reback, and Azam 2016). Each of the seven demonstration grantees implemented complex, broad-ranging innovations in “realworld” contexts. Some projects featured novel approaches, while others put
into practice continuations, replications, or adaptations of existing models.
The PSML demonstration projects were originally scheduled to run for
3 years beginning in late summer 2010. Many of the grantees requested and
received no-cost extensions of varying lengths. All but one of the demonstration projects was completed by June 2014; the New York project received an
extension through June 2015.
Each PSML demonstration project was funded by AHRQ to collect
data on the impact of its own interventions. In addition to providing funds to
each project for a local evaluation, AHRQ also contracted with James Bell
Associates ( JBA), an evaluation research firm, and the RAND Corporation, a
nonprofit research organization, to conduct an independent evaluation of the
PSML Initiative. Although the cross-cutting evaluation plan initially included
both quantitative and qualitative elements, it soon became apparent that the
PSML demonstration grants were highly varied in the specific interventions
that they fielded; in the relevant outcomes that they attempted to impact; and
in the degree to which they were successful in collecting data to measure those
outcomes. Because of this heterogeneity, together with the lack of a common
data infrastructure for capturing adverse event and malpractice claims, qualitative assessment of the PSML demonstration program became the focus for
the cross-cutting evaluation. Case-study methods were used to glean significant insight into each of the PSML demonstration projects and to understand
Address correspondence to M. Susan Ridgely, J.D., RAND Corporation, 1776 Main Street, Santa
Monica, CA 90407; e-mail: ridgely@rand.org. Michael D. Greenberg, J.D., Ph.D., is with RAND
Corporation, Pittsburgh, PA. Michelle B. Pillen, Ph.D., and James Bell, M.A., are with James Bell
Associates, Inc., Arlington, VA.
2416
HSR: Health Services Research 51:6, Part II (December 2016)
how a common set of barriers and facilitators impacted implementation and
outcomes across the sites.
M ETHODS
The evaluation of the PSML demonstration aimed to describe the antecedent
conditions and context into which the PSML projects were introduced; assess
grantee progress in accomplishing the intent and objectives of the program;
and determine the overall impact of the program, including diffusion of program-related interventions beyond the projects funded by AHRQ. It was
designed to provide ongoing feedback to AHRQ and to the demonstration
projects. In accordance with AHRQ specifications, the evaluation was based
on the CIPP (context, input, process, product) evaluation model, an approach
to evaluating system improvement that includes a broad spectrum of factors
that can affect program performance (Stufflebeam 2003).
A final evaluation report on the PSML demonstration was provided to
ARHQ in 2016 (Pillen et al. 2016). This paper reports on findings from the
process evaluation—which focused on lessons learned in implementing
PSML interventions. Data for the process evaluation were derived from site
visits conducted at two points in time (early and late in the demonstration) by
multidisciplinary teams led by senior qualitative researchers using a semistructured interview guide and consisting of interviews with the project leadership and a broad range of stakeholders in each of the seven demonstration
sites. Site visit interview data were supplemented with annual follow-up telephone interviews with the project leaders and material abstracted from annual
project reports to AHRQ. In the next two sections, we describe our findings
and observations, supported by these interview and documentary data.
F INDINGS FROM PSML DEMONSTRATION S ITES
The PSML demonstration projects focused on three broad approaches to
improving patient safety and reducing medical liability: (1) improving
provider–patient communication around adverse events through disclosure
and resolution programs, (2) preventing harm through the implementation of
clinical “best practices,” and (3) exploring alternative methods of settling
claims.
PSML Demonstration Cross-Site Evaluation
2417
Improving Provider–Patient Communication through Disclosure and Resolution
Programs
Four of the PSML demonstration grantees—New York State Unified Court
System, University of Illinois Medical Center at Chicago, University of Texas,
and University of Washington—provided training on error disclosure and
implemented disclosure and resolution programs (DRPs), which were loosely
based on a model developed at the University of Michigan Health System
(Boothman et al. 2009; Boothman, Imhoff, and Campbell 2012). Under a
DRP, health care professionals and institutions disclose adverse outcomes to
patients and families; investigate and explain what happened; use that knowledge to improve patient safety and prevent the recurrence of such incidents;
and, when appropriate, apologize and offer fair financial compensation to the
injured party (or their family). A DRP seeks to reduce litigation pressure by
enhancing open communication between provider and patient in the wake of
an adverse event and facilitating an offer of compensation, when appropriate,
without the need for the patient or family to resort to litigation. A DRP can
also improve patient safety outcomes if it is set up to identify patient safety
problems that are then prioritized and addressed by a well-functioning patient
safety improvement infrastructure in the hospital. As argued by Boothman,
the intent of the DRP is to improve transparency and accountability to
patients and families—not just to improve the efficiency of malpractice claims
management (Boothman 2016).
The PSML experience showed that building robust DRPs involves serious challenges for both implementation and evaluation. For example, convincing hospitals to establish and abide by a DRP, when that program did not
originate from within the hospital itself, requires nontrivial effort, significant
culture change, and serious financial and infrastructure commitments.
Although DRP interventions have the potential to reduce malpractice risk,
they also have the potential to shift risk in complex ways among a hospital, its
physicians, outside liability insurer(s), and patients. Various difficulties in
aligning the interests of these stakeholder groups, and in building institutional
support for DRP adoption, were a challenge described by several of the
PSML demonstration projects. The challenge was not fully overcome by any
of them.
Thus, although there is accumulating empirical evidence (including
some from the PSML portfolio) that DRPs can positively affect both patient
safety and malpractice outcomes (Kraman and Hamm 1999; Hersch, O’Connell, and Viscusi 2006; Boothman et al. 2009; Kachalia et al. 2010;
2418
HSR: Health Services Research 51:6, Part II (December 2016)
McDonald et al. 2010; Helmchen, Lambert, and McDonald 2016; Lambert
et al. 2016; Mello et al. 2016b), there is also ample evidence from the PSML
portfolio to suggest that DRPs are difficult to scale, to disseminate, and to
implement across multiple institutions and in nonacademic settings (Mello
et al. 2014). For example, in the New York project, hospitals implemented the
“disclosure” part of the model but struggled to implement “resolution” when it
involved making an unsolicited offer of compensation. Fear lingered that
introducing a discussion of compensation would encourage litigation (Mello
et al. 2016b). The Illinois project demonstrated the impact of “Seven Pillars”
in its academic medical setting (Lambert et al. 2016) but failed to produce
impact data for the replication of the model across the 10 participating community hospitals. In the Washington project, very few claims were referred to
the nascent DRPs—starving the project of any demonstrable quantitative
impact (Mello et al. 2016a). The Texas project implemented in a state with a
“low-tort” environment—where the malpractice litigation risk for a public university was already diminished substantially (Sage, Harding, and Thomas
2016). The fact that challenges were encountered across the PSML portfolio
of demonstration projects in no way diminishes the accomplishments of individual projects, but the failure to fully implement in nonacademic settings
leaves an unanswered policy question. At present, there is no empirical evidence for speculating on the outcomes that reasonably could be expected
from DRP implementation across a broad range of hospital types (such as academic medical center versus community hospital, fully insured versus selfinsured, and urban versus rural) because the PSML demonstration program
was unable to implement DRPs at scale in nonacademic settings and because
no other studies that have included a broad range of hospital types have been
published to date.
Preventing Harm through Best Practices
Three PSML grantees—Ascension Health, Fairview Health Services, and the
Massachusetts State Department of Public Health—aimed to prevent medical
errors and poor health outcomes while reducing malpractice lawsuits by
implementing clinical “best practices.” Two projects (Ascension Health and
Fairview Health Services) focused on spreading clinical best practices for
safety interventions to hospital-based perinatal units and obstetrics departments. One project (Massachusetts State Department of Public Health)
focused on increasing the efficiency and efficacy of high-risk clinical and communication processes (for example, making sure laboratory test results are
PSML Demonstration Cross-Site Evaluation
2419
communicated to the ordering clinician and to the patient) in a group of ambulatory primary care practices (Schiff et al. 2016).
Several broad observations about these projects are salient. First is that
two of the three demonstration projects achieved some positive impacts on
patient safety outcomes (Burstein et al. 2016; Riley et al. 2016a,b). Second,
although these demonstration projects also intended to assess the impact of
best practice adoption on malpractice claiming, only one of the projects was
successful in testing this possibility because of varied resource, data, and
empirical design limitations. One project documented a significant reduction
in perinatal claims in participating hospitals—while there was no significant
decrease in nonperinatal claims in the same hospitals (Riley et al. 2016b).
Finally, as with the DRPs described above, these grantees also faced some
serious implementation challenges in scaling up on a multi-institution basis.
Exploring Alternative Methods of Settling Claims
One of the PSML demonstration projects sought to improve the dispute resolution process after the filing of a malpractice claim. The New York State Unified Court System worked with the New York State Department of Health and
five academic medical centers on a multifaceted approach to improving
patient safety and reducing malpractice litigation. It was originally designed
with the idea that patient safety interventions in hospitals would reduce
adverse events and thus claims. The thinking was that DRPs would be utilized
by each of the hospitals to disclose adverse events, and the investigations
would lead to litigation-reducing decisions by the hospitals, including providing apologies and offers of compensation. But if claims were nevertheless filed
through the court system, a “judge-directed negotiation” program, expanded
from an existing program (McKeon 2011–2012), would expedite settlement of
those claims. Judge-directed negotiation focuses on early court intervention
with the presiding judge facilitating negotiation among attorneys about claims
and potential settlements. This type of early intervention accelerates the
movement of cases through the claims process; increases the number of settlements; and, over time, lowers malpractice costs and premiums for the participating hospitals.
Hospital leadership and staff were enthusiastic about implementing
judge-directed negotiation. Participation was viewed as both helpful and cost
reducing. Anecdotal accounts have credited the program with decreasing the
number of malpractice claims and annual costs to participating hospitals by a
2420
HSR: Health Services Research 51:6, Part II (December 2016)
material amount (Gamboa 2013); however, as of this date no results from the
formal evaluation have been published.
OBSERVATIONS AND S UGGESTIONS TO I MPROVE
F UTURE E FFORTS
A natural question to ask is, what have we learned across the PSML demonstration projects? Each demonstration grantee was funded by AHRQ to collect data on the impact of its intervention(s). Some grantees focused on
assessing the impact of interventions on malpractice outcomes, but others
focused their assessment on patient safety outcomes such as changes in patient
safety culture in institutions. All seven projects encountered challenges—
some expected and others unexpected—that stretched project resources and
required adjustments to implementation and evaluation expectations and
strategies. Nevertheless, the projects had many accomplishments, such as
developing and refining trainings, tools, products, and data collection instruments and contributing valuable lessons about what it takes to develop and
sustain an operational patient safety and medical liability program. However,
in spite of these achievements and development of useful products, a number
of the grantees struggled to produce outcomes data that were timely in collection, inclusive of patient safety and medical liability outcomes pertinent to the
interventions being piloted, and sufficiently robust (i.e., with comparison populations) to support causal inference. A few observations below help to
explain the grantee experiences.
Efforts to Intervene in Patient Safety and Malpractice Liability Are Complex and
Varied
In its RFP, AHRQ deliberately set out a broad agenda for this body of work,
inviting applicants to proffer diverse ideas. Applicants for the PSML demonstration grants were required to address both patient safety and malpractice liability outcomes. However, AHRQ did not specify how patient safety and
medical liability were supposed to fit together in the demonstration projects,
and the grantees varied considerably in how they sought to bridge those
issues. Because there is more than one way to intervene with patient safety and
malpractice risk, PSML demonstration projects are not directly comparable
to one another, nor are they all aimed at affecting the same outcome variables.
PSML Demonstration Cross-Site Evaluation
2421
Conceptually, patient safety performance in a hospital is both a precursor
to malpractice litigation (i.e., through the occurrence of adverse events that
give rise to litigation) and also a consequence of malpractice litigation (which can
either serve to harden adversarial positions or promote change). Interventions
to address PSML are similarly varied. Efforts to improve patient safety outcomes can, in principle, lead to corresponding reductions in malpractice
claiming. And likewise, efforts to reduce malpractice litigation can, in principle, help to strengthen provider efforts to address the system failures that give
rise to adverse events. Meanwhile, each of these conceptual vectors is itself a
complex pathway. The PSML demonstration program was noteworthy for
the ambition to intervene on both pathways, but also for the lack of clarity in
distinguishing between them.
Loosely speaking, the clinical patient safety interventions that were
implemented by several of the PSML projects (e.g., best practices in obstetrics) can be understood as addressing the first causal pathway, while the DRPs
can be understood as addressing the second. Notably, efforts to document the
impact of any of these interventions require adequate time and robust data.
Both time and data limitations proved to be very challenging for the PSML
demonstrations, given the way that these were structured. Future PSML
demonstration and evaluation efforts will be strengthened by building in a
considerably longer time period for implementation and data-gathering
efforts and striving to improve PSML data infrastructure and common
measures on a regional and national basis, as a fundamental building block for
future intervention efforts along these lines.
There Is No Single, Most Relevant Set of Measures for Capturing PSML Outcomes
across Diverse Studies
Expanding on the last point about data infrastructure, it is important to recognize that future PSML interventions are unlikely to be reducible to a single set
of common outcome measures across diverse clinical settings, widely varying
types of interventions, and competing causal perspectives that either try to
influence malpractice outcomes through patient safety interventions or patient
safety outcomes through malpractice interventions. In light of the complexity
of this landscape, there is no single data pipeline for capturing relevant measures in a consistent way. Patient safety interventions may be focused on narrow clinical contexts (e.g., reducing preventable injuries in labor and delivery)
that put boundaries on the relevant outcome metrics, in ways that are both
challenging to capture and may legitimately involve the use of
2422
HSR: Health Services Research 51:6, Part II (December 2016)
noncomparable patient safety outcome measures across studies. Meanwhile,
our search for a national source for meaningful and consistent data on malpractice claims failed to uncover one—other than the National Practitioner
Data Bank, which has some significant limitations (Mello 2016).
Given these sorts of conceptual and architectural challenges, it is
important that those who decide to field PSML interventions in the future
adopt reasonable starting assumptions about data and measurement.
Future studies could be facilitated by starting from a clear understanding
of the data that are readily available to assess key outcomes of interest,
and how those data correspond to any existing regional or national
resource for warehousing similar data. Here again, improving the national
data infrastructure on patient safety and malpractice performance could
offer a focal point for future efforts, and particularly so with regard to data
on malpractice claims.
“Time” as a Basic Design Challenge for PSML Intervention Projects
One of our earliest observations from the struggles of the PSML grantees
involved the problem of the malpractice “claims tail” (i.e., that malpractice
claims frequently take years from discovery to claim to resolution—whether
through settlement or litigation). Outcomes research on malpractice claims,
consequently, requires years of observation and data collection to carry out,
quite apart from any time that is involved in actually fielding a PSML intervention. When taken together with the time that is required to start up a DRP
or best practice intervention in multiple sites and to gather baseline performance data, the implication is that a PSML demonstration project is more
likely to require 6–10 years, rather than 3–4 years (which was the PSML
demonstration grant period).
CONTRIBUTIONS OF THE PATIENT SAFETY AND
M EDICAL LIABILITY I NITIATIVE
What can the health care and policy communities learn from the PSML initiative? Importantly, PSML grantees generated a plethora of tools, training modules and curricula, program models, evaluation instruments, videos, written
products, and other materials that have the potential to spur or assist in the
replication of these PSML interventions elsewhere—and in the development
of new PSML innovations.
PSML Demonstration Cross-Site Evaluation
2423
Additionally, preliminary findings from several of the PSML demonstration projects suggest positive (or at least promising) outcomes associated
with some facet of the interventions, either in terms of patient safety performance, malpractice claiming, or both. These findings suggest that under the
right conditions, PSML interventions can have a positive impact on various
facets of malpractice and patient safety performance within the health care
and civil justice systems. An equally important aspect of the learning generated by the PSML demonstration involves the implementation barriers and
challenges they encountered and their efforts to overcome them. The reasons
why the PSML demonstrations were difficult to carry out may be at least as
instructive to future efforts in this arena as are the tools and documented
successes of the PSML portfolio itself.
It is important to point out that the initial plans for the PSML portfolio
(dating back to AHRQ’s original Request for Proposals) implicitly assumed
that (1) all the PSML projects would tie patient safety and malpractice liability
together; (2) national data resources for both patient safety and malpractice
outcome variables would be available and appropriate for the projects to draw
upon for use in the evaluation; and (3) the 3-year grant period would allow sufficient time to implement the demonstration projects and collect and analyze
data on all relevant outcomes. None of these initial assumptions proved to be
true. Partly in consequence, results across the PSML portfolio cannot easily
be reduced to a simple quantitative “box score” or effect size estimate.
Notwithstanding the fact that some initial expectations for the PSML
demonstrations were unrealized and formal outcomes data and analysis from
the PSML projects are limited (as reflected by the papers in this special issue),
the projects have nevertheless contributed many useful and important lessons
to the field’s knowledge base. For example, several of the PSML projects did
useful piloting, replication, and dissemination work on DRPs. The projects
helped to identify the conditions under which such programs can readily be
adopted and conditions under which their adoption becomes more difficult.
The projects showed that under optimal conditions, DRPs can produce measurable, positive impacts on a series of patient safety and medical liability outcome measures (Lambert et al. 2016). The projects also showed that under
suboptimal conditions, DRPs can be quite difficult to implement; different
stakeholder groups may have understandably different perspectives regarding
the attractiveness and risk implications of DRP (Mello et al. 2016a,b); and the
early offer (or “resolution”) component of DRPs tends to be more difficult to
carry out than the disclosure (or “communication”) component (Mello et al.
2016a,b).
2424
HSR: Health Services Research 51:6, Part II (December 2016)
Meanwhile, several of the projects generated positive preliminary findings of improved safety practices based on adoption of clinical “best practices.” With this being acknowledged, future efforts to prevent harm through
adoption and diffusion of best practices will likely confront two fundamental
problems. One involves documenting that such interventions go beyond
improving patient safety outcomes to show that clinical best practice interventions are also associated with improvements in malpractice outcomes for providers. A second problem is that best practice interventions tend to be specific
to a particular clinical context. Thus, future studies of best practice adoption
will likely be needed in many different clinical domains to gradually build an
empirical base for broadly concluding that such interventions “work” to protect physicians from malpractice litigation and their patients from adverse
events.
Finally, several of the PSML demonstration projects fundamentally
sought to influence one or more elements of institutional culture (i.e., collective attitudes, practices, beliefs). Across the PSML projects, the nature of the
“culture” focus varied. Regardless, hospital “culture” is interesting as a mediating variable in that it does not directly translate into either patient safety or
malpractice outcomes, but nevertheless has the theoretical potential to influence both. The work of the PSML demonstration projects in carrying out
innovations such as measuring “disclosure” culture (Etchegaray et al. 2012);
collaborating with regulators to respond to adverse events (Gallagher et al.
2016a); and involving patients and family members in disclosure and resolution efforts (Etchegaray et al. 2016; Gallagher et al. 2016b) could offer a lodestar to future efforts in this area, and consequently to the design of new
interventions in the patient safety and malpractice policy space.
CONCLUSION
In the wake of the PSML initiative, continuing efforts to reduce preventable
adverse events will remain a high priority for the U.S. health care system.
Meanwhile, concerns about malpractice liability and cost will likewise continue to generate debate and controversy among policy makers. In our view,
what was most innovative about the PSML portfolio was the simple idea that
these two problems can be understood as fundamentally related and (therefore) addressed simultaneously. This basic idea comports well with the last
decade of the U.S. patient safety movement and the underlying belief that
adverse events in health care are better understood as the result of complex
PSML Demonstration Cross-Site Evaluation
2425
“systems” failure, rather than of individual negligence. In part, the response of
the health care system has been to try to analyze adverse events, prevent their
recurrence, and to become more transparent and responsive while carrying
out related self-improvement processes. None of this logic aligns well with traditional malpractice liability, which instead assumes fault on the part of individual providers, and which creates an adversarial incentive to defend and
protect against potential claims, rather than to collaborate in investigating and
reducing risk. Whatever viewpoint one has regarding the merits of statutory
tort reform, one thing most people can agree on is that if other factors are held
constant, it would be a good idea to reduce preventable injury rates in health
care settings.
The PSML portfolio has demonstrated that related, large-scale interventions can be fielded and such interventions can produce at least preliminary
positive outcomes in some important ways. The PSML demonstration also
lends credence to the idea that targeted interventions that improve some
aspect of patient safety or malpractice performance may also contribute more
broadly to institutional culture and to the alignment of all parties in the system
around reducing risk and preventing harm. Analytically, more work needs to
be carried out to quantify the effects of PSML interventions in the future. And
practically, new PSML interventions will need to demonstrate that they can
overcome substantial implementation barriers in order to scale and to achieve
meaningful systems change in new locations and settings.
President Obama’s purpose in authorizing the PSML initiative was not
to focus on traditional malpractice reform as a “silver bullet” but instead to
stimulate the development of a range of innovative ideas that put “patient
safety first and let doctors focus on practicing medicine” (Obama 2009). The
legacy of the PSML initiative is that it accomplished the goal of addressing the
challenges of the current medical liability system using a nontraditional
approach. The PSML initiative empowered states and health care organizations to think “outside the box” about innovative solutions that would reduce
preventable injuries, foster better communication between doctors and
patients, and award compensation to injured patients in a timely and efficient
manner. Whereas the quantitative results on the impact of the innovations are
equivocal, in part because of time and resource limitations, the products available from the grantees and the lessons learned from the PSML demonstration
program provide direction for policy makers, health care institutions, and
practitioners to continue on the path initiated by the President. Much work
has been carried out; much more work remains. Some of this work can be
2426
HSR: Health Services Research 51:6, Part II (December 2016)
funded by AHRQ as an extension of its current activities—but other funders
need to be involved.
The current portfolio of PSML demonstration grants can reasonably
be viewed as the first step in a longer term effort to build replicable models
as well as a national capacity for tracking and evaluating new policies and
programs on medical liability and patient safety. A reasonable next step is
to incorporate what was learned through the PSML demonstration—and
subsequent efforts—to identify the most promising models for a much more
rigorous “road test” in multiple host organizations around the country. In
any future demonstrations, a particular focus should be placed on (1) planning for, monitoring, and troubleshooting implementation of the chosen
model, (2) creating a realistic timeline for the grants that allows for full program implementation before data collection begins, (3) designing an overarching evaluation that will yield high-quality evidence of effectiveness, and
(4) choosing investigators and host organizations that have a demonstrated
capacity for longitudinal data collection.
Future funders of this work (including AHRQ) also need to proactively
address the data challenges. First, the field may need more investment and
national infrastructure development to improve common datasets and measures relating to PSML outcomes, particularly around malpractice claiming.
Second, at least in the near term, future PSML demonstration projects are
likely to continue collecting their own primary data, based on decisions
upfront about the most relevant and useful outcome measures. In fact, a number of the PSML demonstration projects invested substantial resources to create data collection instruments that can be repurposed for future initiatives;
however, researchers working with clinicians and risk managers need to
reduce the burden of such data collection so that monitoring can be sustained
in “real-world” host organizations after the grant period is over.
In addition, future work on “best practices” would be a natural extension of AHRQ’s existing efforts to disseminate and encourage the uptake of
models (such as patient safety “bundles”) aimed at reducing variation in
care and improving patient safety. Investigators could be encouraged to
focus on risks that contribute to large numbers of medical errors and to
investigate the effects of remediating interventions on malpractice as well as
clinical outcomes.
Finally, two of the PSML demonstration projects together explored the
questions, “What should be the role (if any) for patients and their family members in adverse event reporting, investigation, and remediation? What can
patients and family members (potentially) add to the information that
PSML Demonstration Cross-Site Evaluation
2427
hospitals already gather in root cause analysis and other investigatory processes? What is unique (if anything) about the contribution the patient voice
can make? These intriguing questions have been raised, but not fully
answered, by the work of these grantees. More emphasis on—and exploratory
work examining—the role of patients and families in the investigation and
remediation of patient safety problems is long overdue.
ACKNOWLEDGMENTS
Joint Acknowledgment/Disclosure Statement: AHRQ commissioned James Bell
Associates, Inc., in partnership with the RAND Corporation, to conduct an
overarching, independent evaluation of the Patient Safety and Medical Liability Initiative. This research was conducted under contract #HHSA2902007
10073T. The views expressed in this paper are those of the authors and are not
intended to represent the views of either AHRQ or HHS. The authors are
deeply grateful to Jim Battles, AHRQ Project Officer, and Kate Reback, Irim
Azam, and Karen Migdail for their support and guidance at all stages of the
project. We also thank the principals, colleagues, and collaborators of the
PSML demonstration programs for sharing their insights with us. Finally, we
acknowledge with gratitude the contributions to the project and this paper of
David de Vries, Frances Aunon, and Rebecca Shaw of RAND and Elizabeth
Hayes, Natasha Driver, and Anna Hodgson of James Bell Associates.
Disclosures: None.
Disclaimers: None.
REFERENCES
Battles, J., K. A. Reback, and I. Azam. 2016. “Paving the Way for Progress: The AHRQ
Patient Safety and Medical Liability Demonstration Program.” Health Services
Research 51 (6 Pt 2): 2401–13.
Boothman, R. C. 2016. “Commentary: CANDOR: The Antidote to Deny and
Defend?” Health Services Research 51 (6 Pt 2): 2487–90.
Boothman, R. C., S. J. Imhoff, and D. A. Campbell. 2012. “Nurturing a Culture of
Patient Safety and Achieving Lower Malpractice Risk through Disclosure: Lessons Learned and Future Directions.” Frontiers of Health Services Management 28
(3): 13–28.
Boothman, R. C., A. C. Blackwell, D. A. Campbell, E. Commiskey, and S.
Anderson. 2009. “A Better Approach to Medical Malpractice Claims? The
2428
HSR: Health Services Research 51:6, Part II (December 2016)
University of Michigan Experience.” Journal of Health & Life Sciences Law 2:
125–59.
Burstein, P. D., D. M. Zalenski, J. Edwards, I. Rafi, J. Darden, C. Firneno, and P. Santos.
2016. “Changing Labor and Delivery Practice: Focus on Achieving Practice and
Documentation Standardization with the Goal of Improving Neonatal Outcomes.” Health Services Research 51 (6 Pt 2): 2472–86.
Etchegaray, J. M., T. H. Gallagher, S. K. Bell, B. Dunlap, and E. J. Thomas. 2012.
“Error Disclosure: A New Domain for Safety Culture Assessment.” BMJ Quality
and Safety 21: 594–9.
Etchegaray, J. M., M. J. Ottosen, A. Aigbe, E. Sedlock, W. M. Sage, S. K. Bell, T. H.
Gallagher, and E. J. Thomas. 2016. “Patients as Partners in Learning from Unexpected Events.” Health Services Research 51 (6 Pt 2): 2600–614.
Gallagher, T. H., M. L. Farrell, H. Karson, S. J. Armstrong, J. T. Maldon, M. M. Mello,
and B. F. Cullen. 2016a. “Collaboration with Regulators to Support Quality and
Accountability Following Medical Errors: The Communication and Resolution
Program Certification Pilot.” Health Services Research 51 (6 Pt 2): 2569–82.
Gallagher, T. H., J. M. Etchegaray, B. Bergstedt, A. M. Chappelle, M. J. Ottosen, E. W.
Sedlock, and E. J. Thomas. 2016b. “Improving Communication and Resolution
Following Adverse Events Using a Patient-Created Simulation Exercise.” Health
Services Research 51 (6 Pt 2): 2537–49.
Gamboa, K. 2013. “Positive Prognosis for Judges: A Look into Judge-Directed Negotiations in Medical Malpractice Cases.” Pepperdine Dispute Resolution Law Journal 13
(2): 235–55.
Helmchen, L. A., B. L. Lambert, and T. B. McDonald. 2016. “Changes in Physician
Practice Patterns after Implementation of a Communications and Resolution
Program.” Health Services Research 51 (6 Pt 2): 2516–36.
Hersch, J., J. O’Connell, and W. K. Viscusi. 2006. “Evaluation of Early Offer Reform
of Medical Malpractice Claims: Final Report.” Prepared by The Lewin Group
for the Office of Disability, Aging and Long-Term Care Policy, Office of the
Assistant Secretary for Planning and Evaluation, U.S. Department of Health and
Human Services, Contract #HHS-100-03-0027.
Kachalia, A., S. Kaufman, R. Boothman, S. Anderson, K. Welch, S. Saint, and
M. Rogers. 2010. “Liability Claims and Costs before and after Implementation of a
Medical Error Disclosure Program.” Annals of Internal Medicine 153 (4): 213–21.
Kraman, S. S., and G. Hamm. 1999. “Risk Management: Extreme Honesty May Be the
Best Policy.” Annals of Internal Medicine 131: 963–7.
Lambert, B. L., N. M. Centomani, K. M. Smith, L. A. Helmchen, D. K. Bhaumik,
Y. Jalundhwala, and T. B. McDonald. 2016. “The Seven Pillars Response to
Patient Safety Incidents: Effects on Medical Liability Processes and Outcomes.”
Health Services Research 51 (6 Pt 2): 2491–515.
McDonald, T. B., L. A. Helmchen, K. M. Smith, N. Centomani, A. Gunderson, D.
Mayer, and W. H. Chamberlin. 2010. “Responding to Patient Safety Incidents:
The “Seven Pillars.” Quality & Safety in Health Care 19 (6): 1–4.
McKeon, D. E. 2011–2012. “New York’s Innovative Approach to Medical Malpractice.” New England Law Review 46: 475–87.
PSML Demonstration Cross-Site Evaluation
2429
Mello, M. M., and D. M. Studdert. 2016. “Building a National Surveillance System for
Malpractice Claims.” Health Services Research 51 (6 Pt 2): 2642–48.
Mello, M. M., S. K. Senecal, Y. Kuznetsov, and J. S. Cohn. 2014. “Implementing Hospital-Based Communication-and-Resolution Programs: Lessons Learned in New
York City.” Health Affairs 33 (1): 30–8.
Mello, M. M., S. Armstrong, Y. Greenberg, P. I. McCotter, and T. H. Gallagher. 2016a.
“Challenges of Implementing a Communication-and-Resolution Program
Where Multiple Organizations Must Cooperate.” Health Services Research 51 (6 Pt
2): 2550–68.
Mello, M. M., Y. Greenberg, S. K. Senecal, and J. S. Cohn. 2016b. “Case Outcomes in
a Communication and Resolution Program in New York Hospitals.” Health Services Research 51 (6 Pt 2): 2583–99.
Obama, B. September 9, 2009. Obama Health Care Speech: Full Video, Text.
[accessed on October 15, 2016]. Available at http://www.huffingtonpost.
com/2009/09/09/obama-health-care-speec h_n_281265.html
Pillen M., M. S. Ridgely, M. D. Greenberg, E. Hayes, N. Driver, A. Hodgson, J. Bell,
D. de Vries, F. M. Aunon, and R. Shaw. 2016. “Longitudinal Evaluation of the
Patient Safety and Medical Liability Reform Demonstration Program: Demonstration Grants Final Evaluation Report.” Prepared under Contract No.
HHSA290200710073T (PRISM Order #HHSA29032001T). AHRQ Publication No. 16-0038-2-EF. Rockville, MD: Agency for Healthcare Research and
Quality; May 2016 [accessed on October 15, 2016]. Available at http://www.
ahrq.gov/professionals/quality-patient-safety/patient-safety-resources/resources/
candor/demo-program-eval/demo-grants.html
Riley, W., J. W. Begun, L. Meredith, K. K. Miller, K. Connolly, R. Price, J. H. Muri, M.
McCullough, and S. Davis. 2016a. “Integrated Approach to Reduce Perinatal
Adverse Events: Standardized Processes, Interdisciplinary Teamwork Training,
and Performance Feedback.” Health Services Research 51 (6 Pt 2): 2431–52.
Riley, W., L. Meredith, R. Price, K. K. Miller, J. W. Begun, M. McCullough, and S.
Davis. 2016b. “Decreasing Malpractice Claims by Reducing Preventable Perinatal Harm.” Health Services Research 51 (6 Pt 2): 2453–71.
Sage, W. M., M. C. Harding, and E. J. Thomas. 2016. “Resolving Malpractice Claims
After Tort Reform: Experience in a Self-Insured Texas Public Academic Health
System.” Health Services Research 51 (6 Pt 2): 2615–33.
Schiff, G. D., H. R. Nieva, P. Griswold, J. Ling, M. Biondolillo, and S. J. Singer. 2016.
“Addressing Ambulatory Safety and Malpractice: The Massachusetts PROMISES Project.” Health Services Research 51 (6 Pt 2): 2634–41.
Stufflebeam, D. L. 2003. “The CIPP Model for Evaluation.” In The International Handbook of Educational Evaluation (Chapter 2), edited by D. L. Stufflebeam and
T. Kellaghan. Boston, MA: Kluwer Academic Publishers.
2430
HSR: Health Services Research 51:6, Part II (December 2016)
S UPPORTING I NFORMATION
Additional supporting information may be found in the online version of this
article:
Appendix SA1: Author Matrix.
Copyright of Health Services Research is the property of Wiley-Blackwell and its content
may not be copied or emailed to multiple sites or posted to a listserv without the copyright
holder’s express written permission. However, users may print, download, or email articles for
individual use.