Skip Navigation


GME Evaluation Task Force Recommendation

Back to GME Evaluation
and Assessment Portal
July 1, 2008

SYSTEMS BASED PRACTICE

Overview

The evaluation of systems-based practice poses unique challenges because of the variation in learning and assessments opportunities among GME programs. We recommend that programs first identify where residents are routinely called upon to demonstrate systems based practice skills and then select methods for evaluating and documenting these skills. We list examples of these activities and how they might be assessed.

Morbidity and Mortality Conference Presentation

These occur on a regular basis in many residency programs and include resident presentations of clinical cases with an assessment of how care could be improved. If this assessment routinely includes an analysis of the system’s role in prevention of medical errors and harms, the resident’s analysis can be evaluated using a form or checklist.

Quality Improvement Meeting

Resident involvement in quality improvement activities is an ACGME requirement. If a resident routinely participates in a clinical quality improvement meeting during a particular PGY year or rotation, their participation can be assessed by peers or supervisors on the committee.

Cost-Effective Care Practices

These skills are demonstrated when residents make decisions about screening tests, diagnostic tests treatments, and site of care decisions. If residents routinely present their care plans to faculty in rounds, morning report, conferences (pre-op, tumor board) or other venues, the faculty can evaluate the degree to which the resident demonstrates an awareness of cost-effectiveness.

Quality Improvement Project

There are many venues where individual residents or groups of residents can identify a process or outcome they wish to improve, develop a plan that includes consideration of system issues, implement the plan and measure its success. The full project or any of its components could be assessed by experts in quality improvement. The Quality Improvement Project Assessment Tool (QIPAT-7) is provided below as on example of how a resident project can be assessed. The QIPAT-7 was developed based upon the input of national QI experts and its reliability has been demonstrated in an evaluation of 45 resident QI proposals (Leenstra 2007).


Figure 1. Quality Improvement Proposal Assessment Tool (QPAT-7). The scale is anchored to the bulleted comments on the left. To achieve a score of 3 or higher, all bullets for each domain must be met. The box sizes for each point of the scale are simply determined by the heading labels; that is, smaller boxes do not indicate smaller intervals between scale steps.

The HealthCare Matrix from Vanderbilt uses quality aims from the Institute of Medicine and the ACGME competencies together to assess and improve care. The Matrix is described further in this abstract from the ACGME eBulletin from December 2006, page 10-11 (Bingham 2005).

Using a Healthcare Matrix to Assess Care in Terms of the IOM Aims and the ACGME Competencies, Doris Quinn PhD, John Bingham MHA, Vanderbilt University Medical Center.

The study assessed how residents and faculty are using the HealthCare Matrix to assess and improve care. Whether care is safe, timely, effective, efficient, equitable, or patient-centered is juxtaposed against the ACGME competencies. When care is assessed in this manner, learning the competencies becomes very relevant to the outcomes of care.

It presented the work of internal medicine residents who on their Ambulatory Rotation:
1) utilized the Matrix to assess the care of their patients;
2) demonstrated use of QI tools to improve care; and
3) improved publicly reported metrics for AMI and CHF by focusing in particular, system-based practice and practice-based learning and improvement. Residents first utilize the Matrix to assess care of one of their patient’s. Then, as a group, they choose a publicly reported metric and complete matrices for a panel of patients.

The data from the matrices informs residents as to where more information or improvement is needed. This becomes the basis for an improvement project which is ultimately presented to senior leaders. To date, residents have improved the care of patients with pneumonia, coronary artery disease, diabetes, and processes including obtaining consults, the VA phone Rx system and others. Public metrics of quality from CMS, JCAHO, and Leapfrog are utilized in the assessment. When the ACGME competencies are combined with the IOM aims and used to assess and improve care of patients in “real time”, developing the competencies becomes “the way residents learn” and not a burden or “add on”. This process allows residents, who are the most knowledgeable about workarounds and flaws in the system, to use their experience to improve care. Residents, faculty, the institution, and most importantly, the patients benefit.

References

Bingham JW, Quinn DC, Richardson MG, Miles PV, Gabbe SG. Using a healthcare matrix to assess patient care in terms of aims for improvement and ACGME core competencies. JC Journal on Quality and Patient Safety 2005;32(2): 98–105.

Leenstra JL, Beckman TJ, Reed DA, Mundell WC, Thomas KG, Krajicek BJ, Cha SS, Kolars JC, McDonald FS. Validation of a method for assessing resident physicians' quality improvement proposals. J Gen Intern Med. 2007 Sep;22(9):1330-4. Epub 2007 Jun 30.

SYSTEMS BASED PRACTICE

Back to GME Evaluation and Assessment Portal

Curriculum Home