Skip Navigation

GME Evaluation Task Force Recommendation

Back to GME Evaluation
and Assessment Portal
July 1, 2008

PATIENT CARE




PATIENT CARE: FOCUSED ASSESSMENT OF OBSERVED SKILLS
Evidence of Competency In:

Surgical Skills, Procedural Skills, Specific Communication Skills

Recommended Assessment Tool

Unlike the shared general competencies, Patient Care objectives are defined by each RRC. Thus, no specific tool can be recommended across GME programs. Instead, we offer principles to guide the development or selection of tools. Checklists define the discrete tasks that comprise the overall skill being assessed. Scales evaluate procedural skills that transcend individual tasks, such as knowledge of anatomy, instrument handling, flow of operation, etc. Richard Reznick and colleagues at the University of Toronto developed scales for the Observed Structured Assessment of Technical Skills, or O-SATS (1,2).

Examples of skill checklists are included in the Appendix.
Overview

Focused assessments for skills are used primarily to assess patient care by determining if a psychomotor skill has been acquired. It is possible to combine some communication items if this is the only time a resident interacts with patients, but it is not the primary use of this assessment. It is also possible to develop a focused assessment of specific communication skill tasks, such as an informed consent discussion or specific counseling following a practice guideline.

Reliability and Validity

Skill checklists primarily have content validity. The items for a specific checklist may have come from the literature where someone has decided the checklist has content validity. Alternatively, if designing a checklist within the program, having those who are “expert” in the skill review and approve the checklist would serve as a level of validity. Reliability exists in several dimensions: the ability of different assessors to come to the same decision (inter-rater reliability) and the internal consistency of the checklist items (do they “fit” together). However, if checklists are used to identify when someone has mastered the skill, the internal consistency is not relevant (since everyone should get 100% eventually). Therefore, the best reliability evidence would be consensus of faculty concerning the decision that the resident is competent in that skill.

Administration
  • Timing: Skill assessments should be performed until a resident can demonstrate competency. The sustained level of competency can be measured if a program is worried about “drift” from desired performance. This would require a recheck of the skill at some systematic interval.
  • Who Performs: Skill assessments should be done through observation of the actual performance. It is possible that a faculty member could assess through video review of the performance, but the assessment reflects the skill of the resident on the performance date.
  • Format: A checklist is the most appropriate format, but the checklist may have some gradation reflecting the quality with which the specific step was performed, e.g.: not indicated (n/a), not performed but indicated, performed poorly, or performed well. General scales (such as for O-SATS above) also exist and facilitate comparability across specific procedures. Written comments may be especially helpful for giving feedback.
  • Scoring Criteria and Training: There should be guidelines for the checklist describing the environment for the assessment and a description that accompanies what is meant by each step on the checklist. For example, if the checklist includes “washes hands”, does that mean a resident running his/her hands under water without soap is acceptable? Is scrubbing involved? For a minimum length of time? It is advisable to indicate to learners and evaluators the acceptable standard for checklist items. The training could be by reviewing the written guide. The checklist should contain a written standard by which the resident would know that the performance demonstrated competency. Generally, this would mean achieving 100% of the checklist items and/or overall judgment of competency by the assessor.
  • Documentation: At minimum, twice annually as part of semi-annual review meetings.

    • Workflow Procedures

    A systematic approach is recommended to maximize the use of the focused assessments and facilitate data management. A sample workflow document for focused assessment of surgical skills follows.

    References

    1. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a structured technical skills assessment form. Am J Surg1994;167(4):423-7.
    2. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg 1997;173(3):226-30.

    Patient Care Focussed Assessment Form
    Patient Care: Focussed Assessment of Observed Skills
    Core Measure for UCSF GME

    Appendix C
    Examples of Focused Assessment Tools, Skills Checklist

    E*Value Forms
    (pending)

    Go to Patient Care - CEX Forms
    If you have questions and/or requests about the recommended tools, or how to incorporate these forms into your clinical training program evaluation system (E*Value), please contact: kapurg@medsch.ucsf.edu

    Back to GME Assessment & Evaluation Portal

    Curriculum Home


    Graduate Medical Education
    500 Parnassus Avenue, MU 250 East
    San Francisco, CA 94143-0474
    (415) 476-4562