Article:

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones

Summary:

Milestones are assessments in graduate medical education designed to provide longitudinal data about learners’ progression towards independent practice.  These were instituted within the last decade, after being mapped to the six core competencies set forth by the Accreditation Council of Graduate Medical Education (ACGME).  Dr. Heath et al. evaluated the relationship between internal medicine residency milestones and pulmonary-critical care medicine fellowship milestones using a retrospective cohort analysis. The authors analyzed all pulmonary-critical care fellows who had complete internal medicine residency milestones ratings from 2014-2017, and compared these to their 2017-2018 fellowship milestones. Analysis was limited to the core competencies of professional and interpersonal and communication skills (ICS) due to the shared language in these milestones between residency and fellowship.  They found an association within several subcompetencies.  Specifically, low milestone ratings in several professional and ICS subcompetencies during IM residency were associated with low milestone ratings in the first year of PCCM fellowship in those domains. Individuals with milestone rating of ≤ 2 in the ICS01 subcompetency at any point during IM residency were more likely to receive a rating ≤ 2.5 in their first year of PCCM fellowship.  This is the first set of data showing context-independent associations between milestone ratings in residency and those in fellowship.

Supplemental digital content:

http://links.lww.com/ACADMED/B119
http://links.lww.com/ACADMED/B120
http://links.lww.com/ACADMED/B121
http://links.lww.com/ACADMED/B122
http://links.lww.com/ACADMED/B123

Interview:

 

RQ: Did you always hypothesize that there may be context-independent associations between ratings in residency and those in fellowship, or was this a gradual realization after hands-on experience with the milestones and assessing trainees?

JH: We were definitely suspicious that there were context-dependent associations in some domains, meaning that a rater may expect different skills and expertise in communication for a PCCM fellow as compared to an IM resident. This bore out in prior literature showing professionalism and communication ratings shift from “ready for independent practice” at the conclusion of IM training to a lower rating once that learner transitions to fellowship (despite similar descriptors between the PCCM and IM Milestones).

However, beyond the likely context-dependent nature of some of the milestones, we hypothesized that there may be the associations between IM and PCCM ratings independent of context, as you mention. This definitely was our group’s anecdotal experience working with PCCM fellows. We noted some learners that struggled in certain competencies actually had hints of potential issues in their IM milestone ratings. We thought there might be an opportunity to think creatively about developing individualized-learning plans for fellows based on their performance in residency, in order to structure their training and help them achieve their full potential. But first, we needed to analyze the bigger data set to confirm the suspected association.

RQ: How do you anticipate incorporating this information into graduate medical education training, either from the residency side or from the fellowship side?

JH: This is the first study to show an association between the IM residency milestones and fellowship milestones, so offers additional validity evidence to the use of these within our competency-based medical education framework. We hope this information will be used to help program directors to establish individual-learning plans targeting the needs of the individual fellow, and would favor starting this process at the beginning of fellowship training.

RQ: Isolating the professionalism and ICS competencies clearly made for a more streamlined evaluation. What is next? Longitudinal data, or widening to the other four competencies, or qualitative analysis of how programs are utilizing milestones?

JH: There are so many great questions to ask about the milestones, and I think one important future area is thinking through how programs are using these on the front lines. There have been multiple studies in the residency setting analyzing how Clinical Competency Committees approach CCC meetings and milestones, but limited data for fellowship programs. Further understanding the process could drive faculty development (and ultimately improve the implementation of the Milestones framework in subspecialty training programs).

RQ: I like that the milestones are designed to be formative to allow for coaching trainees, as opposed to helping them match or achieve the next stepping stone. However, there remains concerns about real-life application. What are your concerns regarding milestones?

JH: The intent of the milestones is to ensure trainees achieve minimal competency prior to graduation, but also to provide a framework for an individual to identify areas of further growth. I think they offer an important assessment framework that shifts (ideally) to a criterion-based approach rather than a normative approach. However, optimal use of them requires thoughtful and careful faculty development (and improved understanding of how CCCs and programs use them). I think my biggest concern is the faculty development piece (or lack thereof), which has a major impact on the real-world use of them.

The other piece we considered with this work was the potential for feed-forward bias, in that someone may carry an undeserved halo or horn effort through fellowship training. Similarly, the potential of biases within Milestone ratings is an area of active research. However, we overall felt that there were a multitude of potential benefits to recognizing this association, including the important patient care implications and the opportunity to shape an individual learning plan for formative purposes.

RQ: I find it interesting that none of the individuals ever rated as critical deficiency during residency had milestone ratings at or below 2.5 anytime during the first year of PCCM fellowship, suggesting some degree of remediation or improvement. Albeit a limited sample size, are you able to speak to your impressions of this trend?

JH: This is a great observation, and a very interesting area of future work. As I mention above, there are many questions about the implementation of Milestones in a subspecialty context, and I worry the small sample size here was a limitation. Beyond that, the impact of remediation on Milestone trajectory (and growth of competencies) would be an important area to look into as well (especially for domains that are traditionally challenging to remediate, like professionalism).