MBM – Looking at the Forest (Jarrell)
Bruce E. Jarrell, MD, David B. Mallot, MD, Louisa A. Peartree, MBA, and Frank M. Calia, MD
Abstract
Purpose: To present an alternative approach to mission-based management (MBM) for assessing the clinical teaching efforts of the faculty in the third and fourth years of medical students’ education.
Method: In fiscal years 2000 and 2001, interviews were conducted with department chairs and faculty members with major responsibilities in education at the University of Maryland School of Medicine. Using a standard worksheet, each rotation was categorized according to the amounts of time students spent in five teaching modes. After each department described its rotation and maximum teaching time, the department team and the MBM team negotiated the final credit received for its course. This final determination of departmental clinical teaching was used in subsequent calculations. Adjustments were made to the department clinical education time based on the teaching mode. Groups of medical students were surveyed to determine the relative value of each teaching mode. These relative values were then used to modify the clinical education times credited to the department. The last step was to distribute the effort of the faculty between clinical and educational missions.
Results: The data analysis showed approximately 57,000 credited faculty hours in one year for direct education of medical students in each curriculum year. These hours equal the annual workload of 28 full-time faculty members.
Conclusions: A powerful use of MBM data is to move from thinking about resource allocation to thinking about the effective management of a complex organization with interlaced missions. Reliable data on faculty’s contributions to medical students’ education across departments enhances other MBM information and contributes to a picture of the dynamic interconnectedness of missions and departments.
One outcome of the profound economic changes in medical reimbursement over the last 15 years is a need for more attention to resource allocation within academic medical centers (AMCs).1 One tool currently enjoying interest is mission-based management (MBM), whereby money or effort is matched, albeit with great difficulty, to the AMC’s three traditional missions of education, research, and clinical care. Decisions regarding departmental support by the dean can then be made on a mission-directed rather than on a historical basis.2 Progress has been made in reliably consolidating clinical and research budgets from various accounting systems, allowing a global view of those missions. This consolidation has permitted resource allocation to be data based. The educational activity has been more difficult to measure.3 Many efforts have focused on educational assessment based on faculty-effort surveys and are based largely on self-reporting, with inherent problems of inaccuracy, lack of response, and problems of categorizing various teaching activities.4,5 This is particularly problematic for faculty when responding to their clinically-based teaching activities. Difficulties in accurately and consistently separating clinical care from teaching time and dealing with trainees at multiple levels of sophistication add to the complexity. One additional problem with self-reporting is the tendency to define educational time as the time left over after the amounts of time devoted to other, more definable, missions (i.e. research, clinical care, and administration) have been determined.
From an institutional perspective, teaching responsibilities are assigned to departments and oversight is provided by a curriculum committee. This assignment creates a departmental teaching responsibility that is, in turn, determined by the sizes of medical students’ classes and lengths of rotation. The department must allocate resources, including faculty, to undertake that educational load. Meeting that educational requirement is determined by the teaching philosophy of the department in conjunction with economic factors, residency workforce, school policies, and external review boards. Thus, although the total departmental teaching load is able to be estimated accurately and is relatively predictable from year to year, the contribution of any single faculty member might vary as frequently as daily and is much less predictable. In this study, we present an alternative approach to assessing the clinical education efforts of the faculty in the third and fourth years of medical students’ education.
Method: At the University of Maryland School of Medicine, year one and year two have interdisciplinary curricular blocks that use both basic science and clinical faculty. Faculty are assigned hour-for-hour credit for lectures, small-group sessions, and teaching laboratories. In addition, each of those teaching modes is credited with additional time that reflects class preparation and test development. The additional time assigned to each mode was debated and determined by the Fiscal Affairs Advisory Committee (FAAC), the committee charged with overseeing MBM. Course administration credit is based on the length of the course (see Figure 1).
Departmental teaching responsibility in the clinical years is determined by aggregating faculty effort in a variety of teaching modes. Third- and fourth-year rotations have two main components. The first, small component is formal classroom sessions with no patient interaction, and includes lectures and small-group discussions. Departmental credit for these sessions is determined by the average number of didactic sessions for each rotation. For each hour of formal didactic teaching, an additional half hour is added to reflect the faculty time necessary to prepare for the session. This “prep” time is less than credit assigned for preclinical didactic activities. These hours can be attributed to individual faculty members in the department, but in our analysis the total time for these activities is accounted for only at the departmental level. The second and major component of clinical education is face-to-face teaching with an attending physician in the presence of a patient.
In the clinical setting, the occasion for faculty to educate medical students depends on the number of trainees rotating through the department and the amount of time students spend “face-to-face” with faculty. Assuming every interaction consists of one student with one faculty member for a given number of hours per day, it is possible to define in hours the maximum time for “face-to-face” teaching while delivering care. This number of hours is calculated by multiplying the number of trainees in the rotation by the number of days in the rotation by the number of hours per day of faculty interaction in a clinical setting
Maximum Teaching Time = number of
medical students X length of rotation in
days X agreed-upon number of hours
with faculty in a clinical setting
This maximum time is defined as the upper limit of the department’s teaching load, which would be the total faculty teaching effort if every trainee were taught in a one-to-one ratio with a faculty member and full credit was given for teaching even though clinical care was also being delivered.
After determining the maximum teaching time, in fiscal years 2000 and 2001, we conducted interviews with each department chair and one or two faculty members with major responsibilities in education. Using a standard worksheet, completed at the time of the interview, each rotation was categorized according to the amount of time students spent in large versus small groups and by the trainee mix. We categorized activities into five teaching modes: one student and one faculty attending physician; a small group of medical students (two to four) and a faculty attending physician; one student with one faculty attending physician and one resident; small groups (three to five) of medical students and residents and a faculty attending physician; and a large group of both types of trainees (more than five) and a faculty attending physician. During the departmental interview, the clerkship or rotation was discussed and categorized based on teaching groups. Special teaching situations (i.e., “teaching attending physician”) and unique teaching features of the clerkship were discussed so that department-specific credit could be applied. After the department described its rotation, the department team and the MBM team negotiated the final credit that the department received for its course. We used this final determination of departmental clinical teaching in subsequent calculations.
Using the data gathered from each department, adjustments were made to the department clinical education time based on the teaching mode. We surveyed groups of medical students to determine the relative value of each type of educational interaction (see Appendix A). The resulting relative values were reviewed by key “education” faculty members from a variety of clinical departments. We found the values generated by the medical students to be consistent with the perceptions of faculty reviewers. The students’ values for clinical education time were: an attending physician with a medical student (one hour), an attending physician with two to four medical students (0.77 hours), an attending physician with a medical student and a resident (0.56 hours), an attending physician with five medical students and residents (0.36 hours), and an attending physician with more than five medical students and residents (0.25 hours). These relative values were then used to determine the clinical education times credited to the department. (For a more detailed discussion of our method for measuring clinical education time, see Appendix B).
The last major step in our clinical education method was to designate the efforts of the faculty in fulfilling their clinical and education missions. Up to this point, the method produced a total time that medical students and faculty were together in a clinical setting where the faculty member performs both clinical and educational activities. This total time had then to be split between these missions to give appropriate credit and avoid double counting. Through a series of discussions with the members of the school of medicine’s FAAC, and faculty and leadership in the office of medical education, we decided that for every eight hours of patient care delivered in the presence of a medical student, 1.5 hours (18%) of clinical educational time would be credited to the department. This ratio of clinical education to patient care is based on the student’s, the faculty’s, and the administration’s input and discussion and does not include time directed to teaching residents in the clinical setting. This method of allotting credit also did not include faculty’s scholarly educational activities unrelated to the medical students’ curriculum, e.g., developing innovative teaching materials for future use, writing textbooks, and advising students.
Finally, we determined unique education endeavors with the department. An example would be teaching attending physicians with no clinical responsibilities who receive full hour-for-hour teaching credit without any reduction in their credit for patient care. Another example would be credit given to the department of radiology for teaching medical students during basic third-year clerkships that include significant radiology components. Credit hours for education administration were also given for clerkship directors. We derived data from discussions with the individual departments to determine allocations for these special situations. After these data were summarized, they were given to the departments for review and further input. The data were then submitted to the FAAC and became a key component in institutional decision making.
Results: The total faculty times allotted to teaching medical students are summarized in Figure 2. In addition, this figure shows the breakdown between basic science faculty’s and clinical faculty’s contributions. The data exclude the participation of residents, fellows, and staff in the curriculum. The total hours for teaching students in their first two years are nearly equal in faculty time, approximately 10,000 hours, reflecting the similarity in curriculum structure. The distribution of hours in the second year between preclinical and clinical departments reflects clinicians’ participation in the current curriculum. The large number of clinical faculty’s hours in the third year of medical school reflects the individual and small-group teaching modes in the clinical setting as well as the increased time medical students spend with faculty. The time in year four is significantly less than that in year three because the year itself is shorter, and many students spend considerable time in community sites or other medical institutions.
Figure 3 shows the distribution of faculty education hours summarized for all four years of medical school and shows the relative ranking of education hours among the departments of the school of medicine. By using our method, clinical departments that have required clinical rotations are credited with large numbers of hours. For example, faculty in the department of medicine received the greatest amount of credit due to the length of the third-year clerkship, the amount of teaching required in fourth year sub-internships, as well as a significant teaching contribution in year two. On the other hand, many medical students are assigned to community sites for the obstetrics and gynecology clerkship, resulting in fewer hours credited to the department.
Our method produced additional data not directly tied to the MBM process. For instance, the distribution of clinical teaching in third-year clerkships was an attending physician with a medical student, 19%; an attending physician with two to four medical students, 7.1%; an attending physician with a medical student and a resident, 21.5%; an attending physician with three to five medical students and residents, 47.5%; and an attending physician with more than five medical students and residents, 4.9%. This distribution also shows the efficiency of teaching within a department. These data, originally collected for MBM purposes, can then be reviewed and analyzed by the curriculum committee and individual departments.
Our data analysis showed approximately 57,000 credited faculty hours in one year for direct education of medical students in each curriculum year. Using a standard work-year definition of 2,080 hours (52 weeks 40 hours), these 57,000 hours of credited education time equal the workload of 28 full-time faculty members. However, this credited time is not the total cost of medical students’ education, because it does not include residents’ student teaching or indirect expenses the department incurs coordinating the educational effort or mentoring the students. In addition, the data do not take into account the number of faculty members necessary to generate patient volumes to sustain a teaching program. While our method produces a relative ranking among departments, it also provides an overall faculty effort number for the medical students’ education mission, which can then be compared with faculty’s efforts in the clinical and research missions.
Discussion
The Association of American Medical Colleges (AAMC) has identified six core principles as central to MBM: integrating a school’s financial statements, measuring faculty and departmental activities and contributions to mission, building organizational support for reporting tools and metrics, guiding the dynamics of leadership, holding faculty and department and institutional leaders accountable, and building trust and institutional perspective.2 With regard to the education mission of an AMC, the AAMC’s second core principle—faculty and departmental activities and contribution to mission— has historically been measured through an aggregation of individual faculty members’ teaching activities. The crux of the problem, however, is that individual faculty members’ activities, even if accurate, do not necessarily reflect the educational mission of the school. That mission is defined by external accrediting bodies, the dean’s office, departmental chairs, and faculty education committees. Individual faculty members assume that a variety of teaching activities are central to the education mission, especially in the clinical years. These activities may enrich the students’ experiences and augment the curriculum. However, our method of measuring the faculty’s contribution seeks to segregate those activities tied to the core education mission for purposes of MBM.
Our method yields data that describe the educational effort in a reproducible manner. The data verify our impressions about the effort and time expended by each department and should encourage discussion about the relative educational effort. Our method does not focus on individual faculty members, but it does spotlight different departmental teaching activities and raises questions about the merit and/or cost of those activities, furthering the MBM core principle, “building trust and institutional perspective.” At our medical school, the FAAC uses MBM to make fiscal recommendations to the dean. In fiscal year 1999, the FAAC recommended redistributing $3.0 million over two years among departments that were critical to the educational mission, but that were in financial difficulty. These decisions were based on perceptions of the faculty’s educational activities but had little supporting data. In fiscal year 2002, the school of medicine redistributed $1.0 million of the dean’s funds among departments important to the education of medical students. In this redistribution, the method we describe in this paper provided key education data for the FAAC’s discussions.
Previously, annual departmental support from the dean’s office has been a continuation of historic allocations. The data in Figure 4 represent a scattergram of the historical allocation of dean’s funds in fiscal year 2001. The data show the lack of correlation between the dean’s historical financial support to departments and the medical students’ education data derived from our analysis. Thus, a redistribution based on our method could align resource allocation with educational effort. The FAAC operates under the assumption that medical students’ education should be a factor in departmental support.
Our method has other advantages. Gathering and compiling the data are less time-consuming than surveying individual faculty members. Our method permits separation of teaching medical students from teaching residents. It can also be used for other education evaluations. The next applications of our method will be to analyze faculty’s time spent teaching residents and graduate students and to refine the measures of education merit described above. By meeting with the chair and lead educators in each department, our method adhered to the MBM core principle of “building organizational support for reporting tools and metrics.” Each department is able to demonstrate the uniqueness of its own educational approach.
Conclusion
One of the major stated objectives of MBM is to provide a medical school’s decision makers with accurate information with which they can allocate resources.6 A more powerful use of MBM data is to move from thinking about resource allocation (“accounting”) to thinking about the effective management of a complex organization with interlaced missions. The addition of reliable data on medical students’ education across departments enhances other MBM information and contributes to a picture of the dynamic interconnectedness of missions and departments. In contrast, summation of individual faculty member’s efforts (i.e., survey methods) does not necessarily reflect the overall mission of the school and is unlikely to produce an accurate picture of a complex organization. These summations may even obscure direct educational activity and hinder an open discussion of the place of education in a school’s mission. The University of Maryland School of Medicine’s experience with MBM and the use of our method produce an informative image of a complex environment. In describing this image, it is more informative to describe the forest than to count the trees.
This article was originally published in the December 2002 issue of Academic Medicine.
References
1 Cohen JJ. Financing academic medicine:
strengthening the tangled strands before they
snap. Acad Med. 1997;72:520.
2 Cohen JJ. Introducing the Mission-Based
Management Resource Materials.Washington, DC:
Association of American Medical Colleges, 2000.
3 Nutter DO, Bond JS, Coller BS, et al.
Measuring faculty effort and contributions in
medical education. Acad Med. 2000;75:200–7.
4 Hilton C, Fisher W. A relative-value-based
system for calculating faculty productivity in
teaching, research, administration, and patient
care. Acad Med. 1997;72:787–91.
5 Mallon WT, Jones RF. How do medical
schools use measurement systems to track
faculty activity and productivity in teaching?
Acad Med. 2002;77:115–23.
6 Watson RT, Romrell LJ. Mission-based budgeting:
removing a graveyard. Acad Med. 1999;74:627–40.
Appendix A
Estimation of the educational value of learning in a group setting:
The medical school is developing an internal methodology to measure education in the clinical setting. As part of this methodology we are trying to understand how trainees perceive the amount of teaching they receive in certain situations while clinical care is being delivered. We will evaluate this information along with responses from department leadership and selected faculty to the same questions.
Instructions:
Please think about the amount of teaching that you as an individual receive in-one-on-one interaction with an attending. With that in mind, compare it to the teaching that you as an individual receive in the situations delineated below. Please try to generalize your answers across all of your third-year required rotations and not bias your answer based upon your current position or upon some good or bad anecdotal experience.
Please respond to each question in comparison to one-to-one teaching. That is, you and one attending in the clinical setting. This will be considered a 1:1 teaching value. Please do not consider lectures, small group or seminars in your answers. For the purpose of this survey, we are interested in education during rounds, ambulatory clinic, operating room, reviewing films/test, etc.
Example:
Consider a rotation where one aspect of the rotation has you and several other students with an attending discussing a clinical problem. If you consider that the amount of teaching that you receive from the attending is similar to what it would have been if it had been just you and the attending, you would answer 1:1. If you thought that this experience is less than that, for example one half the benefit, then you would answer 1:2.
- Individual an attending 1:1
- Individual an attending one resident __
- Individual an attending group of students (3–5) __
- Individual an attending group of students and residents (3–5) __
- Individual an attending group of students and residents (5) __
- Does teaching directed to a higher-level trainee have a similar benefit as that directed to a lower-level trainee?
Appendix B
An In-depth Look at the University of Maryland School of Medicine’s Method for Measuring Faculty’s Clinical Education Time
Example:
Suppose that for Department 1, the analysis shows that ten third-year students rotate through this department every four weeks. The students spend five hours a day with faculty in the clinical setting. On average, the following teaching modes for the students are described by the department:
▪ Group A: Attending one medical student (10% of the rotation)
▪ Group B: Attending two to four medical students (15% of the rotation)
▪ Group C: Attending one medical student one resident (11% of the rotation)
▪ Group D: Attending three to five medical students and residents (32% of the rotation)
▪ Group E: Attending more than five medical students and residents (32% of the rotation)
Implications:
Calculations for Group A or B. Suppose for the 10% of time spent in Group A that the typical experience is an attending with one third-year student (and no residents) for four hours in a medical clinic. Of that clinical care time, 18% or 0.72 hours is credited as educational time. Since this is Group A teaching mode, the department is credited with one student x 0.72 hours x 1.0 0.72 hours of clinical education time per day for third-year students. Thus, the department received 0.18 hours (or 10.8 minutes) of credit for teaching this student for each hour. Since there are five clinical days per week and four weeks per rotation, the total time for Group A is 0.72x5x4 14.4 hours per rotation. For students in Group B, the department would receive 0.138 hours (or 8.25 minutes) of credit for teaching each student for each hour.
Clinical implication. One student and no residents (Group A) would slow down the clinical activity rate, but still allow the attending to perform other independent activities. In one hour, the student might be able to see one patient, develop a diagnosis and treatment plan, discuss the case with the attending and revisit the patient with the attending at which time the attending could evaluate and treat the patient. There is significant time for one-on-one teaching at the student level. The attending has a relatively low “overhead” of getting to know the student, teaching the student, and evaluating the student’s ability and performance. Several students with no resident participation (Group B) would likely slow down the clinical activity rate more significantly and not allow the attending to perform independent activities. There would still be significant time for one-on-one teaching at the student level. The attending’s overhead is higher than for one student.
Calculations for Group C. Suppose for the 11% of time spent in Group C that the typical experience is an attending with one student and one resident for three hours while delivering clinical care in the operating room. Of that clinical care, 18% or 0.54 hours is credited as educational time. Since this is Group C teaching mode, the department is credited with one student x 0.54 hours x 0.56 0.30 hours of clinical education time for the third-year student. Thus, the department received 0.10 hours (or 6.0 minutes) of credit for teaching this student for each hour. Since there are five clinical days per week and four weeks per rotation, the total time for Group C is 0.56x5x4 11.2 hours per rotation.
Clinical implications. One student with one resident could have a positive or negative effect on patient flow depending on the training level of the resident and whether the student works with the resident or independently. Since some of the attending’s time is needed to supervise and teach the resident, less time would be available for one-on-one teaching with the student. In addition, the teaching that is not one-on-one with the student would have to be directed at two levels of trainees, reducing its effectiveness at the student level. The attending’s overhead is low for this combination of trainees.
Calculations for Group D or E. Suppose for the 32% of time spent in Group D that the typical experience is an attending making rounds with two third-year students and two residents on the inpatient floor each day for six hours. Of that clinical care, 18% or 1.08 hours is credited as clinical education time. Since this is Group D teaching mode, the department is credited with two students x 1.08 hours x 0.36 0.78 hours of clinical education time for the third-year students. Thus, the department received 0.065 hours (or 3.9 minutes) of credit for teaching each student for each hour. Since there are five clinical days per week and four weeks per rotation, the total time for Group D is 1.08x5x4 21.6 hours per rotation. For students in Group E, the department would receive 0.045 hours (or 2.7 minutes) of credit for teaching each student for each hour.
Clinical implications. A small group of students and residents (Group D) would most likely have a positive effect on the clinical activity rate but would leave less time for teaching one-on-one with students. Also, group teaching would have to be directed to the multiple levels of the trainees. The attending’s overhead would be higher in this setting. With a larger group of students and residents, the attending would be required to supervised the residents and, therefore, would have less time for one-on-one teaching with the students. Note that although the attending’s teaching time in Groups C, D, and E might be less than in Groups A and B, students in Groups C, D, and E might receive additional teaching from residents, a factor which is not assessed in this analysis.