University of Missouri-Kansas City: Impact Analysis Informing Decisions to Maximize the Efficacy & ROI of Supplemental Instruction

Civitas Learning 4-year Analytics Change Management Civitas Learning Space Impact Impact Analysis Results ROI

Share this Post

One of the hardest things institutions try to do is to stop engaging in some of the practices that have long been part of our culture and our history — but if we are going to better serve each and every one of our students and change their outcomes, this is what we need to do. After observing 1-2 percent bumps in retention at University of Missouri-Kansas City, and realizing that the increases were not taking place within our minority student populations — who are central to our mission — we chose to look deeper at our existing programs to measure what was working and what we needed to stop doing (or start doing differently). One of the largest academic support initiatives on our campus is Supplemental Instruction (SI)‚ which UMKC pioneered in higher education in the early 1970’s as a non-remedial approach to learning that integrates “what to learn” with “how to learn.”

CASE STUDY: 7.8%pp Persistence Lift and $600K in ROI

Even though UMKC was seeing 1-2% bumps in overall retention, outcomes were not increasing for all students. Learn how UMKC measured the impact of supplemental instruction program, and how the findings are helping the university optimize, prioritize, and allocate resources toward what works.
DOWNLOAD THE CASE STUDY
The Search for Clarity
Prior to Impact, we were primarily looking at changes in students’ GPA and course grades in order to report on the efficacy of Supplemental Instruction. We also ran analyses to see the impact on retention, but we were limited by our inability to do a true apples-to-apples comparison of students attending SI vs. those who did not. This, in turn, limited our ability to have conversations with the academic departments we were asking to offer SI in their courses, and to fund the program. Without clarity about the impact of the program on student outcomes, we could not have these conversations confidently. Impact allowed us to implement prediction-based propensity score matching (PPSM) in order to analyze — within hours — the true impact of SI on the success of our students.
Impact Findings
Results revealed a 7.8 percentage point (%pp) overall lift on persistence for students who attended SI three or more times each semester. We also found that the lift was more significant for African American students, students in their first semester, part-time students, and students with the lowest probability to succeed. Finally, we calculated $600,000 in return on investment (ROI) with a potential to achieve an additional $1 million in ROI by encouraging more students to attend SI at least three times each semester.
Transforming Culture
Armed with Impact results, we are now having more productive conversations with academic departments. Now, instead of simply offering SI in courses where 25 percent or more of students had Ds, Ws, or Fs… we are using the Civitas Courses application to identify additional courses where SI could help elevate student success — particularly “yellow flag” courses, which are those where earning a C lowers a students’ likelihood to graduate. We are also collaborating more with the academic departments that pay the associated fee to offer SI. As they lean into SI, the institution is leaning as well – by sending nudges to the students who are most likely to benefit from this program. We are using Illume to send emails to those students, and using the Civitas student mobile engagement application, ClearScholar, to reach students directly on their phones.

CASE STUDY: 7.8%pp Persistence Lift and $600K in ROI

Even though UMKC was seeing 1-2% bumps in overall retention, outcomes were not increasing for all students. Learn how UMKC measured the impact of supplemental instruction program, and how the findings are helping the university optimize, prioritize, and allocate resources toward what works.
DOWNLOAD THE CASE STUDY

Barbara A. Bichelmeyer, Ph.D., Provost and Executive Vice Chancellor, Professor of Management, University of Missouri-Kansas City

Prior to joining UMKC in August 2015, Dr. Bichelmeyer held numerous leadership roles at Indiana University and IU-Bloomington, including Professor in the Department of Instructional Systems, Executive Associate Vice President for University Academic Affairs, and Senior Director of the Office of Online Education. She also spent a year as interim chancellor of Indiana University Southeast, in the Louisville metropolitan area as well as interim chancellor at UMKC. Barbara’s research and grant-activities are focused on the study of how educational systems integrate new technologies and how the introduction of new technologies change educational systems in order to better support student-centered learning and improved human performance. She has served as a consultant for the design, development and evaluation of instructional programs and performance improvement initiatives with numerous organizations. Barbara holds four degrees from the University of Kansas: B.S. in Journalism; B.A. in English; M.S. in Educational Policy and Administration; and a Ph.D in Educational Communications and Technology.