One of the hardest things institutions try to do is to stop engaging in some of the practices that have long been part of our culture and our history — but if we are going to better serve each and every one of our students and change their outcomes, this is what we need to do.
After observing 1-2 percent bumps in retention at University of Missouri-Kansas City, and realizing that the increases were not taking place within our minority student populations — who are central to our mission — we chose to look deeper at our existing programs to measure what was working and what we needed to stop doing (or start doing differently).
One of the largest academic support initiatives on our campus is Supplemental Instruction (SI)‚ which UMKC pioneered in higher education in the early 1970’s as a non-remedial approach to learning that integrates “what to learn” with “how to learn.”
The Search for Clarity
Prior to Impact, we were primarily looking at changes in students’ GPA and course grades in order to report on the efficacy of Supplemental Instruction. We also ran analyses to see the impact on retention, but we were limited by our inability to do a true apples-to-apples comparison of students attending SI vs. those who did not.
This, in turn, limited our ability to have conversations with the academic departments we were asking to offer SI in their courses, and to fund the program. Without clarity about the impact of the program on student outcomes, we could not have these conversations confidently.
Impact allowed us to implement prediction-based propensity score matching (PPSM) in order to analyze — within hours — the true impact of SI on the success of our students.
Impact Findings
Results revealed a 7.8 percentage point (%pp) overall lift on persistence for students who attended SI three or more times each semester. We also found that the lift was more significant for African American students, students in their first semester, part-time students, and students with the lowest probability to succeed. Finally, we calculated $600,000 in return on investment (ROI) with a potential to achieve an additional $1 million in ROI by encouraging more students to attend SI at least three times each semester.
Transforming Culture
Armed with Impact results, we are now having more productive conversations with academic departments. Now, instead of simply offering SI in courses where 25 percent or more of students had Ds, Ws, or Fs… we are using the Civitas Courses application to identify additional courses where SI could help elevate student success — particularly “yellow flag” courses, which are those where earning a C lowers a students’ likelihood to graduate.
We are also collaborating more with the academic departments that pay the associated fee to offer SI. As they lean into SI, the institution is leaning as well – by sending nudges to the students who are most likely to benefit from this program. We are using Illume to send emails to those students, and using the Civitas student mobile engagement application, ClearScholar, to reach students directly on their phones.
