It’s a rare point of consensus between higher ed administrators, faculty and staff: Initiative fatigue is nearing crisis levels. The morass of well-intentioned but hard-to-measure student success programs stretches budgets and workloads, spawns countless follow-on initiatives and spurs wariness about what’s really working.
Using Illume Impact to Better Understand Student Needs & Combat Initiative Fatigue
Most of the 1,000 CEOs and presidents surveyed in the League for Innovation’s most recent Trends Report listed initiative fatigue among their top concerns. Separately, when we asked a president with one of our partner schools how many student success initiatives they were running, he estimated around 35. A subsequent initiative inventory put the number closer to 100 – a suffocating amount of simultaneous and likely overlapping effort.
So how did we get here, and what can we do to prevent this fatigue from spiraling into exhaustion? And, equally as important, how do we answer the hard resource allocation questions that also affect student outcomes? When student success metrics improve, where is the lift coming from? When is selection bias clouding effectiveness measurements? If a program merits scaling up, which students should we be marketing it to?
To help our partners answer these questions and many others, Civitas Learning developed Illume Impact. Having vetted the module with some innovative partners, we are pleased to announce very promising early results.
Illume Impact: Finding the True Impact of Your Initiatives
The newest module in Illume – Illume Impact – brings clarity to initiative effectiveness. We can use the power of Illume® and the Student Success Platform to show the persistence lift of any of your programs to best understand which of these is the most impactful for the goals you have set, as well as which ones work well together for specific student populations.
In our early work with Illume Impact, our partners brought forward some interesting initiatives to test and are now gaining new knowledge and powerful confidence about their investments. For some, it’s about making the big investments pay off. For others, it’s about the power of many marginal gains for a specific population. Either way, it’s about gaining a deeper understanding of what is measurably improving persistence from among your many efforts. In our spirit of a community Learning Together® we are sharing some early examples of how partners are gaining insights, taking action, and learning with Illume Impact.
Case #1: Finding the Powerful Persistence Lift in a Writing Center
Often, the hardest to earn gains are for populations with already high success rates because so much improvement has already been introduced. One of our partner universities has a large academic support center, a large student body of approximately 50,000 students and an impressive persistence and graduation rate in the very high 80 percent range. They have begun to test and measure the various initiatives in the Center to determine which ones are most beneficial to certain populations of students. With this knowledge they can build awareness and usage, and make the difficult gains in persistence and graduation to reach a goal of 90 percent retention.
While the Center offers the full spectrum of support, the focus is on the writing center, math tutoring and supplemental instruction. We analyzed four years of student data from Spring 2011 – Fall 2015 during which time 50,000 students attended the Center. Illume Impact measured a 2.3 percentage point increase in persistence (weighted average) which translates to an additional 587 students persisting. This equates to at least $3.3 million in additional tuition per term.
We then isolated just those students who attended the Writing Center. The Writing Center helped drive a 2.7 percentage point increase in persistence for students overall who used it, compared to those who did not. For first-year students the lift in persistence was higher at 3.4 percentage points – an important learning given this institution’s focus on first-time, full-time freshmen.
Taking the analysis a step further, we then divided all Writing Center users in the 2011-2015 sample into quintiles based on most likely to persist to lowest likelihood to persist. Our team used prediction-based propensity score matching to identify like students (across many variables) whose data footprints were even matches, but who did not attend the center. This minimizes the selection bias often clouding the quest to measure supplemental academic support efficacy.
By looking at the data Illume Impact surfaced, this institution learned three important and highly actionable insights:
- The first was that they could use Illume to identify discreet sets of students (quintiles) to target based on powerful predictors rather than by examining one attribute like grade book, attendance or GPA.
- The second insight was the significant lift of 8.2 percentage points increase in persistence in the bottom quintile, with statistical significance. This lift was approximately 600 percent greater than for students with a moderate probability to persist.
- The third was that they could identify the individual students in the bottom quartile by name, using the Student Lists feature in Illume, and then take action.
This is informing advisors about exactly who to reach out to, and encourage more use of the writing center for measurable lift in student success, now and in the future.
Case #2: Large Urban College Measures Their Investment in Giant Computer Lab
This large urban college is among the top three institutions in its state and serves more than 50,000 students a year, many of them entering underprepared for college-level work. Traditional developmental education courses have had costly effects with high withdrawals (27 percent). The time and cost to complete the non-credit bearing, three-course sequence of Math, Elementary Algebra and Intermediate Algebra has been problematic for many working adults with financial constraints and busy life schedules.
This college invested in a large computer lab with more than 600 stations and is working to bring their students of all academic capabilities through the center. Since its opening in 2014, the center has seen over 16,000 unique students representing well over 200,000 interactions. Students are drawn to the lab not only because of its size, but also because of the innovative programs being offered within its glassed walls. For example, the center offers a redesigned developmental math course – one that combines personalized hands-on learning with adaptive online education.
We used Illume Impact to analyze first-year data and found that the center is the most powerful tool in the college’s suite of student success initiatives. This is especially true among groups that have historically struggled the most with persistence and completion.
Key findings include:
- Students who visited the center at least once each term persisted at a rate of 6.15 percentage points higher than students who never visited the center.
- The impact was significantly greater for DevEd students who showed a lift in persistence of 12.44 percentage points, compared to 3.29 for non-DevEd students.
- The lower the persistence prediction scores for DevEd students, the better the outcomes – from 10.25 to 17.75 percentage points for those students with lowest probability to succeed.
- Additionally, we found that part-time DevEd students who frequent the math center improve their persistence by 14.07 percentage points compared to full-time DevEd students who experienced a lift of 9.01 percentage points.
There’s much to learn, individually at campuses like these and together as a community. Learn More.
