Measuring Impact: Recent Journal Article Details How to Scale & Evaluate Support Services

Civitas Learning Civitas Learning Space ROI

Share this Post

How do you know if your student success initiatives are moving the needle? How will you report to leadership on its effectiveness and truly understand if it worked or had any impact? How do you evaluate your impact of your campus innovations? We know that measuring the effectiveness of new strategies or programs is no simple task. Typically there are limited internal resources and time to conduct the analysis needed to understand impact. And, we know it can be hard to get that information in time to support students during the term. Opening Doors for Outcomes In a recent Planning for Higher Education Journal article, we unpack how two universities – Austin Community College (ACC) and the University of Arizona (UA) did that using Illume Impact. Through this rigorous analysis, they measured the effectiveness of their programs and enhanced institution-wide decision making for their current and future use of resources and investments. Austin Community College ‘Starts Here to Get There’ Austin Community College created a high-tech, learning laboratory called the ACCelerator to help students develop core skills and complete developmental education coursework. This program provided a vast array of support services, including a network of faculty, counselors, advisors, tutors, librarians and other staff to help students achieve their educational goals. After its opening year, ACC collaborated with us to evaluate the ACCelerator’s impact on student outcomes. They specifically wondered if the number of visits correlated to higher persistence rates. For three semesters, attendance data were recorded. Since this was an observational study, to control selection bias and to meet specific data guidelines – PPSM with baseline equivalence was used. This surfaced three different groups – students who did not visit, visited only once, or those who visited multiple times. Outcome Reveal #1 The data revealed that ACCelerator did have a positive impact for students who attended even at least once and an even greater impact for those who participated with the developmental education programs. With this level of understanding, ACC had confidence in the program’s impact on student outcomes and then effectively prioritized their resources according to the signal generated on program’s impact. Alongside this, they designed specific outreach to the students who would benefit most from it. University of Arizona ‘Bears Down’ University of Arizona provides their 50,000+ student population with a student support center called the THINK TANK, which includes a Writing Center, math tutoring, and supplemental instruction with four different locations across campus. UA had already completed propensity score matching (PSM) statistical analysis on this student resource to gather persistence rates for full-time freshman. However this type of analysis doesn’t account for selection bias, and UA wanted to quantify impact at a deeper level. Outcome Reveal #2 The data revealed an increased 2.3 percentage point persistence rate for those students who took advantage of the writing center, drop-in math tutoring and other services. This translated to 587 students continuing, $3.3 million in term tuition and an estimated $7.5 million in unlocked tuition from the same students persisting through their academic career. This level of insight provided UA the capability to generate strategic, targeted campaigns to promote the student support services and was shared with faculty and academic departments for referrals to the THINK TANK services. Moving ‘From Innovation to Impact’ Through the use of impact analysis, institutions like UA and ACC are able to leverage this information and manifest a culture of optimized decision making to reach institutional goals for student outcomes. No longer are institutions left to do the best they can with minimal resources, but instead they can lean on accurate, timely signal strength to provide a more comprehensive picture and unlock the power of student initiatives. Most importantly, institutions can make decisions in real-time and then take action to support students during the term


Civitas Learning Chief Learning Officer and Co-Founder Mark Milliron, Chief Data Scientist David Kil, Senior Vice President of Outcomes & Strategy Laura Malcolm, and Product Manager Grace Gee explain why rigorously evaluating the impact of innovative student success initiatives is key in meeting institutional goals for student outcomes, resource allocation and return on investment.