How to Assess Student Success Initiative Efficacy

Civitas Learning Sustainable Outcomes

Share this Post

Despite considerable investment in student success programs over the last decade, student success outcomes remain stagnant at many colleges and universities. It’s still unclear who needs help, services are underutilized, staff capacity is limited, information is siloed, and it’s challenging to align approaches across departments

Institutions successfully moving the needle on student outcomes are doing something different – and it starts with comprehensively evaluating their student success initiatives to improve them iteratively and intelligently. Regular evaluation of student success practices provides the specific insight needed to make informed decisions and take action to incrementally improve student outcomes and generate more revenue over time.

After working with hundreds of higher education student success and assessment leaders and helping evaluate more than 1,000 student success initiatives, we’ve learned there’s a better way to know which student success initiatives are working (and for whom), and how to use that insight to drive institutional improvement.

Step 1: Identify All Existing Student Success Initiatives

It’s important, to begin with, a thorough audit of what initiatives exist today. We’ve worked with many higher education leaders who aren’t quite sure what student success initiatives exist across their institutions. Many can list the most obvious or visible, like the tutoring center, or those closely related to their own work; however, when they set out on a mission to understand the full scope of initiatives across all departments on campus, they’re often surprised by how many they uncover.

Initiatives can range from ongoing services and support programs to more sporadic, one-off resources and events. Examples might include counseling or mental health services, guest speaker events, or a student job fair.

When you begin your audit, you might start by conducting interviews with department heads, deans, career center leaders, students, advisors, faculty members, etc. Throughout the process, document what you learn. Make sure you uncover the following:

  • What is the initiative designed to do?
  • Who is the initiative designed for?
  • What department, person, or team oversees the initiative?
  • Who is eligible to participate in the initiative?
  • Does it potentially overlap with another initiative?

This foundational understanding will serve as an important input in Step 2.
Here is an example of how you might document the information you gather in step one. Want your own copy? Download the inventory template here.

Step 2: Understand the Efficacy of Each Initiative 

Take a look at your initiative inventory and select a handful of initiatives that warrant further investigation. Some of our most successful customers choose to hone in on services that are either clearly duplicative, services that require measurable outcomes to meet funding requirements, or services that require significant investment each year.

During this step, answer questions like:

  • Who is mostly taking advantage of this service?
  • How much do different student groups benefit from this service? (consider quantitative and qualitative evidence)
  • Are the benefits we see clearly linked to the outcome the service intends to achieve? If not, why not?

Here is an example of how you might document the information you gather in step two. Want your own copy? Download the inventory template here.

Step 3: Prepare for Action… Then Take It

Once you have decided where to focus and completed your preliminary discovery, it’s time to ensure you have enough information to take action.

During this step, consider more nuanced questions like:

  • Where overlap exists, is it intentional or simply duplicative?
  • Would there be a benefit to combining duplicative efforts after better understanding the relative impact of each?
  • What is the “theory of change” of this program… why would we assume it achieves these outcomes?
  • If Student Learning Outcomes (SLOs) are the focus, do they have any logical or evidence-based relationship to persistence or are they geared toward some other outcome of value to the institution?
  • How has this initiative changed over time, and has that been in response to data of some kind?
  • Does the initiative have any special cultural, historical, or emotional significance to the campus community, and why?
  • Is there a relationship between what the initiative means to people and what it achieves?
  • Most importantly … what should we do now?

Some institutions have used the information they gather to change policy or seek additional funding for certain services, while others have sent highly targeted nudges to students most likely to benefit. For example, the University of Missouri-Kansas City updated the required number of supplemental instruction visits, and Monroe Community College sent personalized messages to students shown most likely to benefit from their Center for Academic Reading.

The key here: with the right level and kind of intelligence, it becomes possible to focus your efforts and begin the process of iterative improvement.

Improve Upon What Works, Rethink What Doesn’t

Every student success initiative was created by someone with the best of intentions. As we seek to understand and improve the effectiveness of these interventions, we need to appreciate them in context. Why did they begin, how have they evolved and changed, what challenges do they face today, and upon what concepts they are based?

We find that a process like appreciative inquiry, aided by great measurement, helps ground our investigation of efficacy in strengths and creates a safe space for improvement. Almost every program is effective for someone… so how can we value the intention of the initiative, recognize and improve upon the pieces that work, and rethink what doesn’t?

Want to take the first step?

Download this helpful inventory template to guide you in your discovery.

TAKE THE FIRST STEP

Tina Donahoo

Tina Donahoo is a Principal Strategic Consultant of Outcomes at Civitas Learning, where she works with colleges and universities on the use of advanced analytics for assessment and optimization of student success initiatives and financial aid allocation. Utilizing Civitas’ Impact software tool, Tina helps institutions perform rigorous efficacy measurement of student success initiatives to optimize finite resources, personalize student support and demonstrate the impact of student programs to donors, board members, institutional leaders, and the community.

Angela Baldasare

Dr. Angela Baldasare, Senior Principal Strategic Consultant at Civitas Learning, is an experienced researcher, analyst, and consultant with a demonstrated history of success in data strategy and change management, helping organizations of all kinds use data to inform strategy and achieve measurable outcomes. Angela’s work at Civitas Learning is informed most recently by her eight years at the University of Arizona (UA). From 2014-2018 Angela served as the assistant provost for institutional research, leading the university’s student data strategy, including the use of Civitas Learning analytics, and helping to achieve record high retention and graduation rates for the institution as well as a significant rise in US News and World Report rankings.

Related Posts


« »