gears and wheels art

Initiative Impact Analysis for Institutional Researchers


Share this Post

At the recent AIR Forum event, Civitas Learning’s Chief Data Scientist, Dave Kil, was joined by Montgomery County Community College’s Executive Director of Institutional Research, David Kowalski, on a panel hosted by Rob Robinson, Principal Solutions Consultant at Civitas Learning. Kowalski leads the college’s analytics work. This video is of that conference session.

Kowalski and Kil shared how initiative impact analysis and on-demand predictive data have enabled Montgomery County Community College’s (MCCC) Institutional Research team to scale its capacity, get to results faster, reallocate more time to decision support, empower institutional stakeholders to take action and reach out to students, and spark productive conversations about program effectiveness and ROI/investments.

Measuring Impact

In this video, they’ll walk viewers through the progress MCCC has made in moving from descriptive analytics to predictive analytics. Kowalski shares impact the college is seeing by being able to have rigorous analysis of initiative outcomes, including a measured 6.2%(pp) lift in persistence overall for students using the tutoring center. Listen as Kowalski explains the importance of being able to drill in to see segmented student population impact for each initiative.

“As part of our iPASS work we were looking at tutoring services,” said Kowalski. “It’s something we offer all students at the college and we wanted to know the effect of our bigger initiatives on our students. There had not been a lot of time or space to evaluate the tutoring program. The general assumption is that programs designed out of good intent will be useful. But, being able to do a rigorous evaluation with prediction-based propensity score matching is valuable. Tutoring is having a significant impact. Knowing this allows us to ask more targeted questions and create more targeted outreach.”

Kowalski speaks to the added importance of having this data so that when teams running initiatives want to ask for more resources or funding they can demonstrate the value and effectiveness of their programs.

To understand how this impact was measured David Kil explains the process of prediction-based propensity score matching and the value of providing near real-time impact feedback to the college.

Adding Capacity

Kowalski shares how the Civitas solutions are adding capacity for his IR office to spend more time focusing on gaining deeper understanding of what is working and for whom, while using dashboards to still provide some of the required ad hoc reporting to meet requirements.

“Adding this tool is like adding another person in the IR department because you can do the analysis quickly. Being able to evaluate programs in this way makes our decision making more fine grained,” said Kowalski.

He and Kil speak about the effectiveness of a new text nudge campaign MCCC has implemented and offers advice to others doing student outreach.

“We’re focused on prioritization of initiatives and what is going to have the most impact on students – that is where a lot of the work we’re doing with Civitas Learning will factor in,” said Kowalski.

Enjoy this video to see how the use of analytics at MCCC has changed the conversation across the campus and helped faculty and staff take action. Then, download this document to read about impact analysis at additional partner institutions.


Civitas Learning Chief Learning Officer and Co-Founder Mark Milliron, Chief Data Scientist David Kil, Senior Vice President of Outcomes & Strategy Laura Malcolm, and Product Manager Grace Gee explain why rigorously evaluating the impact of innovative student success initiatives is key in meeting institutional goals for student outcomes, resource allocation and return on investment.


Related Posts

« »