A stumbling block to innovation in higher education is often not technology or talent, but simply initiative fatigue. An overload of well-intentioned pilots and programs can exhaust the dedicated faculty and staff working to bring them to the students of any campus. As pointed to in the League for Innovation’s most recent Trends Report, campus leaders are struggling as they work to scope, try and test initiatives with the goal of ascertaining what is working best, and for whom.
Examples of good ideas that result in hard-to-measure student success programs include sweeping policy changes, targeted scholarships, course redesign and direct outreach to students. The University of Arizona and Austin Community College recently shared how they are creating, executing, evaluating and optimizing their student success work through an integrated analytics strategy that measures the impact of their initiatives.
At the 2016 EDUCAUSE conference, partners from those two institutions explored the way they are combatting initiative fatigue by getting to the truth of what is working and what isn’t in their session: Initiative Impact Analysis to Prioritize Action and Resource Allocation.
The presenters (pictured above from left to right) included:
Melissa Vito, SVP Student Affairs and Enrollment Management and Senior Vice Provost, University of Arizona
Angela Baldasare, Assistant Provost for Institutional Research, University of Arizona
Virginia Fraire, Vice President of Student Success, Austin Community College
Laura Malcolm, VP Outcomes and Strategy, Civitas Learning (moderator)
The session focused on first-year students and supplemental education, demonstrating for the audience how leaders across higher education can move their teams past initiative fatigue to make strategic decisions around resource allocation and prioritization of programs to positively affect student outcomes.
The University of Arizona’s Think Tank
Melissa Vito and Angela Baldasare from the University of Arizona shared findings from their impact analysis of Think Tank services.
Melissa Vito, the University of Arizona
The Think Tank was launched in 2009 to house a number of student services offered by nationally certified tutors. The University’s previous work to measure the efficacy of the Think Tank’s offerings met with challenges. “It took about two weeks per assessment for our IR office to corral the data, put it all together and run the models needed,” said Angela Baldasare. “They were rigorous and did good work to control for selection bias, but it just wasn’t scalable. So we were thrilled when Civitas Learning introduced an automated, scalable solution for us in Illume Impact. Illume Impact uses prediction-based, propensity score matching to measure the impact our Writing Center is having on persistence.”
Angela Baldasare, the University of Arizona, speaking at the Civitas Learning Summit
Illume uses an institution’s data to develop personalized predictions that are timely, accurate and actionable. Illume Impact allows users to see the persistence lift caused by their initiatives and provides valuable insights to help leaders better understand the needs of a diverse student body.
By combining Illume’s probability predictions with propensity-score matching, the team was able to get to rigorous, statistically significant results, and could then sort these results by student segments.
Propensity score matching allows for the creation of a control group that shares characteristics with the test group, and can be used to demonstrate how individuals would fare in the absence of an intervention.
Prediction-based propensity score matching (PPSM) is a statistically rigorous method for evaluating intervention efficacy, and meets guidelines by the U.S. Department of Education’s What Works Clearinghouse.
It also allows institutions to account for confounding factors like selection bias. “The beauty of it,” said Baldasare, “is it’s quick and scalable but it also can break the results into student segments.”
The university found that students using the Think Tank’s Writing Center showed a 2.7 percentage point increase in persistence overall. The lift was higher, at 3.4 percentage points, for first-year students, an important cohort for the university and the focus of some of their most strategic initiatives. Students at highest risk of not persisting showed the greatest gains in persistence, at 7X greater than moderate risk students.
“We also ran this on supplemental instruction,” said Baldasare, “which is based on nation-wide best practices. Asking faculty to refer students to a writing center is not necessarily a tough sell, but asking them to participate in something like supplemental instruction is a much bigger ask,” she explained. “They have to actively collaborate with the Writing Center, and they need to bring in instructors and build out curriculum.” Baldasare says being able to show statistically significant results of efficacy of the programs has engaged faculty as she takes the message across campus.
Austin Community College’s ACCelerator
Virginia Fraire shared how Illume Impact was helping her team understand the lift for students using Austin Community College’s ACCelerator. As part of a larger $4.9 million grant-funded guided pathways project, Austin Community College built the highly specialized, technology enabled ACCelerator learning environment which offers, among other things, Math Development Education (DevEd) courses to a portion of the students Austin Community College serves each year.
Virginia Fraire, Austin Community College
“We partnered with Civitas Learning to deploy Degree Map, which has become the central application that we have used to enhance our advising practice at the institution, but today I want to talk about another initiative – the ACCelerator,” said Fraire. “We enroll approximately 100,000 students across 11 campuses when you account for academic, workforce and continuing education programs. We recently bought a former shopping mall and created one of the largest learning environments in the US with more than 600 computers and multiple study rooms. It’s changing the way we engage with our students. The primary focus is on developmental education. We wanted to help our students succeed faster, and move them away from traditional 12-16 week DevEd Courses.”
Students are each assigned a faculty member that serves as their academic coach and also receive multiple wrap around services to create a comprehensive package of academic support.
Using Illume Impact, ACC was able to see an overall impact of 6.5 percentage points increase in persistence for students who used the ACCelerator as compared to those who did not. When segmenting to just DevEd students, the impact was almost 4X greater for DevEd students with a lift in persistence of 12.44 percentage points compared to a 3.29 percentage point lift for the non-DevEd students.
The ACCelerator, with Associate Registrar for ACC, Tomi Welch
“We are pleased to see this tremendous success over the school year and now plan to build additional ACCelerators at other campuses. In order for this kind of work to be successful you have to break down a lot of silos,” said Fraire. “Staff members at the ACCelerator report to me but the faculty report along different lines. We have to work together and there has to be trust.” Fraire said being able to measure and know what’s working is helping to build and maintain the communication and collaboration necessary to succeed. They are now leaning into research about accelerated learning pathways for DevEd students and plan to use a combination of Civitas Learning solutions to reorganize advising and counseling around effective developmental math options and guided pathways.
In closing, Laura Malcolm added, “In looking at the initiatives around this country, and at these two institutions, the work they are doing is compelling. As this new data comes into the conversation it allows the conversation to extend. It’s allowing them to have confidence in their initiatives as they move forward. We’re excited about the potential this has to get us to the gains we’re really looking for in higher education.
RELATED: FROM INNOVATION TO IMPACT
Civitas Learning Chief Learning Officer and Co-Founder Mark Milliron, Chief Data Scientist David Kil, Senior Vice President of Outcomes & Strategy Laura Malcolm, and Product Manager Grace Gee explain why rigorously evaluating the impact of innovative student success initiatives is key in meeting institutional goals for student outcomes, resource allocation and return on investment.