What We’re Learning Together

partners at summit session

We’re excited to announce our first Community Insights Report, which was just featured on NPR Marketplace. The report pulls data from more than 2 million active students at more than 55 institutions together. Getting to this moment, however, has not been easy.

When you pull together institutions from across the higher education spectrum committed to helping students learn well and finish strong in an effort to learn together from the mass of data at their fingertips, you have to decide where to start. You can start at the highest level first. Indeed, one strategy might be to benchmark publicly available research reports and then draw on a common set of data elements across a community and build generalized models to see what you can learn. We didn’t go that route for a number of reasons. First, you run the risk of using common data elements to reinforce common beliefs at a general level. As a result, you’re very likely to end up in an End of Average quandary of generalizing across institutions to the extent that insights and recommendations for specific institutions, or specific students, aren’t useful—and can add to an already troubling amount of initiative fatigue. But most of all, unlocking the richness of institution-specific insights and better informing and inspiring front-line actions are important if the goal is not just to research and report, but to really move the needle for any given striving student.

With this in mind, our cross-cutting community insight work began by going first to the individual-student level in a specific institution. It was hard work, but great work. We’ve drawn data from diverse sources, created and competed thousands of derivative variables that are more powerful and incisive than traditional demographic or point variables, and then developed more than 1,500 institution-specific models that optimize available data and power precision with specific students. These personalized models were then used not just to power one tool, but to better inform, instrument, and measure the impact of policy development, practice changes, and a variety of apps that went directly to students, faculty, advisors and administrators. As we launched this first-of-its-kind Student Success Platform in higher education, we leaned on Summits and digital sharing sites like this one—our Civitas Learning Space—to socialize and spread learnings from individual campuses. Case studies, webinars, research reports, and learning briefs were available, and conference sessions from partners began to showcase their compelling work to turn the lights on and turn the DIAL.

whatwerelearning_insidepic1

Now we’re finally at a scale where we can step back and look at what we’re learning across institutions in a more compelling way. This week we’re showcasing our first-of-many-to-come Community Insights reports. These reports will look at issues and insights from across the Civitas Learning community to shed light on what we’re learning, as well as areas ripe for further research, innovation, and action. To download the full report in e-book form, click below.


DOWNLOAD REPORT

This report focused on four key insights, which I’ll summarize below. However, I’m also going to add a bonus one at the bottom of this post. Here’s the handful:

  • Engagement Matters: For years, the feeling that “demography is destiny” has dominated research reports. Indeed, pronouncements about the challenges of diverse students, low-income students, and males in particular have been the leading headlines. This isn’t too surprising when far too often, research in education ends up being regression against demographic variables. While we clearly are seeing equity gaps as an issue, what comes out loud and clear when you look across our institutions is that what students do matters in a big way. For example, 80% of our institutions have variables derived from LMS or tutoring-center activity data in their top ten predictors list. Think participation in class discussions relative to given sections, activity streaks, or new-student access patterns for curriculum resources in the first weeks of courses. These kinds of engagement data are predictive for both on-ground and online students. And more than 64% of students’ success predictions shift throughout the term because of activity data.
  • It’s Not Just Failure: Enormous energy and innovation is focused on improving courses, courseware, and course pathways. Newer innovations tackle academic issues with competency models, core curricular expectations, and new credentialing strategies. So much energy goes into this work because many in education think academic challenges are the main issue. What we see across our institutions is that isn’t necessarily the case. Indeed, the majority of students leaving our institutions—in both access and selective institutions—are leaving in good standing. On average, 42% of non-persisting students have a 3.0 GPA or above. And 76% of non-persisting students did not have a failing grade in previous terms. It seems that the reasons they are leaving are more likely related to life and logistics—can’t get the courses or schedules they need, work isn’t cooperating, money issues, etc—or psycho-social issues—mindset, grit, or belongingness.
  • Course Grade Signals Matter: While nonacademic issues are powerful, grades remain an important signal on academic pathways. They are often, however, subtler than we think. In the Community Insights report, four types of common grade signals are called out. The first signal won’t surprise you: it’s called Challengers, typically defined by courses that we would traditionally call “Gate Keepers” or “Gateway” courses with high “DFW” rates. Qualifiers are another type and include courses where only getting an “A” is predictive of success—think of a biology major taking Intro to Biology. Late Hurdles are signals from courses taken later in programs where students need to get an “A” or “B” to have a solid chance of succeeding. One of the more interesting sets, however, are Yellow Flags. These cautionary signals come in courses taken early in programs where the vast majority of students pass. However, getting a “C” puts a student on a perilous path. They are very likely to persist for a year or more, but significantly less likely to graduate. Think of this signal as a counter on a curricular time bomb of challenge. English 102 Composition courses often show up at yellow flags, with students showing early challenges in writing, but not being required to write significantly until much later in their programs. These signals are institution, program, and major specific; and will be incredibly interesting to unpack in specific institutions.
  • Knowing What Works: Some of the most interesting work across the Civitas Learning community comes in the form of impact analytics. Because we have student-level predictions for every student at an institution, we’re able to power prediction-based propensity score matching (PPSM) to create control groups and study the impact of different interventions. This tool, along with Course Insights, is being built right into Illume. What we’re seeing in studying innovations across the Civitas is the emergence of findings surrounding both our action applications and policy and practice strategies at colleges. For example, we’re seeing consistent persistence gains for initiatives like Student Success Courses, Math Tutoring, Inspire for Faculty, and Inspire for Advisors. What’s compelling about this work is that most of our partners are reporting real challenges with initiative overload and initiative fatigue. I’m sure we’ll be seeing more of our partners parse and focus their initiatives in the coming years to maximize impact.

Now I’d like to add a final insight that we didn’t include in the report. Nonetheless, it is arguably one of the most important to note.

  • This Work is Changing Things. We are definitely seeing our partners move the needle on student success. From big policy and practice changes to the smallest of nudges to students, it’s not hard to imagine that this work will change the way our students experience education in the months and years to come. Moreover, our institutions are telling us that they are experiencing a substantive shift in how they operate. Higher education’s data focus for so long has been primarily on accountability and reporting. Readying our colleges and universities to house, govern, and best leverage these types of institution-specific student success platforms and apps involves change of another kind. We’re seeing new leaders emerge in IR, IT, academics, and student services. We’re also seeing new positions being created, such as Chief Data Officers and Directors of Institutional Analytics, to clearly differentiate this work from reporting and compliance. But most of all, we’re seeing a bias toward action. Indeed, a virtuous cycle of insight and action, where insight guides teams and individuals—e.g., administrators, faculty, advisors, and students—to action, and this instrumented action leads to learning which leads to more insight, which leads to ever-more-precise action. As this cycle continues, we hear from our institutions that a sense of a moral imperative is emerging, where the fact that they have clearer signal that a student is in danger drives urgency to act. Likewise, signal of students moving in promising directions is motivating outreach to further guide student journeys toward higher achievement, greater agency, deeper learning, and timely completion. It’s inspiring to see.

I pointed to this last finding of organizational change work in the Seven Strategies for Analytics and the Road Ahead; and we are certainly seeing these strategies playing out. This is why we are diving deeper into consulting and even developing leadership development programs for our campuses (more on that in the next blog). It’s also one of the reasons being in this learning community matters. Because one of the most powerful strategies at our fingertips is to continue to learn together.

Exciting times at Civitas Learning. But most of all, we’re honored to be in common cause with these committed educators and institutions. Learning together about how to best improve learning and completion is compelling work to say the least!

 

*Just FYI, we’ll dive deeper into this and future Community Insights reports on a webinar in August. Stay tuned! 

4 Comments

Trackback  •  Comments RSS

  1. Curt Eley says:

    Did your research explore and make any findings about the difference in student success between institutions using 10 point grade bandwidths (B = 80 to 89 for example) versus institutions using more granulated grades (B- = 80-83, B = 84-86, B+ = 87 to 89)?

    I’m wondering about both the motivational incentive and the signaling aspect of more granulated grading systems.

    thank you

    • Mark Milliron says:

      Great question, Curt! We didn’t do that analysis for the community report. However, that’s exactly the kind of action research our partner institutions are doing with Civitas in their own student success platform work using their own institution-specific data.

  2. Alice Horning says:

    The Marketplace report quotes you as saying that 45% of students who begin any post-secondary education don’t finish. Do you have a source for that claim? I am a linguist and reading scholar and professor very interested in student success. I am surprised that your report, which I have read, does not examine the role of student reading abilities (or lack thereof) in persistence and graduation. Research by ACT and other scholars shows that about half of high school graduates do not have the reading skills they need to be successful in college. You might want to investigate this part of the problem.
    Alice Horning
    Professor of Writing & Rhetoric/Linguistics
    Oakland University
    Rochester, MI
    horning@oakland.edu

    • Mark Milliron says:

      Alice, the 45% number comes from NCES, Gates, and Complete College America; and, remember we’re talking across 4-year and 2-year institutions. It’s pretty widely used. On reading, it’s a great point. MANY of our individual institutions have been doing work w/ reading and have found interesting trends, including reading data’s power in predicting success/challenge in everything from overall persistence to completion in health professions to the passing of gateway math courses–some really interesting work. In the Community Insights reports we only pick a few global trends to report on across the Civitas Learning Community. We’ll definitely slate reading dynamics as one for a future report! Mark

Post a Comment

Your email address will not be published. Required fields are marked *

Top