Intervention design and data instrumentation in the context of causal inference need to be intentional. That is, intervention, data, and associated analytics must be designed to answer specific business questions with minimal attributional ambiguity. Here are a few business questions related to causal inference.
Is my investment in tutoring paying off?
I am running a mindset nudging campaign. How can I tell if it is working?
How do I target the right students and when should I reach out to them? How do I know I am doing the right thing?
Here we explore how to design interventions that leverage all the salient concepts from causal inference, machine learning, and behavioral science.
First of all, each intervention needs to have a specific student success metric that it is trying to influence based on good science. That is, for a particular student segment identified for an intervention, what are the impact levers? If one can move these levers in a favorable direction, which student success metric will serve as an arbiter of program efficacy or even ROI? In Bayesian network inference, we create a directed acyclic graph, where we identify variables that influence another variable or, more specifically, cause changes in that variable. A similar exercise can be helpful here. The more direct the linkage and the shorter the time duration between the impact levers and the success metric, the better.
Now, we need to think about intervention design. Will the program be available to everyone or will there be eligibility criteria? Will the program be based on A/B testing or in an observational setting? Are my historical success metrics consistent over time or do they vary wildly, thus making pre-post matching a very difficult task?
Next, what kind of data should be collected to assess not only participation but also the quality of participation? Furthermore, which input data about students should be used to ensure proper matching if A/B testing is not feasible?
Finally, how can we improve the intervention program further? This is particularly important if the initial ROI based on impact analysis is negative.
For example, let’s say that you are trying to implement a student success program consisting of (1) mind-body health programs (meditation, yoga, cognitive behavioral therapy, etc.) and (2) career-related meetups, where successful alumni in various industries come in and talk about their experiences and learnings. How will you leverage the intentional design concepts discussed here?
First of all, these two components of the student success program should be split into two since they are trying to influence different student success metrics. Success metrics for the mind-body health programs can be reduction in stress level and improvement in persistence due to a sense of belonging and camaraderie formed with other participants through working out together. How can we obtain stress level? Micro survey or ecological momentary assessment.
What about meet-ups? What metrics would be affected by students listening to successful alumni? How about tracking changes in majors associated with the careers of the meetups students went to? For example, if a higher % of students stayed in or switched to the major of the meetups they went to, would you consider the program a success? On the other hand, by exposing students to the realities of the careers they are interested in, we may be improving their persistence since they are more highly motivated to stay in and finish so that they can pursue the right careers aligned with their life goals.
In terms of program participation data collection, we can use a card-swipe system to track students’ ins and outs at these events. As long as students understand that this kind of data can be used to assess the efficacy of various stunted success programs to help optimize their school experience, there may not be a need for IRB approval. Manual attendance tracking is not recommended due to its unreliability. Quality of attendance can be measured by frequency and duration, along with social network analysis to see if some students attend events together.
In terms of student data to be used in matching, we would like to have variables in incoming factors, academic performance, academic progression, engagement, financial aid, and social psychological factors available through survey data. While matching algorithms will accommodate and personalize to available data, the higher the accuracy of predictive models, the lower the minimum detectable effect size.
If the eligibility criteria are too restrictive to allow baseline matching and historical trends on success metrics are unstable, we should seriously recommend A/B testing since there is a lot of noise even for pre-post matching. Another option is to relax the eligibility criteria such that baseline matching is feasible.
In summary, we outlined the steps involved in intentional intervention design and impact analytics. Always map the objectives of the program to success metrics. Even more importantly, use analytics to understand and quantify impact levers for the targeted student segment since the objectives of an intervention program should be aligned with the prioritized impact levers. After that, data instrumentation strategies need to be developed along with matching strategies for baseline matching or pre-post matching under certain conditions. The reward of such a thorough process is a scientific impact measurement that will facilitate continuous impact and process improvement in student success.
How we leverage signal analysis to evaluate new data sources
At Civitas Learning, we are on a constant quest to explore which data sources can contribute to the art and science of student success. In this white paper, we explain how we leverage salient concepts in event signal processing, causal impact analysis, and learning models to quantify the relative contributions of different data sources. Such knowledge is invaluable in helping us build the most effective, personalized student success platform.
Using predictive and impact analytics to identify and rank intervention opportunities
While predictive models have been helpful in identifying students based on their success scores, they are a piece of a big jigsaw puzzle of student success science. In this white paper, we explore how we can fuse salient concepts from predictive science, impact analytics, economic theories of elasticity, and institutional knowledge to create and ramp up a rapid learning cycle of impacting student success through targeted intervention ranking and recommendations.
Dave Kil, Chief Data Scientist
Dave Kil has more than 20 years of experience in building various analytics apps and solutions spanning nonlinear time-series analysis to predictive analytics, outcomes research, and user experience optimization. He and his team are working on (1) improving predictive algorithms to provide much more actionable insights, (2) adding new capabilities to automate ROI and outcomes analyses as part of action analytics, and (3) making the Civitas Learning analytics platform self-learning and more intelligent over time. He holds 14 U.S. patents, is the author of a book on pattern recognition and predictions, and has published a number of articles in journals. He currently serves as chief data scientist with Civitas Learning.