The One Experiment Where the Split Didn't Matter

Reading about adaptive learning for this thread led somewhere I didn’t expect: not another EEG dataset, but a question about whether the entire leakage story I’ve been telling even generalizes past neuroscience.

Why not EEG this time

The original idea — log question timing, retries, and hint use from my own study sessions — needs a logging tool that doesn’t exist yet. Rather than fabricate data to keep the thread moving, I looked for a real public dataset with the same shape of signal, and found one: ASSISTments, a tutoring platform whose 2009-2010 interaction logs (523,534 question attempts, 4,213 students) are a standard benchmark in educational data mining. Corbett & Anderson, 1994 Each row records whether an answer was right, how long the first response took, how many attempts and hints were used, and how many times that student had practiced that specific skill before.

A detour through learning styles first

Before touching the data, I read around the “learning styles” idea — the claim that matching instruction to a visual/auditory/kinesthetic style improves learning. The honest summary: weak to no support. Pashler et al., 2008 found the supporting research lacked the randomized designs needed to trust it, and a more recent Clinton-Lisell & Litzinger, 2024 meta-analysis found only a small effect, described as too small to justify adopting the theory. A better-supported individual-difference claim is “structure-building” ability — McDaniel, Marsh & Gouravajhala, 2022 found it predicts comprehension consistently, in the lab and the classroom. None of this fed directly into the experiment below, but it shaped what I was willing to assume going in: not much.

The experiment

Same structure as every leakage experiment in this thread: predict a binary outcome (here, right vs. wrong) from a small set of features, using logistic regression, compared across a random 5-fold split and a student-wise split (GroupKFold by student ID), plus a shuffled-label null.

The features are deliberately generic — nothing about which skill or question, only about the interaction itself: log response time, attempt count, hint count, total hints available, log(opportunity — how many times this student has seen this skill before), and whether a bottom-out hint was used.

The result

ASSISTments: predicting correctness, random vs. student-wise split
Split	Balanced accuracy
Random (leaky)	0.866 ± 0.002
Student-wise	0.865 ± 0.006
Null (shuffled)	0.500 ± 0.000

The gap is 0.001. For comparison, the smallest gap anywhere else in this thread was 0.027 (motor imagery), and the largest was 0.182 (EEGMAT). This one is effectively zero, on the largest sample in the whole series — 4,213 students, against 10-36 in the EEG and wearable experiments.

What this adds to the checklist

The checklist from earlier in this thread said the leakage gap’s size depends on the task and the dataset. This result sharpens that: it depends specifically on how much person-identifying information is baked into the features, not the raw signal type or the task difficulty. A gap near zero is not proof that evaluation discipline doesn’t matter here — the student-wise check is still the only way to find that out — but it is evidence that this particular feature set isn’t quietly encoding “which student is this” the way spatial EEG patterns do.

Sources & papers I used while learning this

Pashler et al.2008

Learning Styles: Concepts and Evidence

The classic review behind the "learning styles" discussion in this post — argues the research supporting style-matched instruction lacked the randomized designs needed to trust it.
Read paper
Clinton-Lisell & Litzinger2024

A meta-analysis of learning styles

A more recent, more sympathetic look at the same question — found a small positive effect of matching instruction to style, but describes it as too small and infrequent to justify adopting the theory.
Read paper
McDaniel, Marsh & Gouravajhala2022

Individual Differences in Learning From Educational Materials: The Case of Structure Building

A more defensible individual-difference claim than learning styles: structure-building ability (forming coherent mental models of text) predicts comprehension consistently across studies.
Read paper
Corbett & Anderson1994

Knowledge tracing: Modeling the acquisition of procedural knowledge

The foundational paper behind knowledge tracing, the technique adaptive tutoring systems use to estimate what a student currently knows from their response history.
Read paper