How Much Data From One New Person Does a Model Need?
Research · NeuroAI & peopleStarting from a model that has never seen you, how many of your own trials does it take to start closing the gap? A small calibration-curve experiment on real motor-imagery EEG.
On this page
Last post ended with a weak but real cross-subject gap: a model trained on nine people and tested on a tenth it had never seen scored noticeably below one trained and tested on everyone pooled together.
That leaves an obvious next question. The model has never seen this tenth person. What if it got to see a little of them first?
The setup
Same data as last time: 10 subjects, PhysioNet motor-imagery dataset, left vs. right imagined fist, band-power features (log power across four frequency bands, 32 channels), logistic regression.
For each of the 10 subjects in turn, I held them out completely: trained only on the other 9. Then, before testing, I let the model see a small number of that held-out subject’s own trials too — added into the training set, not swapped in — and retested. I used calibration sizes of 0, 2, 5, 10 and 20 trials.
To keep the comparison fair, I fixed 20 of each subject’s trials as an evaluation set that is never used for calibration, no matter how many calibration trials were added. Calibration trials are drawn from the remaining pool. This is the same subject-wise discipline from last post, just with a controlled leak of a few trials back in on purpose — which is the whole point of calibration.
The result
| Calibration trials | Mean balanced accuracy | Std |
|---|---|---|
| 0 | 0.559 | ±0.106 |
| 2 | 0.554 | ±0.121 |
| 5 | 0.577 | ±0.122 |
| 10 | 0.597 | ±0.127 |
| 20 | 0.634 | ±0.069 |
Two trials of calibration did nothing — if anything the mean dipped slightly, which I read as noise given the ±0.121 spread, not a real effect. From 5 trials on, the trend turns upward and keeps climbing: by 20 trials the average balanced accuracy is 0.634, about 7.5 points above the zero-calibration baseline.
The standard deviation also shrinks as calibration trials increase — 0.106 at zero calibration down to 0.069 at 20. That’s a second effect worth naming separately from the average going up: calibration isn’t just raising the mean, it’s making the model’s performance more consistent from person to person. Some of what a leave-subject-out model gets wrong is genuinely person-specific, and a handful of that person’s own trials fixes exactly the part that pooled training couldn’t.
Where this leaves the thread
Two posts, one real loop: pool everyone and you inflate your score with person-recognition; hold a person out entirely and the score drops to something closer to the truth; let the model see a little of that person and some, but not all, of the drop recovers. That’s the “sense, interpret, adapt, measure again” cycle this whole project keeps coming back to, run once, honestly, on real EEG.
Sources & papers I used while learning this
-
A review of classification algorithms for EEG-based brain-computer interfaces: a 10-year update
Per-subject calibration is an established topic in the BCI literature, not something new here — this review is where I learned the vocabulary and the usual tradeoffs.
Read paper -
BCI2000: A General-Purpose Brain-Computer Interface (BCI) System
The acquisition system behind the PhysioNet EEG Motor Movement/Imagery dataset used for this experiment.
Read paper -
MOABB: trustworthy algorithm benchmarking for BCIs
Shows the same pipeline behaving very differently across different motor-imagery datasets — a reminder not to generalize a single-dataset calibration curve too far.
Read paper