Reproducing a Working-Memory Figure — and What Didn't Match

Every post so far in this thread has been my own experiment, on my own terms. This one is different on purpose: take a specific, concrete claim from a published paper, try to reproduce it on the same public data, and report what actually happened — including if it doesn’t match.

The claim

Pavlov et al., 2022 collected EEG from 86 people doing a digit-span task: listen to a spoken sequence of 5, 9, or 13 digits, then either recall them aloud (memory condition) or do nothing (listen-only control). They report, in their Fig. 3d, that frontal midline theta — EEG power around 6 Hz at electrode Fz — increases in the memory condition compared to the control condition. No numbers are given in the paper text for this, only the figure and the qualitative claim.

That’s the target: a specific electrode, a specific frequency band, a specific direction of effect.

What I did

I downloaded three subjects’ EEG from the dataset (OpenNeuro ds003838, publicly available) — the first three subjects marked as having usable EEG in the dataset’s own participant table. For each trial, I found the last digit presented, took the 3-second window right after it (the retention interval before recall, per the task’s own timing description), and computed log power in the 4–8 Hz theta band at channel Fz.

The result

Frontal (Fz) theta log-power, 3 subjects
Subject	Control	Memory	Memory − Control
sub-032	-1.513	-1.624	-0.111
sub-033	0.063	-0.008	-0.071
sub-034	0.211	0.243	+0.031

Two of three subjects went the opposite direction from the paper’s claim. The third went the right direction, barely. Averaged across all three: a mean difference of −0.050 (memory lower than control, not higher), with a paired t-test of t = −1.190, p = 0.356 — nowhere near significant, which with n=3 was never going to tell me much either way.

Why I’m posting a non-replication anyway

The honest version of this post is “my small, fast version of this analysis didn’t show the effect the paper reports,” not “I disproved a published finding” and not “I’m quietly not posting this because it didn’t work.” The checklist from the last post applies to my own results as much as anyone else’s: small sample, one dataset, one set of methodological choices I made without matching the original paper’s exact pipeline. A result that doesn’t replicate under those conditions is informative about the limits of a quick reproduction attempt — it’s not strong evidence either for or against the original claim.

The code

Every script in this thread — the leakage experiments, the calibration curve, the wrist-vs-chest comparison, and this reproduction attempt — is public: github.com/anonymnd/EEG. The commit history follows the same order as the posts, including the CSP-pooling failure and the class-imbalance metric bug, left in rather than cleaned up after the fact.

Sources & papers I used while learning this

Pavlov et al.2022

Pupillometry and electroencephalography in the digit span task

The dataset and the claim this post tries to reproduce: frontal midline theta increasing in the memory condition vs. a listen-only control, reported in their Fig. 3d.
Read paper