← CHANGELOG
Leave-one-out cross-validation: where the joint model helps, and where it hurts.
29 June 2026·4 min read·Onkydra · Faith Ogundimu
Note added 2026-07-30. The n=49 figures below are left exactly as they were, because they record an experiment actually run against that anchor. The DMG anchor was afterwards corrected to n=60: the original derivation queried the wrong HIST1H3B symbol and missed 11 H3.1 cases. The finding this post reports, that a joint model helps Brier and hurts log-loss at very small n, has not been re-run against the corrected anchor.
The methodology preprint claims that fitting a joint model to the anchor cohort preserves mutation structure in a way a per-gene marginal model can't. This week we tested that claim with leave-one-out cross-validation. The result is worth reporting because it does not unconditionally support the claim.
The setup
For each held-out patient we re-fit the model on the remaining cases, then predict each gene's mutation status two ways: a per-gene marginal baseline, and a joint model that conditions on the patient's other genes. We score both against the truth with Brier loss and log loss, aggregated across folds. Lower is better.
DMG (n=49): joint helps Brier, hurts log-loss
Pooled across the panel, the joint improves Brier loss (6.40% to 6.06%, about 5% relative). The gains concentrate on the well-supported driver genes, which is where a joint model should help.
Log loss tells a different story. Pooled marginal log-loss is 0.2302; joint is 0.2500, so the joint is worse overall. The genes with only two or three carriers in n=49 become overconfident in the wrong direction. At very small n, joint information helps the headline drivers and hurts the uncommon events.
Brier rewards calibration on the whole-cohort distribution. Log-loss heavily penalises confident wrong predictions on rare events. The fact that they disagree at n=49 is a real finding: at small n, joint information helps where you have support and hurts where you don't.
Ewing (n=222): joint wins on both
Pooled across the Ewing panel, the joint wins on both metrics: Brier 3.41% to 3.26%, log-loss 0.1389 to 0.1326. At n=222 it is unambiguously better than the marginal baseline.
Headline finding
The threshold for an unambiguous joint-over-marginal gain sits around n=200 at our panel sizes. Ewing clears it; DMG doesn't yet. The honest implication for the methodology preprint: report both metrics, not just the Brier win, so the small-n caveat stays visible. The room for the joint to consolidate as the anchor cohort grows is real, but it has to be shown, not assumed.