← CHANGELOG
Leave-one-out cross-validation: where the joint model helps, and where it hurts.
29 June 2026·4 min read·Onkydra · Faith Ogundimu
Note added 2026-07-30, corrected 2026-08-13. The n=49 figures below are left exactly as they were, because they record an experiment actually run against that anchor. The DMG anchor was afterwards corrected to n=60: the original derivation left HIST1H3B (H3C2) out of the queried gene set entirely, so every H3.1 case was missed. This note previously said the derivation queried the wrong HIST1H3B symbol, which is not what happened and is not supported by the code in any commit. The finding this post reports, that a joint model helps Brier and hurts log-loss at very small n, has not been re-run against the corrected anchor.
The methodology preprint claims that fitting a joint model to the anchor cohort preserves mutation structure in a way a per-gene marginal model can't. This week we tested that claim with leave-one-out cross-validation. The result does not unconditionally support it.
The setup
For each held-out patient we re-fit the model on the remaining cases, then predict each gene's mutation status two ways: a per-gene marginal baseline, and a joint model that conditions on the patient's other genes. We score both against the truth with Brier loss and log loss, aggregated across folds. Lower is better.
DMG (n=49): joint helps Brier, hurts log-loss
Pooled across the panel, the joint improves Brier loss (6.40% to 6.06%, about 5% relative). The gains concentrate on the well-supported driver genes, which is where a joint model should help.
Log loss tells a different story. Pooled marginal log-loss is 0.2302; joint is 0.2500, so the joint is worse overall. The genes with only two or three carriers in n=49 become overconfident in the wrong direction. At very small n, joint information helps the headline drivers and hurts the uncommon events.
Brier rewards calibration on the whole-cohort distribution. Log-loss heavily penalises confident wrong predictions on rare events. The fact that they disagree at n=49 is a real finding: at small n, joint information helps where you have support and hurts where you don't.
Ewing (n=222): joint wins on both
Pooled across the Ewing panel, the joint wins on both metrics: Brier 3.41% to 3.26%, log-loss 0.1389 to 0.1326. At n=222 it is unambiguously better than the marginal baseline.
Headline finding
The threshold for an unambiguous joint-over-marginal gain sits around n=200 at our panel sizes. Ewing clears it; DMG doesn't yet. The implication for the methodology preprint: report both metrics, not just the Brier win, so the small-n caveat stays visible. The room for the joint to consolidate as the anchor cohort grows is real, but it has to be shown, not assumed.