The axis was the wrong way round for five days, and every test agreed.

CellOracle gives us an in-silico knockout over a gene regulatory network. For each transcription factor we knock out, we read the shift in three marker gene programmes, measured separately in each of four cell-state clusters. The result is a table with three axes, and it is stored as nested JSON.

The file is nested [target][marker programme][cell-state cluster]. From 1 August the serving layer read it as [target][cell state][programme]. On 6 August a commit whose subject line announced it was correcting an inverted claim rewrote the reader to match the wrong order, so both halves were wrong together. The product's headline mechanistic claim was inverted for five days, and the test suite was green the whole time.

Why nothing noticed

The three programme keys are OPC-like, AC-like and OC-like. The four cell-state keys are OPC-like, OC-like, Other and AC-like. Three of the four are spelled exactly like the three programmes, because the programmes are named after the states whose identity they mark. Both axes are string-keyed maps, so TypeScript cannot tell them apart, and a swapped read does not throw. It returns a real number from the wrong cell.

The second mechanism is the one that kept it alive. OPC-like against OPC-like is the diagonal where the two axes coincide, so that cell is identical under either reading. It is also the only cell anybody ever quotes, because it is the headline. Five days of people looking at the number saw the correct one.

Settling it without trusting any of the code

The awkward part of an argument like this is that the code, the prose and the test had all been corrected to match each other rather than to match the data, so none of them is admissible evidence about which way round the file is. Four lines settled it, and no line depends on the serving layer or on the test that restated its assumption.

• The artefact has not moved since it was generated. The checked-in file, the copy in Cloud Storage, the generator script as it was run, the reference dataset and the base network all match on sha256. There was no undocumented transpose between generation and check-in.
• The generator writes the programme first. Its sibling script parses the result with the second subscript as a cell state, which only works on one of the two orders.
• The key vocabularies settle it without reading any code at all. The middle level has exactly three keys, the inner level has four, and the fourth is Other. Other is not a gene programme. It is a cell population, and it sits on the inner axis.
• The reference dataset confirms it empirically. Its per-cell stratum column carries exactly four categories whose cell counts match the artefact's own recorded counts and sum to 40,000, and marker expression per cluster behaves the way a real cell-state axis should.

Other is not a programme. It is the only key that distinguishes the two axes by inspection, and it sits on the inner one.

The part that generalises: a test cannot restate the reader

The guard that shipped alongside the bad fix re-implemented the same indexing as the serving layer and called its output truth. It passed, because it was asking the reader to confirm itself. This is not a careless test. It is the default one: whoever writes the reader also writes the fixture, and the fixture inherits the assumption.

The replacement puts the fixture on the other side of the boundary. The Python generator emits three fixtures, and the TypeScript contract test decodes them. Each leaf value encodes its own coordinates arithmetically, so every cell in the fixture is unique and the test can check that the coordinates it asked for are the coordinates Python stamped in. There is no hand-written expected value anywhere in it.

A fixture hand-written in TypeScript, or hand-written in JSON by whoever wrote the reader, restates the reader's assumption and therefore cannot detect that disagreement.

Three fixtures, because they fail differently. One has disjoint key sets on the two axes, so a transposed read finds no key at all and fails loudly. One reproduces the real vocabulary, where three of four cell-state names collide with the three programme names, and that is the quiet case, the one that actually shipped. The third carries the per-marker atoms.

Three changes, in order of how much work they do

• The artefact refuses to be emitted if it is undetectably transposed. The schema validator now rejects any artefact whose two axes carry identical key sets, on the grounds that no consumer and no test could then tell a transpose from a correct read. That check would have caught the original bug at generation time.
• The two axes have distinct nominal types on the reading side, and there is exactly one accessor. It takes the programme and the cell state as separately branded arguments, so it does not compile with them the other way round. The axis order is read from the artefact's own declared schema rather than assumed, and a mismatch throws when the module loads rather than when a number is served.
• No module indexes the table by hand any more, and the contract test asserts that too.

The deployed service went further and dropped the nesting. Its client now takes a flat list of cells, each naming both of its axes explicitly. There is then no nesting order to get wrong.

What we take from it

The failure needed three things at once: two axes over an almost identical vocabulary, a diagonal cell that reads the same either way and is the only one anybody quotes, and a test written by the same hand as the reader. Remove any one of them and it surfaces in an afternoon. That combination is not rare in scientific software, where the axes of a result are usually named after the biology and the biology reuses its own names.

The cheap defence is not a better test. It is making the two axes distinguishable by inspection, so that a fourth key like Other exists on exactly one of them, and then refusing to serve the artefact at all when they are not.