When describing an experiment that did not deliver the result we expected, we should be able to distinguish the expectation, the observation and the explanation we developed afterward.
Imagine ending a research report with this sentence: the expected gain was not achieved.
The sentence may be correct. On its own, however, it is not enough to understand the work behind it.
What were we expecting? Under what conditions did we try it? How far was the result from the target? Did the experiment run properly? Could the records we have answer another question?
This is where the difficulty of publishing an unsuccessful experiment begins. We do not want to conceal the result. Nor do we want to present every negative result as a valuable discovery.
The reader needs something more concrete: to understand what was tried, what was observed, and what decision we drew from it.
First, say what failed
“The experiment failed” can cover several different situations.
The program may not have run. There may have been a measurement error. The experiment may have finished without enough evidence for a decision. Or the hypothesis we wanted to test may not have met the criterion set in advance.
The work that follows each of these situations is different.
If the measurement is broken, it needs fixing first. If uncertainty is high, we need to make the limits of the decision supported by our data explicit. If the expected gain was not achieved, we should not leave that decision behind another measurement that looks more favorable.
A report’s first task should therefore be to describe which situation we face, before assigning meaning to the failure.
Do not change the success criterion along with the result
The experiment I discussed earlier in ADL makes this distinction concrete.
The question was: does measuring two learning interventions at each step and choosing the better-looking one lead to a better result within the same amount of time?
The success condition set in advance required the candidate method to improve final evaluation accuracy by at least 3 percentage points against each of four comparison methods.
The observed difference from the strong comparison method was about 0.09 percentage points. The upper bound of the uncertainty interval for this comparison also remained below the target. The report’s primary quality hypothesis was therefore rejected.
Other comparisons in the same records looked more favorable. Choosing the better of two proposals showed an advantage over a control that measured both but used the first. That observation did not replace the original success condition.
This is the boundary to preserve when writing the report: an observation we find interesting afterward should not change the answer to the question we asked beforehand.
It may open another question. That needs a place of its own.
Separate the observation from its explanation
When a result does not meet our expectation, we want to explain why.
Perhaps the selection mechanism took too much time. Perhaps preparing the proposals was expensive. Perhaps the task we tried was not one on which the mechanism could show an advantage.
These are explanations we could investigate. But a plausible explanation is not the same as a measured one.
The ADL record showed fewer model forward passes alongside an increase in the time spent by the mechanism preparing and managing interventions. No saving in total measured time was confirmed. The same records did not establish which sub-operation caused the difference.
There is an honest place for the report to stop here:
We observed this difference. We have not yet isolated the operation responsible for it.
That sentence also explains why another experiment is needed. If we settle on an unmeasured cause, we build the next study on an explanation that has not yet been tested.
Leave the reader a route back to the result
Publishing an experiment is not complete merely because the text describing its result is accessible.
A reader should be able to follow the version used, how the data was split, the budget under which the comparisons ran, and the records from which the result was calculated. It should also be clear which runs finished, which returned errors, and which were excluded from the calculation.
This does not require every article to carry all the technical details in its main text. The main text can explain the question and the decision, with the detailed protocol, configuration and result files behind that account.
What matters is that the path from the claim in the text to the record supporting it remains open.
Being able to rerun the same code is part of that path. Yet reproduction alone does not show that the experiment asked the right question. A measurement error can also recur under the same conditions.
Alongside executability, we therefore need to understand what was measured.
Do not lose the old decision inside a new question
Trying another mechanism after a negative result is natural.
We can change the budget, choose a different task, or test only one part of the method. Each may be a meaningful continuation.
But the earlier decision should remain visible when the new work begins.
If a candidate missed the target under one budget, its result under another should not be presented as a retrospectively corrected answer to the same experiment. We should be able to write down which condition changed and why that change requires a new test.
In the previous essay on the trace of a correction in Evidrel, I considered a related need for records: when a new explanation arrives, preserving the grounds for an earlier acceptance or rejection.
Evidrel’s research notebook addresses some parts of these recording and review decisions. This should not be taken to mean that it is a finished infrastructure managing the entire experimental process. The connection returns to the same responsibility: preserving the grounds and scope under which a decision was made.
The starting point for the next study
An unsuccessful experiment does not have to yield a new method.
Sometimes the result is enough to stop advancing a particular candidate. Sometimes it reveals that the measurement setup is inadequate. Sometimes it lets us describe an unanswered question more narrowly.
When publishing, we should be able to say which of these we reached.
In the ADL example, the tested candidate was not shown to deliver a quality gain of the expected size under the specified task and budget. That decision did not close the entire research program. But it also provided no grounds for presenting the candidate as a successful improvement.
What makes such a record valuable to me is not failure itself. It is the clearer starting point it leaves for the next study.
A well-kept experimental record should be able to tell us this when we return to the same idea months later:
We tried this route. Under these conditions, it did not meet this expectation. If we try again, we need to explain what we are changing and why.
Source note — 18 October 2026: This essay considers findings from the previously reviewed ADL F2-041 experiment report and the published ADL working essay through the question of experimental reporting. The Evidrel connection is limited to the local research notebook’s approach to recording and review. No new experiment, reproduction run or independent evaluation was conducted for this essay.
Connections: ADL research page · Evidrel research page · Previous essay: The trace of a correction in Evidrel
Atlas concepts: Evidence · Uncertainty · Context · History · Knowledge · Systems
This connects to
Examines ADL F2-041’s quality result that missed the target through its predeclared success condition and the limits of the report’s decision.
Connects preserving the grounds and scope of decisions with Evidrel’s local research notebook, without claiming a finished experiment-management infrastructure.
Moves from the trace of a correction to preserving the grounds of an earlier experimental decision when a new explanation arrives.
Makes the question of leaving a route to reconsider an experimental result concrete through ERM’s existing reproducibility records.