What do we learn when an experiment does not deliver the expected result?

ADL working notes — 01

Sometimes an experiment ends with a single sentence: the expected gain was not achieved.

Yet questions remain behind that sentence. Where did the mechanism help? What did its small advantage cost? Which expectation should we let go of, and which question should we pursue?

One of the recent ADL experiments made these questions concrete. Comparing two learning interventions and choosing the one that looked better was a reasonable starting idea. The result showed that assessing the value of this choice required more careful accounting.

Does a better choice lead to better learning?

ADL is a research project that treats the learning model and the mechanism selecting interventions separately. How a change is chosen is as much a subject of research as how the model changes.

This experiment tested a narrow question:

If we measure two proposed changes at each step and choose the one that reduces training error more, do we get a better result within the same time?

The mechanism worked as follows: two separate trial copies were made from the model’s current state, and one proposal was evaluated in each copy. The proposal with lower training error was selected; if the required improvement condition was met, only that change was transferred to the live model.

The decision therefore used more information than trying a single proposal. Obtaining that information also had a cost.

Setting the criterion before seeing the result

A method that looks good in some comparisons can easily seem convincing. That is why the experiment’s success criterion was set before the runs.

Within the same one-second CPU budget, the candidate was expected to improve final accuracy by at least 3 percentage points against each of four comparison methods.

This condition mattered. Beating a weaker method would not establish sufficient progress against a strong one.

The experiment used a small synthetic classification task. There were twenty different random seeds, six methods, and three timing repetitions. Accuracy on held-out evaluation data not used during training was measured at the end of each run. The best-looking intermediate point was not selected.

The results were very close

The candidate that selected the better of two proposals and the strong comparison method produced these results:

Method	Final held-out accuracy
Evaluate two proposals and select the better one	91.11%
Select a single coordinate proposal uniformly	91.02%

The difference was approximately 0.09 percentage points.

The target set in advance was 3 percentage points. The upper bound of the uncertainty interval for this comparison also remained below the target. The experiment’s primary quality hypothesis was therefore rejected.

The scope of this decision is specific: the mechanism was not shown to deliver a gain of the expected size under this task and budget. This single result did not close every research question in ADL.

Even so, the decision for the tested candidate was clear. There was insufficient evidence to advance it as a successful quality improvement.

Selection looked useful; the accounting needed to be completed

The experiment included an additional control. It also evaluated two proposals, but used the first rather than selecting according to their results.

The candidate selecting the better proposal finished approximately 0.97 percentage points above this control. This was an exploratory finding about the contribution of selection. It did not meet the success condition set in advance against the strong comparison method.

The cost findings also pointed in two directions.

The candidate used approximately 36% fewer model forward passes than the strong comparison method. However, the time spent in the Controller, which prepares, evaluates, and manages interventions, increased by approximately 10.5%. A saving in total measured time was not confirmed.

Counting fewer model calls was not a sufficient calculation on its own. The time used by the selection mechanism also had to be accounted for within the same budget.

Energy consumption and the end-to-end cost of the entire workflow were not measured in this experiment. The result therefore cannot support savings claims in those areas.

The same records became a new question

After the result, the existing experiment records were examined again. No new learning run was performed for this examination.

The question was narrower this time: how much progress did the selection mechanism deliver per intervention? How did that progress look when divided by the time it consumed?

In the exploratory analysis, selecting the better of two proposals increased training progress per round by approximately 15% relative to the control that used the first proposal. Yet training progress per Controller second was approximately 12% lower than with the strong comparison method.

Together, these observations were informative. Selecting a better change within a round appeared possible; the time needed to prepare and evaluate that round could limit the advantage.

The analysis did not identify which sub-operation caused the difference in total Controller cost. There was therefore insufficient data to attribute the issue solely to copying, proposal generation, or ranking.

What should the next experiment change?

This experiment branch was closed. Rather than changing the candidate’s seeds, budget, or settings in search of a more favourable result, the decision was to preserve the records and test a different mechanism with a separate question.

The next candidate explores an acceptance criterion that adds a fixed penalty on weight magnitude, instead of evaluating a single proposal solely by training error. Its implementation and tests are complete; its contribution to quality has not yet been tested.

Here too, there are two separate thresholds: the mechanism working as specified, and delivering the expected contribution to learning. Passing the first provides no result about the second.

The most useful thing left by this ADL experiment was an expectation turning into a clearer question:

Is the computation we spend choosing an intervention worth the progress that intervention delivers?

That is where the next experiment will look for an answer.

---

Source note — 15 October 2026: This article draws on the F2-041 experiment report, the F2-042 exploratory analysis of existing records, and the F2-043 implementation report. Results are limited to a small synthetic CPU task. This working note describes the stated source snapshot; it is not a live experiment status report.

ADL research page