Keeping the code is not enough to obtain a result again. We also need to preserve the inputs, version and evaluation rules through which that result was reached.
A report may tell us that all 40 of its 40 questions matched expectations.
But when we return to the same work months later, that number alone will not be enough. The questions may have changed. The expected answers may have been rewritten. A new version of the engine may be answering the earlier questions by a different route.
There is a distance between seeing the same number again and obtaining the same result again.
In the previous essay, I discussed leaving readers a route to reconsider an experiment when publishing its result. ERM’s existing validation records provide a concrete example of how that route can be built.
I am not reporting the outcome of a new run here. I am opening the records dated 6 October 2026 to examine what was preserved for reproducibility.
First, identify which ERM
This essay concerns the ERM v0.12 relational memory prototype.
The prototype evaluates the relational structure of a particular event together with explicit declarations recorded about that same event. For example, relations may support a temporary-use structure while another record says that the same transfer was not return-based use.
In that situation, the system reports a conflict and shows both sets of support together.
This research line is separate from the RGM / Relation Space H1a retrieval experiments. Putting their documents or results into the same reproduction record would obscure which mechanism was tested.
The first record therefore comes before the command to run the code: which question, in which version, are we examining again?
Preserve the old questions and separate the new ones
When adding a feature, we want to learn two things.
Has earlier behavior been preserved? How does the new feature behave in the situations prepared for it?
It is easy to merge those questions into one result count. That makes it harder to follow where an improvement or regression came from.
ERM’s 6 October record preserves the earlier question and counterexample inputs unchanged. Scenarios for the new combined meaning query are kept in a separate file.
| Check group | Result reported in the record |
|---|---|
| The earlier 40 questions | 40/40 matches; earlier detailed answers and evidence preserved |
| The 36 earlier counterexample queries | 36/36 matches; earlier detailed answers and evidence preserved |
| The 25 staged judgments in 20 new scenarios | 25/25 matches with expected judgments |
These are different groups of checks. Adding them together into one overall success rate would make it harder to explain what was tested.
The new scenarios separately address opposite declarations in the same frame, records belonging to a different time or object, ambiguous transfer references and source changes.
Preserving the earlier test and testing the new behavior can therefore be read separately.
If the filename is the same, is the content the same?
Keeping a file under the same name does not establish that we preserved the same input.
A single relation, expected label or evaluation rule inside it may have changed. Even if the result looks the same, we may now be discussing a different test.
The protocol therefore locks the engine sources and four JSON inputs with SHA-256 file digests before the final run. A file change requires a new test version.
While preparing this essay, I compared the digests of the four existing input files with the lock record; they matched. This was only a file-integrity check. I did not rerun the engine or independently validate the result.
A file digest does not tell us that its content is correct. It helps distinguish which content was used.
That distinction matters for a reproduction record: first we need to know whether we are examining the same pieces.
The grounds behind the same label
Is it enough for a query to return conflicting again?
Perhaps the result label stayed the same, but the relations of another event were used. Perhaps a withdrawn source still counts as support. Perhaps the answer relies on a record absent from the current revision.
ERM’s protocol therefore checks more than the final label.
Evidence in a judgment must belong to the current revision and visible source contexts. A structural explanation must contain the transfer being queried, and its required premises must have live source support.
At each stage, the persistent state before and after the query is compared as well. Reading is expected not to modify memory. Obtaining the same detailed answer after saving and reloading the state is checked separately.
The reproduction question thus becomes broader:
Did we obtain the same answer, and did we obtain it on the same grounds?
This matters especially when sources are withdrawn or revisions change. A previously supported result should not be counted as successful again merely because the old answer persists after its support is gone.
Who wrote the expected answer?
When reading the matches in the record, we should ask another question: where does the expectation come from?
The developer wrote the expected judgments for the new scenarios. They are not calculated from the engine’s output. The protocol also checks that changing the expected label does not change the engine’s answer.
This is a control intended to prevent the evaluation label from directly determining the answer.
It does not, however, provide an independent human evaluation.
The relations and the meaning link are manually defined. The prototype does not read a natural-language account and extract the entire structure itself. Its current meaning bridge also rests on an explicitly defined, narrow relation.
The report therefore keeps two states separate: passing the synthetic technical protocol and human meaning validation.
The latter remains recorded as human_meaning_validated=false. The judgment and rationale fields in the human evaluation forms are left blank.
That blank space may look like an unfinished sentence in the report. It actually makes clear which evidence is still absent.
Leave a record we can return to
The 6 October report also states that 348 engine tests, 16 protocol tests and 127 technical checks passed. These checks cover different responsibilities; they do not establish a guarantee of correctness independent of the research scope.
What this record can provide is more specific: comparing earlier behavior under preserved inputs, examining the new feature’s answers in defined scenarios, and following the sources of those answers.
If the work is rerun in another environment, that run will need a record of its own. The version, environment, command, inputs, completion status and differences from the earlier result should be written down separately.
If a difference appears, simply closing it with “could not reproduce” will not be enough. We will first need to investigate whether it comes from the input, the environment, the implementation or the evaluation rule.
What I expect from a research record is that it makes this return possible.
When opening the report months later, we should be able to read more than how many checks passed. We should also see which pieces were used to seek an answer to which question. Running the same work again can then become a way of understanding it again.
Source note — 18 October 2026: This essay draws on the ERM relational memory v0.12 documents and the Pattern–Declaration Consistency v0.4 protocol, comparison, lock and artifact-audit records dated 6 October. The SHA-256 digests of four existing input files were compared with the lock record. No engine tests or reproduction run were executed for this essay, and no independent human evaluation was conducted.
Connections: ERM research page · Previous essay: How do we publish an unsuccessful experiment?
Atlas concepts: Memory · Relation · Context · History · Evidence · Meaning
This connects to
Examines the ERM v0.12 relational memory prototype’s 6 October records of preserved inputs, evidence, reloading and validation, separately from RGM/H1a.
Makes the question of leaving a route to reconsider an experimental result concrete through ERM’s existing reproducibility records.