Fifteen Gated Iterations

Where this sits

This began as a three-hour timed technical assessment and did not stop there. After submitting, I kept going for another two weeks across two repositories: one for the matching algorithm, one for catalogue profiling and the human review interface. Fifteen iterations in total, each with a written plan and a pre-declared pass/fail gate before any code ran, and a report with the measured result afterwards — including the four iterations that failed their gates and were not promoted.

It is the densest piece of applied retrieval and evaluation work I have done, and the part I would defend hardest is not the accuracy number. It is the evaluation discipline, including the two occasions when that discipline produced a worse number about my own work and I published it anyway.

The problem

A manufacturer receives free-text requirement lines from German construction and industrial tenders and must find the matching product in its own catalogue. Input is a short requirement plus a longer detail paragraph; output is one product ID from a 2,180-product catalogue. Training signal is a labelled history of which product was historically chosen for which requirement.

Three things make it hard, and only the first is obvious.

The German is messy. Umlauts inconsistently encoded, compound words sometimes spaced and sometimes joined, article numbers in several formats, dimensions written five different ways.

The catalogue disagrees with itself. Profiling it found 12 categories, 289 cases where a product's name asserts a type its description contradicts, dimension-like strings on 20.5% of rows with explicit units on fewer, and 116 distinct normalised description keys. There is no clean attribute table to match against; building one is its own project.

A large class of requirements names no product at all. Positions that say only "wie vor" (as before) or "oder gleichwertig" (or equivalent) carry no product information in the row itself. They are resolvable only from tender context that was never in the dataset. This class is roughly a tenth of the rows and it caps what any method can reach.

The method ladder, and what each step cost

I started fully offline, deliberately. Going to an API first would have meant never finding out how much of the problem was solvable without one, and the fallback path matters in a product that must answer when a key is missing.

Iteration 1 — lexical baseline. Word and character TF-IDF over the catalogue, a history index over labelled past requirements, an explicit article-number boost. Reported 45.5% Top-1 on a random 200-row split.

Then I found the leak. Exact requirement text appeared on both sides of that random split — the same request string present in training and test. The split was not measuring generalisation. I rebuilt it as a grouped split on near-duplicate text and reran: 40.6% Top-1. That is the number in the repository. The first lesson of the project was that my own first result was five points of self-deception.

Iteration 2 — the honest baseline. Grouped split, versioned normalisation for umlauts and compound words, type-safe article-number resolution (a resolved SKU is only boosted when the product type does not contradict the request), and structured evidence scoring on type, series and dimensions. 45.3% Top-1 / 70.3% Top-5 on its grouped test. This became the public offline default and stayed the default to the end.

Iteration 3 — cross-fitted logistic reranker. Recall-preserving candidate pool at 89.0% out-of-fold recall, then a learned reranker over it. Top-1 moved 44.92% → 45.81% on 788 development rows. The paired Top-1 and SKU guard gates, declared before the run, both failed. Not promoted.

Iteration 4 — pinned multilingual cross-encoder. A local neural reranker blended with the lexical scores, guarded so it could not override a safe SKU. 47.34% / 68.78% on development rows. Quality gates passed, but it cost a model download and runtime, and Top-5 went down. Kept as an optional blend, not promoted to the default.

Iteration 5 — feedback loops and a memory layer. Covered below; the most interesting result in the project.

Iteration 6 — one LLM call, in the right place. This is the version that worked. The offline ranker builds its top-30 shortlist exactly as before. A single call to a cheap hosted model then chooses the first suggestion from that shortlist only. Ranks 2–5 stay the offline ranker's, and rows with a safe, type-compatible article number bypass the model entirely.

Against a reference pipeline measured on its own held-out rows (51.2% / 67.2%), this is +5.6 pp Top-1 using one model call per row instead of roughly a hundred. Total API spend across the whole evaluation: $1.54, about $0.0012 per call, 1.3 s median latency.

The design decision that made it cheap is the one worth keeping: the model is not the ranker. It is a chooser, over a shortlist the offline system already produced, on the rows where the offline system is uncertain.

Iterations 7–10 tested whether more helped. Mostly it did not. A better offline shortlist was confirmed out of sample (+2.6 pp, p = 0.002). A preset library missed its pre-declared recall gate by two rows and was rejected. Running the LLM over 50 candidates instead of 30 moved Top-1 by +0.51 pp [−1.17, +2.14] — not significant, and not promoted. A locally fine-tuned cross-encoder learned the task standalone (17.1% → 35.3% Top-1) and still added nothing inside the blend (+0.38 pp [−0.49, +1.36]); its gates failed, so the LLM stage was never run and the frozen test was never touched. That iteration cost $0 by design.

The memory layer I built and then rejected

Iteration 5 asked whether the system should learn from what reviewers actually pick. Two separate questions, two separate answers.

Learning from decisions: yes. Appending trusted reviewer decisions to the history index improved Top-5 by +1.8 pp [+0.9, +2.7] and Top-1 by +1.4 pp [+0.5, +2.3] over a static index, replayed prequentially — rank a batch with the memory as it stood before that batch, then learn from it. Restricting learning to carefully-compared picks added under 0.6 pp even when 30% of quick accepts were wrong, so the recommended policy is to learn from every final pick and audit a random sample.

A memory layer in front of the ranker: no. It placed remembered products — exact and near-duplicate text matches from past decisions — at the top of the list before the ranker filled the rest. It improved the shortlist around both a history-based ranker (+1.2 pp Top-5) and a history-free one (+4.2 pp Top-5). And it lowered Top-1 by about 2 pp on both, with both lower bounds past the −1 pp non-inferiority limit declared before the run. Rejected.

The diagnosis is the part that transfers. The cause was placement, not learning. The loss was already present with a seed-only memory that had learned nothing, and it concentrated exactly where the un-remembered answer was already right: rows with a resolvable article number lost 7.4 pp Top-1, multi-label positions 8.3 pp. Memory was displacing correct rank-1 answers with plausible remembered ones.

I have since found that this generalises to almost every memory or personalisation feature I look at: the question is never "should the system remember" but "is remembered context allowed to outrank the current request". The answer is usually no.

Evaluation discipline

This is the part I would want a reviewer to check first.

Three separate evaluation sets, used for different things. Out-of-fold development rows for iteration. A companion held-out set for confirmation. A frozen test set of 212 rows, scored exactly once — the harness refuses a rescore, and most iterations never touched it at all.

Gates declared before the code. Every iteration has a plan file written first, naming the metric, the threshold and the decision rule. Four iterations failed those gates and were not promoted. Writing the gate before knowing the result is what makes "not promoted" mean anything.

Intervals on every headline number. Merged-cluster bootstrap, 10,000 resamples, paired where the comparison is paired. This is why the project can conclude "not significant" and act on it, instead of reading a +0.5 pp move as progress.

Stratified error analysis, not a single accuracy number. The remaining headroom is not evenly distributed:

That last row is the split that matters: when the right answer is in the shortlist, the chooser picks it 67% of the time. So roughly a third of the remaining loss is a ranking problem and the rest is recall. The generic stratum at 24.8% is the "wie vor" class — the rows that need tender context the dataset never contained.

The second integrity event. In the companion repository, an early exploratory pass had summarised all 1,000 request rows before I had fixed the evaluation boundary. That means the 900 rows I had intended to hold back are not a pristine blind test set. No model was trained on them, nothing forced me to say so, and the report says so anyway, with a recommendation to use fresh unseen requests for the cleanest evaluation.

I treat both events as the main evidence that the rest of the numbers can be trusted. Evaluation discipline that has never cost you anything has not been tested.

The human half

Accuracy at 58.5% is not a system anyone should run unattended, and saying so is part of the design. The last iterations were about making a reviewer's day better rather than making the number bigger.

Calibrated routing. A model that predicts whether the first suggestion is right, calibrated rather than merely scored: about 21% of suggestions are marked auto-confirm at 90–93% measured precision (90.5% out-of-fold, 93.2% on held-out rows), with expected calibration error 0.03. The rest go to review. That is the project's first output a product decision could actually be built on.

The reviewer sees a band, never a score. "Quick confirm" or "Review", not 0.87. A raw similarity number invites a reviewer to negotiate with a system that cannot negotiate, and it is meaningless unless calibrated — which is exactly the thing most such numbers are not.

The interface is designed against its own biases. Historical labels are hidden by default; revealing one is itself recorded as a bias event. A side-by-side comparison counts as a comparison only when the reviewer actually opens the dialog. Decision records are append-only and hold IDs, hashes and numbers — never raw request text, never a reviewer profile.

Graceful degradation is the default, not the fallback. The offline ranker is the public predictor. The LLM path is opt-in and falls back on a missing key or any failure. A system that stops answering when a vendor has an outage is not finished.

What I would do next

1. Get the tender context. The generic stratum cannot be solved with better ranking. An oracle ablation — supplying the correct product line for rows that name no product, which I took from the label, so this is an upper bound and not a result — moved Top-1 from 56.2% to 61.6% and Top-5 from 71.5% to 80.5% with no model involved on those rows. That bounds what one piece of context the system never had would be worth, and it is a product question, not a modelling one.
2. Re-run the error taxonomy as a first-class artefact, not as a table inside an iteration report. The stratified view above is the most useful thing in the repository and the easiest to overlook.
3. Measure real reviewer behaviour. Every rate in the feedback simulation is Assumed. The loop is recommended on simulated streams and should not ship on that basis alone.

Reflection

The strongest version of this work is not the +13.2 points. It is that the repository contains four ideas I built, measured and refused to ship, two corrections that made my own results look worse, and a frozen test set I declined to rescore when rescoring would have been easy and unobservable.

The habit I did not have at the start, and have now, is to look at which cases fail before reaching for the next method. Trying a better method is the comfortable move when a number plateaus. Reading the failures is the one that tells you whether a better method can help at all.