A Create Order request begins to fail. A vector-backed RAG system returns a detailed guide in first place. The guide is excellent, except for one detail: it still tells the caller to use a field the API no longer accepts.

This is the kind of moment when it helps to slow down. Similarity found something related to the question. It did not establish that the material is the right answer now. I use this deliberately small demo to separate the stages: where the old document enters the result, what a reranker changes, and what the application still has to decide.

Four documents are enough to make the problem visible

The demo has four documents: a v1.0 integration guide, a v2.0 migration notice, and two topical distractions about cancelling orders and paginating lists.

  • The v1.0 integration guide is long. It describes account_id, and its language resembles the error. It becomes two text blocks during ingestion.
  • The v2.0 migration notice is short. It says the current field is buyer_account_id. The old guide was once correct; it has been superseded.

These are demo materials, not a postmortem for a customer incident. The four documents become five source blocks. That is far too small to call a benchmark, but it makes the path inspectable. We can look at each stage instead of guessing whether the issue came from the corpus, recall, or ranking.

The vector baseline uses PostgreSQL and pgvector. The documents and the question are embedded with BGE-M3 and ordered by exact cosine distance. The table stores the version and original text, but this first query orders only by vector distance. No document lifecycle rule participates in the ranking.

The result is unsurprising. The first block of the old guide scores 0.7532; the migration notice scores 0.5937. The old guide resembles the wording of the error more closely, so it comes first.

Relevance scores and lifecycle status for two documents in the demo
Figure 1. A slice of the vector result from this demo. These scores rank relevance; they are not probabilities that an answer is correct.

pgvector did the job the query gave it. The question we actually need to answer is, “What should this request use now?” This query never defined what “now” means. The baseline is a retrieval foundation, not a complete RAG system with answer generation and business validation. It would be a mistake to use it as evidence that RAG as a whole does not work.

The straight-line fix: add a current field

The obvious reaction is that the demo is making this harder than it needs to be. Add a field, filter on status = 'current', and move on.

That can be the right answer. If there is one application, four documents, and an unambiguous definition of the current version, it is probably the first answer to try. There is no reason to replace a data layer before the small problem requires it.

But then someone has to maintain that field everywhere it matters. When a version is released, who marks the old guide as superseded? What happens if an interface rolls out by region? If an old SDK is still in its support window, is the old guide still usable? When someone investigates a historical incident, should the old document return at all?

Every one of those questions can be handled in code. The harder question is whether the same decision gets reimplemented in every application. A support agent writes it once, a developer agent writes it again, and an operations tool writes it a third time. After a while, current can mean three different things.

That is why I care about what can be maintained together and what must remain a business decision. It is also the point of the second path I call semantic storage. In our engineering work, that is the direction we explore in Cortrix: keeping semantic processing and evidence connected.

Move semantic work into ingestion

The second path begins with the same four documents, but it does not store only vectors for the raw text. Each of the five source blocks keeps its identity. We also create other retrieval views: HyPE question blocks capture ways a user might ask about a passage, and document summaries express the material at a different level of granularity.

Those views still have to lead back to the source. Generating three questions around one piece of text does not create three independent pieces of evidence. In this scenario, document summaries remain ingestion evidence; they are not query candidates.

At query time, the original question produces three rewrites. The original and those rewrites make four expressions. They go through five retrieval paths: dense vectors, context enrichment, FTS5/BM25, HyPE questions, and sparse retrieval. Reciprocal Rank Fusion then combines the candidate ranks.

Multiple representations at ingestion and five retrieval paths at query time
Figure 2. Candidate generation in this demo. The record contains 20 path hits, which become four unique blocks from three documents after fusion.

This work mostly creates more chances to recover useful material. A question may be worded differently, or a short migration notice may express the same idea more compactly. More than one route can bring it back.

That still does not mean the resulting order matches the current business need.

In the semantic-storage path, BGE reranking still places old guide block 0 first at 0.9467. Old guide block 1 is second at 0.6609. The v2 migration notice we want is third at 0.6197.

So attaching a reranker after vector search does not automatically assume responsibility for document lifecycle. A reranker can judge how closely a question and a passage match. We should not turn that score into proof that the passage is safe to use today. And we should not compare 0.9467 with the earlier cosine score of 0.7532 and conclude that the system became more correct. Those scores come from different systems.

Who changes the order, and who adopts it?

The LLM then orders the candidate set as a whole. In this instance, it moves the migration notice to first place. The demo application walks the final order and takes the highest-ranked document whose lifecycle state is current.

BGE ranking, final LLM order, and demo application adoption check
Figure 3. The ranking and adoption record from this demo. The LLM changes the order; the application checks lifecycle and keeps the migration notice already in first place. The adopted field is buyer_account_id.
for item in final_order:
    if item.status == "current":
        return item
return None

Those lines do not settle the business question by themselves. The cancellation notice is also current, but it cannot answer which field a Create Order request should use. If the migration notice is missing, we still cannot tell whether the request comes from an old SDK in its support window. Checking only for current is not enough to establish the needed version context.

The value of writing the stages down is that we can name their separate responsibilities: what order did the model produce, and what condition did the application use to adopt a result? We can examine that causal chain without pretending that a correct field at the end proves every step was sound.

Keep the evidence for the questions you will need later

The demo does end with the new field. That is not where the conclusion should stop.

Because an LLM helps shape the result, the next failure needs a path for investigation. Did the migration notice enter the system? Did recall return it? Where did BGE rank it? Did the LLM change its position? What rule did the application actually apply? Each question points to a different repair.

If the notice never entered the corpus, a new reranker will not help. If it was already a candidate, the issue may be ranking or adoption conditions. If the source lifecycle state is wrong, a careful model still has to operate from the wrong premise.

I call this a causal chain. That phrase is about observable system stages, not a claim that we can expose a model's private reasoning. This demo does not preserve every internal detail. When we did not keep per-path ranks or the complete derived representations, we cannot honestly reconstruct them later as though they had been recorded. The practical lesson is simple: if you want to diagnose a layer, retain the evidence that layer requires while the system is running.

Someone asked about latency

A practical question is what one retrieval takes once the LLM is involved. This demo is not a load-test result or a performance promise for the current version.

The next question was more practical. An agent may need to consult a knowledge base many times while handling one task. If each call follows a long chain, how long does the whole task wait?

We should not keep adding stages to retrieval while showing only the one time the final answer was right.

The stages can be used selectively. If version and scope are already clear, filtering before retrieval may be enough. If an agent is looking up one exact identifier, there is no reason to ask an LLM to produce four query expressions first. If semantic-storage capabilities are offered to an agent through optional MCP or SDK interfaces, the agent should be able to choose the tool that matches the task.

When is additional waiting and inference worth it? I would evaluate that inside a concrete agent task: given the same materials and request, a simple filter may be faster; what extra result does reranking and an LLM actually bring? This four-document demo does not isolate the contribution of those components. It cannot stand in for that comparison.

What belongs in a shared foundation?

Michael Dubakov, in “Malleable software = solid bases + custom code”, makes a useful case for stable foundations that custom code can build upon. That leads back to the four documents.

An interface can be rewritten. An agent can be replaced. But the identity of a document, the version that applies to a request, the original source for a passage, and the processing evidence behind an answer should not need to be reinterpreted from scratch by every application.

That does not mean every business rule belongs in one platform. Orders, customers, permissions, transactions, and constraints still have their own authoritative systems. A retrieval system can organize material, retain source identity and evidence, and pass candidates downstream. The business still needs explicit rules for what it is allowed to adopt.

That is the approach we want to keep testing in Cortrix: make recurring semantic processing and evidence maintenance reusable, while leaving adoption conditions with the application that owns them. In this example, would you start with a version filter or keep the old documents in retrieval? When regions, SDK support windows, and historical lookups differ, where would you put the applicability rule?

The version-pinned synthetic scenario is public on GitHub. Explore Cortrix on GitHub. The more useful conversation is how to make a path easier to inspect, not how to add one more model to it.