Most talk about LLMs in RAG is about the query path: rewriting the question, ranking the results, generating the answer. Cortrix moves part of that work into ingestion and runs the LLM when a document is written. We call this write-time LLM inference. In this article we walk through what Cortrix has the LLM do at write time, how much that changes retrieval, and which kinds of corpora are worth enabling this feature for.

Before a passage reaches semantic storage

Take a customer-support knowledge base as an example. Somewhere in the refund policy is this sentence: “Requests filed after the period above are handled under Section 4.” A typical RAG pipeline cuts that sentence into its own chunk and stores it. Then a user asks, “Can I still return something I bought forty days ago?” The chunk doesn’t contain “forty days”, doesn’t contain “return”, and doesn’t say how long “the period above” is. As a result, the chunk sits at least two steps away from the question: the two are worded differently, and the context that would connect them has been lost.

An embedding model closes some of that distance, but it only sees the chunk itself. An LLM that has read the whole document can fill in much more if it gets to process the chunk before it is stored. Cortrix’s write path is called SPC, short for Semantic Processing Chain. Once an LLM is configured, SPC does a few extra things between chunking and indexing:

  • It adds a short piece of context to each chunk, saying what the chunk is about within the whole document, and then generates a second vector. The original vector without the context is kept, so both exist side by side.
  • It generates a few questions a user might ask about each chunk and indexes those questions too.
  • It extracts entities and generates a summary of the content.
  • It generates a summary of the whole document and stores it as a chunk that can be retrieved.

All of this generated content is stored in semantic storage alongside the original text. At query time three kinds of it each become a recall path: the contextualized vectors and the pre-generated questions are fused by rank at the chunk level with the existing vector search and full-text search, and the document summary takes part in fusion at the document level. The extracted entities and content summaries are not recall paths; they are used to score results and are returned along with them. Because all of this content was already generated at write time, a query does not need another LLM call to produce it.

The diagram below shows write-time LLM inference in Cortrix: the extra work done when a document is written, and the extra recall paths a query gets as a result.

Write-time LLM inference in Cortrix: at write time the LLM adds context, pre-generates questions, extracts entities and summaries, and generates a document summary; at query time the contextualized vectors, the pre-generated questions, and the document summary each become a recall path, fused with vector and full-text search, with no LLM call at query time

We are not the first to run an LLM at write time. Anthropic proposed the Contextual Retrieval method for knowledge-base retrieval, which adds context to each chunk before it is stored.

Screenshot of the Anthropic post: Contextual Retrieval prepends chunk-specific explanatory context to each chunk before generating embeddings and building the BM25 index

Microsoft’s Azure AI Search can call a large language model at indexing time to generate new content such as summaries. With Amazon Bedrock Knowledge Bases you can attach a custom function at ingestion to do the same thing.

Screenshot of the Microsoft Learn page: the GenAI Prompt skill in Azure AI Search calls a large language model to generate new content that can be indexed and retrieved

Screenshot of the AWS blog post: Amazon Bedrock Knowledge Bases uses a custom Lambda function at ingestion to generate contextual information for each chunk

What these offer is a method or an extension point you can attach to, so the user still has to implement and orchestrate how the content is generated and how it is used for retrieval. Cortrix takes its own approach: it generates several kinds of content together, gives three of them their own recall path, and always keeps the original vector. The storage write path does all of it, so the agents and applications on top don’t have to orchestrate anything.

What we measured

Here is nDCG@10 on two corpora from our public evidence pack. The numbers in parentheses are the change relative to the first column.

Corpus Dense + full-text + Rerank + Write-time LLM inference Full stack
SciFact 0.5942 0.6184 (+4.1%) 0.6449 (+8.5%) 0.7784 (+31.0%)
NFCorpus 0.2991 0.3121 (+4.3%) 0.3124 (+4.4%) 0.3772 (+26.1%)

“Full stack” is write-time LLM inference plus an LLM ranking the top 30 candidates as a whole at query time. Both steps use the same model, deepseek-v4-flash. Every number was measured on the same hardware configuration with top_k=10. The measurements were taken in August 2026 against Cortrix core commit 79a4eb17, which predates v1.0.0-rc.2.

The full stack shows the clearest effect: it is about 30% above the baseline on SciFact and about a quarter above it on NFCorpus.

If we look only at the write-time LLM inference step, it adds another 0.0265 on SciFact on top of reranking, and it is essentially flat on NFCorpus.

Query-time reranking and write-time LLM inference solve two different problems. Reranking reorders candidates that have already been retrieved, and it has nothing to work with for content that was never retrieved. Write-time LLM inference builds extra recall paths when the document is written, which gives chunks that used to be missed a chance to become candidates. That is why the two can be used together, and using both is the best configuration we have measured so far.

Which corpora write-time LLM inference suits

Our advice is to look first at the shape of your documents and the shape of your questions.

If the chunks in your documents cannot be understood once they are taken out of context, then adding context at write time will help. Contract clauses, policies, and technical specifications are like this, and so are long documents full of references such as “the above” and “said item”.

If the way your users ask questions is far from the way your documents are written, then generating questions in advance will help. SciFact has this shape: you take a scientific claim and look for paper abstracts that support or refute it, and the two sides rarely share wording.

If your documents are written once and queried many times, this step is also more valuable, because the processing happens only once at write time and every query after that benefits from it.

On the other hand, if the corpus is already question-answer pairs, or every chunk stands on its own, then there is little for write-time processing to add, and the value of enabling it is low.

You don’t have to wait for an aggregate score to find out which kind of corpus you have. If you use Cortrix, the ?explain=true query parameter shows you directly what each recall path contributed. Take a representative batch of real questions from your application and run them one at a time, like this:

curl -fsS -H 'Content-Type: application/json' \
  -d '{"namespaces": ["your_namespace"], "query": "Can I still return something I bought forty days ago?", "top_k": 10, "route": "complex"}' \
  'http://127.0.0.1:8420/api/v1/query?explain=true'

The response includes an explain section. Its rrf_path_counts field counts which paths ranked the returned results in. It counts five chunk-level paths: dense, sparse, and fts5 are the existing vector and full-text paths, contextualized is the vector with context, and hype_question is the pre-generated questions. It counts only the results that were returned, and it does not cover the document-summary path. The route parameter in the request is covered in the next section.

This is the explain we got from running that command against one sample document on v1.0.0-rc.2, trimmed to the relevant fields and otherwise shown as it was returned:

{
  "complexity_score": 1.0,
  "query": "Can I still return something I bought forty days ago?",
  "routing_decision_source": "force_route",
  "routing_path": "complex",
  "rrf_path_counts": { "contextualized": 1, "dense": 1, "fts5": 1, "hype_question": 1, "sparse": 1 },
  "via_path_counts": { "hybrid": 1 }
}

The one result returned was ranked in by five paths at once: the three existing paths, dense, sparse, and fts5, plus contextualized and hype_question, both added at write time. When we ran the same question again without the route parameter, the query was judged simple and rrf_path_counts contained only the three existing paths.

If the added paths show up often, it means the content generated by write-time LLM inference is doing real work for your questions. If they rarely show up, you can leave the feature off for that kind of corpus for now.

If your application needs higher retrieval quality, add query-time LLM reranking on top of write-time LLM inference. That combination is the Cortrix full stack. Query-time reranking needs one more role to be configured, semantic_llm.

How to enable it in Cortrix

The Quickstart does not enable any LLM by default, and retrieval runs end to end using only the local embedding and rerank models. The LLMs used for write-time LLM inference are configured by role: enricher_llm handles enrichment and doc_summary_llm handles document summaries.

enricher_llm:
  provider: "openai"
  api_key: "your-api-key"
  model: "your-model"
  base_url: "https://your-endpoint/v1"

doc_summary_llm:
  provider: "openai"
  api_key: "your-api-key"
  model: "your-model"
  base_url: "https://your-endpoint/v1"

A role counts as configured once provider, api_key, and model are all filled in, and any role you leave unfilled stays off. The server speaks the OpenAI-compatible protocol, so pointing it at a local model gateway works too.

When Cortrix receives a query, it first judges the query’s complexity and sorts it into one of three tiers: simple, complex, or chat. The three tiers do not use the same write-time content. The complex tier brings the contextualized vectors and the pre-generated questions into the fusion. The simple tier uses only the original chunk vectors and the full-text path, so those two added paths take no part; the document summary, however, takes part according to retrieval granularity, and at the default granularity it is fused in the simple tier as well. The chat tier does not search chunks at all, because it takes a memory-only path. The routing_path field in the response’s explain tells you which tier a query actually took. If you want a query to always take the complete path, you can add "route": "complex" to the request, which is what we did for every column in the measurements above except the baseline in the first column. For the full request parameters, check the profiles in the public runner. Before you enable any of this, we recommend reading the v1.0.0-rc.2 release notes.

Use Cortrix, or write the glue code yourself

If you don’t use Cortrix as the semantic storage under your agents, you are still welcome to read the Cortrix source, which is open under Apache-2.0. To implement write-time LLM inference inside the data architecture you already have, these are the four things you must get right:

  • Keep the contextualized vector and the original vector together. Both paths take part in retrieval, so when the new one performs poorly the original is still there as a fallback.
  • Keep generated content tied to its source text. When the original is updated or deleted, the added context, the pre-generated questions, and the summaries all have to be regenerated or cleaned up, or retrieval will hit content that no longer exists.
  • Don’t let a failed LLM call block writes. The document should be stored first as if there were no LLM, with the LLM processing filled in afterward.
  • Make each recall path’s contribution visible across the whole write-time LLM inference process. A total score can’t tell you which path is doing the work, and the agents on top can’t make their next business decision without that.

Each of the four takes glue code, and that glue code then becomes a separate module you maintain for the long run. If you would rather not hand-write it, or add to your own maintenance load later, just use Cortrix. Keep your attention on your application and leave the complicated data management to us. These of the four are already in Cortrix: the two vectors are kept side by side, enrichment has a budget cap and a fallback, the document summary falls back to text matching when the LLM is unavailable, and every query can show each path’s contribution through explain.

Frequently asked questions

What is write-time LLM inference?

Write-time LLM inference is a feature of Cortrix semantic storage. When a document is written, an LLM adds context to each chunk, generates questions a user might ask about it, extracts entities and summaries, and writes a summary of the whole document. At query time, the contextualized vectors, the pre-generated questions, and the document summary each become their own recall path.

Once write-time LLM inference is enabled, does a query still need to call an LLM?

No. The content is generated at write time and kept in semantic storage, so a query only does recall and fusion. If you want to push retrieval quality further, you can add query-time LLM reranking on top (it needs semantic_llm to be configured), which is the Cortrix full-stack configuration.

Which corpora are a good fit for write-time LLM inference?

Documents whose chunks cannot be understood out of context, such as contract clauses, policies, and technical specifications, and corpora where the way users ask is far from the way the documents are written. If the corpus is already question-answer pairs, or every chunk stands on its own, the value of enabling it is low.

References

These measurements cover retrieval quality at top_k=10 on two corpora; they show direction and are not a comparison between Cortrix and other products. The behavior described in this article corresponds to Cortrix v1.0.0-rc.2, which is a pre-release version; the numbers in the table were measured in August 2026 against core commit 79a4eb17.