Building Something That Holds Up

The requirement that shapes everything is that outputs must be traceable to a source span. Not because it is good practice, but because a number nobody can check is a number nobody can act on, and in a regulated firm a decision nobody can explain is a finding.

The practical form is that an extraction pipeline returns evidence alongside every value.

{
  "metric": "fy_revenue_guidance_low",
  "value": 2400000000,
  "currency": "USD",
  "source_doc": "8-K_2026-02-11.pdf",
  "source_span": [14822, 14901],
  "quote": "we now expect full year revenue of $2.4 to $2.6 billion",
  "model": "provider/model@2026-01",
  "prompt_hash": "9f2c..."
}

Every field is doing a job. The span and quote let a human verify in seconds. The document identifier makes the claim reproducible. The model version and prompt hash make it reproducible against the same artefact, which the look-ahead lesson showed is not automatic.

A pipeline that returns only the number is not a weaker version of this. It is a different thing: an unverifiable assertion, which is unusable for anything with money attached.

The design rule is that the model's job is to locate and normalise, never to state a fact of its own. If a value cannot be pointed at, it is not extracted, and the correct output is an abstention.

The great advantage of extraction over prediction is that it can be measured now, against a standard that is not the market.

Build a labelled set. A few hundred documents with correct extractions, produced by people who know the domain, is enough to be informative and is a week of work rather than a year of paper trading.

Then measure the things that matter separately, because they fail differently.

Precision. Of the values extracted, how many are right? This is the number that governs whether downstream users can trust the output without checking.

Recall. Of the values present in the documents, how many were found? Silent omission is the more dangerous failure, because nothing signals it.

Abstention quality. When the model declined to answer, was the value genuinely absent? A model that abstains on hard cases is far more useful than one that guesses, and this is the metric that captures it.

Consistency. Run the same document repeatedly. Variation is a direct measure of the non-determinism problem, and it bounds how much any single output can be relied upon.

None of these requires a single trade or a single day of market data, and none is contaminated by the look-ahead problem, because whether a guidance figure was correctly pulled from a filing does not depend on what happened next.

Two properties of this shape matter more than any individual component.

Validation is deterministic and sits outside the model. A revenue figure must be a number in a plausible range; a quarterly breakdown must sum to the annual total; a date must fall inside the reporting period. These are ordinary rules, they catch a large fraction of extraction errors, and they cost nothing.

That point generalises. The most effective guard on a probabilistic component is usually a deterministic check downstream, because it fails loudly, reproducibly, and without a model's involvement.

The uncertain path goes to a person, not to a default. The alternative designs are worse in ways that are easy to miss: dropping uncertain values silently creates gaps nobody notices, and passing them through unmarked pollutes the store.

A review queue also does something valuable over time. It is a stream of exactly the cases the system finds hard, which is the highest-value labelled data available, and feeding it back into the evaluation set makes the measurements sharpen where it matters.

That is the same closed loop the execution lesson argued for: measurement is only worth its cost if it changes the next iteration.

Inference cost is not a footnote here, because the volumes are large and the cost model interacts with the value.

Work it through. A universe of several thousand companies, each producing quarterly filings, earnings calls, and a steady stream of news. Long documents, so many tokens each. Processing the universe every quarter is a substantial recurring bill, and re-processing history to build a research dataset is a much larger one-off.

That has three architectural consequences.

Route by difficulty. Most documents are routine and a smaller model handles them; escalate only the ambiguous ones. The lesson LLM Pricing and Latency: What Actually Drives Cost develops this properly.

Cache aggressively. Documents are immutable once published, so an extraction is a pure function of document, prompt and model version. Anything already computed for that triple never needs recomputing, which makes the content-addressing idea from the Git and container cursus directly applicable.

Extract once, query many times. The expensive step is converting a document into structured data. Do it once, store the result with provenance, and answer subsequent questions from the store rather than by re-reading.

That last one is the difference between a system that scales and one whose bill grows with usage rather than with the corpus.

This domain has a threat property most do not: the documents are written by interested parties, and some of those parties know their filings are read by machines.

That makes prompt injection a first-class concern rather than a theoretical one. Text embedded in a filing, a press release or a web page can attempt to steer a model reading it, and the input arrives through the normal ingestion path because it is a legitimate document.

The defences are structural rather than clever.

Never let document content be instructions. Keep the system prompt and the document strictly separated, and treat everything from the document as data to be analysed rather than as text to be followed.

Constrain the output. A model returning a value from a fixed schema, validated on the way out, has a narrow channel through which to do damage. A model returning free text has a wide one.

Validate against the source. A quoted span must actually appear in the document. This single check defeats a large class of both injection and fabrication, and it is cheap.

Cross-check independent sources. A figure appearing in a filing and a data vendor's feed should agree, and disagreement is worth a human's attention regardless of cause.

The general lesson from LLM Guardrails and Red-Teaming applies with the volume turned up: the input is untrusted by construction, and here the untrusted party has a financial interest in the output.

Once a model influences a regulated activity, model risk management applies, and the questions are the same ones asked of any model the firm relies on.

What does it do and where is that written down? An inventory entry with a stated purpose and scope.

How do you know it works? Evidence, which is why the evaluation set in the second step is not optional. "The vendor's benchmark" is not an answer about your documents.

Who validated it, independently of the people who built it? Separation of development and validation is standard and it applies here.

How would you know if it stopped working? Ongoing monitoring, which for this kind of system means tracking abstention rates, validation failure rates and review-queue volume, because those move before accuracy visibly degrades.

What happens when it is wrong? A defined escalation path and a demonstrated ability to fall back.

Can you reproduce a past decision? This is the one that catches teams. It requires the model version, the prompt, the document and the output all retained together, which has to be designed in and cannot be added afterwards.

The lesson AI Guardrails for Regulated Industries covers the general framework. The specific consequence here is that auditability is a design input, and a system that produces good answers without provenance has failed a requirement that is not about quality.

Pulling the cursus together, the applications that survive all four lessons share a shape.

Coverage expansion. Producing consistent, comparable assessments across a universe far larger than analysts can read. The output is a dataset, and its value is breadth rather than depth.

Research throughput. Summarising filings, calls and research so a person reads the relevant ten pages rather than the full two hundred. The human still decides; the model changes what reaches them.

Structured extraction into a store. Turning documents into queryable data with provenance, which then feeds conventional quantitative work that was previously impossible for lack of the inputs.

Document-heavy compliance. Surveillance, obligation tracking, contract review. High volume, checkable answers, and a domain where recall matters more than elegance.

What these share is that the model produces an input to a decision rather than the decision, and that its output is checkable against something other than a return series.

That is the summary of the path. The look-ahead problem makes return-based validation structurally unreliable, so the defensible applications are the ones whose correctness is established some other way, and those turn out to be the ones the technology is best at anyway.

The framing worth keeping: an LLM in a trading firm is best understood as a very fast junior analyst who reads everything, never gets bored, must show their working, and whose conclusions you check. That is a genuinely valuable colleague and not an oracle, and building for the first produces systems that survive audit while building for the second produces demonstrations.

Every row is a consequence of something established earlier: contamination makes return-based evidence unreliable, non-determinism makes reproduction hard, adversarial authorship makes input untrusted, and regulation makes provenance mandatory.

The closing observation is that none of this is exotic engineering. It is provenance, validation, evaluation and monitoring, which is what any serious data pipeline has had for decades.

What the technology changed is the class of input that can enter such a pipeline. Text used to require a person; now it does not. The discipline around what comes out did not change and should not.

Systems that treat the model as one component inside conventional engineering tend to work. Systems built around the model as the product tend to demonstrate well and fail audit.