Institutional-grade financial RAG

Module 4 left the institutional documentary sources (EDGAR, transcripts, regulatory texts) ingested and normalized. This module turns them into a comprehension system: a Retrieval-Augmented Generation pipeline (RAG — retrieval-augmented generation: document fragments are retrieved and injected as context into the LLM to ground its answer) at the evidence standard of a research or risk desk. The central thesis, backed with data in every section: in finance structure is the pipeline — generic chunking, retrieval, and evaluation destroy exactly the signal the analyst needs, and the difference between 19% and 83.7% accuracy is engineering, not the model. (API EvangelistAPI Evangelist)

5.1 Why tutorial RAG fails in finance

Tutorial RAG — load a PDF, RecursiveCharacterTextSplitter with 1,000 tokens, generic embeddings, top-k by cosine similarity — rests on three assumptions that the financial document systematically violates. First, that the text is flat prose: a 10-K is a structured artifact (Items, notes, tables, XBRL tags), and fixed-size chunks "flatten structured disclosures into arbitrary blocks of text", splitting risk factors apart and separating every figure from the section that gives it meaning. (🦜️🔗 LangChain) Second, that semantic similarity is enough: asking about year-end inventory can retrieve prose from the wrong period "because the embedding looked close enough" — flat RAG "confuses fiscal periods, misroutes balance-sheet facts, or cites narrative that does not support the answer's figure". (csdn.net) Third, that retrieving the right text implies answering correctly: the financial answer demands multi-step numerical reasoning over table and text, where the LLM alone is structurally weak. (getvocal.ai)

5.1.1 Benchmark evidence: what they measure and why they matter to the buy-side

The empirical reference is FinanceBench (Patronus AI, arXiv 2311.11944, Nov 2023): 10,231 expert-written questions on SEC filings from 40 U.S. companies, with an open 150-question subset and binary scoring with a ±2.5% tolerance on numeric answers. (API EvangelistAPI Evangelist) Its original results are the X-ray of naive RAG: GPT-4-Turbo closed-book gets 9% right; with a vector store shared across all documents, 19% — that is, it "answered incorrectly or refused to answer 81% of the questions"; with a vector store per document, 50%; with the full filing in context (long-context), 76%; and with the evidence pages provided (oracle), 85%. (API EvangelistAPI Evangelist) In 2026, an independent evaluation on the same benchmark (average of 10 runs with an LLM judge) places exhaustive agentic RAG — hybrid retrieval, rerank, structure navigation, and tools — at 83.7%, above full-context with 1M-token models (76.0%). (langchain.js) The 19% → 83.7% trajectory is the quantified demonstration that pipeline engineering moves the needle in document comprehension, the domain where the institutional ROI of LLMs is proven.

Source: original FinanceBench results (arXiv 2311.11944, 2023) (API EvangelistAPI Evangelist) and independent agentic-RAG evaluation (2026, average of 10 runs) (langchain.js); own elaboration, data as of July 2026.

The remaining benchmarks bound the problem from the numerical-reasoning side. FinQA (arXiv 2109.00122): 8,281 QA pairs annotated by eleven professionals on S&P 500 earnings reports, with heterogeneous evidence (table + text) and a numerical reasoning program; GPT-4 reaches 62.87 versus 91.16 for the human expert. (Model Agent Platform) ConvFinQA (arXiv 2210.03849) extends it to 3,892 multi-turn conversations with anaphora over figures: GPT-4 achieves 76.48 versus 89.44 human, and errors cascade — if one turn fails, "there is very little chance that subsequent turns will be correct". (🦜️🔗 LangChain) TAT-QA (ACL 2021): 16,552 hybrid tabular-textual questions (42% arithmetic); the paper's best model, TAGOP, reaches 58.0 F1 versus 90.8 F1 human. (博客园)

Table 5.1 — Financial QA and numerical-reasoning benchmarks: size, best documented result, and gap versus the human expert

5.2 Structure-aware chunking

Chunking is the number-one lever of financial RAG because it decides which units of meaning survive indexing. The operating rule: the document's structure marks the boundaries; size only subdivides within them. LangChain offers the double-split pattern — first structural (for example, MarkdownHeaderTextSplitter, which preserves header lineage as metadata) and then by size with RecursiveCharacterTextSplitter, inheriting the structural metadata in each sub-chunk. (stackademic.com)

5.2.1 10-Ks by Items, earnings calls by turns, regulatory texts with hierarchy

10-K by regulatory Items. The semantic unit of a 10-K is the Item (1 Business, 1A Risk Factors, 7 MD&A, 8 Financial Statements): an analyst reads by "selecting the filing and period, inspecting the table of contents, and navigating to Item 1A, 7, or 8", and retrieval must mimic that path. (🦜️🔗 LangChain) Each chunk carries a lineage section_path ("10-K > Item 1A > Regulatory Risk > EU Operations") that enables section-precise citation and filtering before searching. (🦜️🔗 LangChain)

import re
from langchain_core.documents import Document

ITEM_RE = re.compile(r"^\s*Item\s+(\d+[A-Z]?)\.?\s+(.+)$", re.IGNORECASE | re.MULTILINE)

def chunk_10k_by_items(text: str, ticker: str, fiscal_year: int) -> list[Document]:
    """Split a 10-K into chunks by regulatory Item, with lineage metadata."""
    matches = list(ITEM_RE.finditer(text))
    docs = []
    for i, m in enumerate(matches):
        start = m.end()
        end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
        docs.append(Document(
            page_content=text[start:end].strip(),
            metadata={
                "ticker": ticker,
                "fiscal_year": fiscal_year,
                "filing_type": "10-K",
                "item": f"Item {m.group(1).upper()}",
                "item_title": m.group(2).strip(),
                "section_path": f"10-K > Item {m.group(1).upper()} > {m.group(2).strip()}",
            },
        ))
    return docs
# Then: re-split long Items with RecursiveCharacterTextSplitter,
# PROPAGATING the metadata to every sub-chunk.

Earnings calls by speaker turns. Transcripts already come with turn boundaries and speaker names (XML format), and the financial NLP literature validates the turn as the natural processing unit, dropping operator turns and turns under 10 tokens (signal-free greetings). (LangChain Forum) A sector practitioner (FinTech Studios, Feb 2026) reports a 34% improvement in retrieval precision for earnings queries with turn-based chunking versus fixed 512-token windows — a vendor figure with no published methodology, but whose direction (structure > fixed window) is corroborated by independent sources. (🦜️🔗 LangChain)

from langchain_core.documents import Document

def chunk_earnings_call(turns: list[dict], ticker: str, quarter: str) -> list[Document]:
    """turns = [{'speaker': 'Jamie Dimon', 'role': 'CEO', 'section': 'prepared_remarks',
                'text': '...'}, ...]"""
    docs = []
    for t in turns:
        if t["role"] == "Operator" or len(t["text"].split()) < 10:
            continue  # noise with no signal (cf. arXiv 1906.02868)
        docs.append(Document(
            page_content=f"[{t['speaker']} — {t['role']}] {t['text']}",
            metadata={
                "ticker": ticker, "quarter": quarter,
                "speaker": t["speaker"], "role": t["role"],
                "section": t["section"],  # prepared_remarks | qa
            },
        ))
    return docs

Regulatory texts. Circulars, technical standards, and rulebooks have article hierarchies and cross-references ("pursuant to article 12.3.b"). Chunking must respect the chapter-section-article hierarchy and resolve cross-references by materializing the referenced text or its identifier in metadata; otherwise, a chunk that says "the provisions of the previous section shall apply" reaches the LLM without the content that completes it. The following table summarizes the technique by document type.

Table 5.2 — Structure-aware chunking by financial document type: unit, metadata, technique, and the risk of generic chunking

5.3 Advanced retrieval

With the corpus chunked by structure, the retrieval layer decides which candidates reach the LLM. The 2026 production stack has four ordered components: metadata filter → hybrid search → reranking → parent-context expansion, all pluggable as a BaseRetriever into LangChain v1's canonical chain, create_retrieval_chain(retriever, create_stuff_documents_chain(llm, prompt)). (Github)

5.3.1 Hybrid search, reranking, metadata filtering, and parent context

Metadata filtering first. The most common financial RAG error is retrieving the right figure from the wrong period; the first line of defense is narrowing the subset before vector similarity with the lineage fields from chunking. (LangChain Forum)

retriever = vectorstore.as_retriever(search_kwargs={
    "k": 6,
    "filter": {
        "$and": [
            {"ticker": {"$eq": "JPM"}},
            {"fiscal_year": {"$gte": 2023}},
            {"item": {"$in": ["Item 7", "Item 8"]}},
        ]
    },
})

Hybrid BM25 + dense search. Dense search fails on exact identifiers (tickers, ISINs, instrument names, taxonomy codes) and BM25 fails on semantics; combining them with Reciprocal Rank Fusion (RRF — a fusion that scores each document by the sum of the inverses of its ranks in each list) is the production standard: 2026 vendor figures report 91% recall@10 versus 78% (dense only) and 65% (BM25 only), with ~6 ms of fusion overhead. (LangChain) In LangChain, EnsembleRetriever implements RRF with configurable weights; the typical starting point is 0.6 semantic / 0.4 keyword, raising the BM25 weight when queries contain exact terms such as tickers or codes. (🦜️🔗 LangChain)

from langchain_classic.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

bm25 = BM25Retriever.from_documents(chunks); bm25.k = 10
dense = vectorstore.as_retriever(search_kwargs={"k": 10})
hybrid = EnsembleRetriever(retrievers=[dense, bm25], weights=[0.6, 0.4])
# In finance, tickers/ISIN/instrument names argue for raising the BM25 weight.

The embedding is also a domain decision: in a financial retrieval benchmark (HR@5), finance-adapted embeddings reach 88% versus 86% for OpenAI text-embedding-3-large, 85% for Cohere embed-english-v3.0, and 84% for Google text-embedding-004, (CSDN博客) and replacing ada-002 with financial embeddings on FinanceBench adds +8 accuracy points. (Towards AI) Budgeting for a domain embedding (e.g., voyage-finance-2) is among the cheapest improvements in the pipeline.

Reranking. Hybrid retrieval returns candidates; the reranker reorders them by reading the full query and document. Cohere Rerank 3 accepts up to 4,000 input tokens, which allows reranking with full-document context instead of the isolated chunk, recovering relevance that chunking destroyed. (🦜️🔗 LangChain) It integrates as a compressor over any base retriever:

from langchain_classic.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank

reranker = CohereRerank(model="rerank-v3.5", top_n=5)
final_retriever = ContextualCompressionRetriever(
    base_retriever=hybrid,      # BM25+dense ensemble, or multi-query
    base_compressor=reranker,
)

Parent context. The precise-embeddings vs. wide-context dilemma remains: ParentDocumentRetriever resolves it by indexing small child chunks (~400 tokens, precise embeddings) but returning the parent document (~2,000 tokens) on retrieval. (arXiv.org) Financial application: the child is the paragraph with the figure; the parent, the Item's full subsection (the entire "Liquidity and Capital Resources" note), so that the LLM receives the figure with its management narrative.

from langchain_classic.retrievers import ParentDocumentRetriever
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.storage import InMemoryStore

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=InMemoryStore(),
    child_splitter=RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50),
    parent_splitter=RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200),
)
retriever.add_documents(filing_docs)  # indexes children in vectorstore, parents in docstore

MMR and multi-query. When the top-k results are near-duplicates (typical in repetitive annual filings), Maximal Marginal Relevance (MMR) balances relevance and diversity over a pool of fetch_k candidates, with lambda_mult between 1.0 (pure similarity) and 0.0 (pure diversity): (🦜️🔗 LangChain)

where $R$ is the candidate set, $S$ the set of already-selected documents, $q$ the query, and $\lambda$ the weight of relevance versus diversity. Multi-query retrieval generates several paraphrases of the query with an LLM, retrieves for each one, and fuses, improving recall and robustness to phrasing; it is a standard production pattern alongside MMR, at the cost of additional LLM calls. (🦜️🔗 LangChain) LLM-based contextual compression improves precision at the cost of ~doubling latency: reserve it for high-value queries. (🦜️🔗 LangChain)

The resulting full pipeline is summarized in the following diagram.

5.4 The text/numbers split

5.4.1 Principle: text goes to the vector store, figures to structured tools

The embedding of a number does not preserve its value: it encodes distributional context, not magnitude. Hence the first-order rule of financial RAG — the golden data rule from Module 4 (§4.4.1) applied to retrieval —: prose goes to the vector store with section lineage; figures go to structured bases (SEC XBRL facts, SQL tables) accessible via tool calling; and every cited figure must exist verbatim in the source. Three pieces of evidence support it. Tables "are where the exact figures live", and standard chunking "turns tables into word soup and separates numbers from the sections that give them meaning": the documented solution (Red Hat, Jul 2026) keeps rows and XBRL facts as first-class nodes with concept, period, and currency metadata. (csdn.net) XBRL-Agent quantifies the benefit: RAG + symbolic calculators produced +17 points on taxonomy queries and +42 on numerical reasoning versus base LLMs, which "hallucinate or misinterpret financial content" over XBRL. (getvocal.ai) And scale errors (millions read as thousands) are 15% of extraction failures: an explicit scale field and reading from the structured source eliminate them by construction. (🦜️🔗 LangChain)

The resulting architecture splits the query into two planes:

Verbatim verification is code, not prompting: a second pass checks that every figure in the output appears literally in the cited excerpt (the two-step verification pattern; the dominant failure mode of LLM reasoning is arithmetic, not logic, as PAL demonstrated: 16 of 25 chains of thought were identical except for the numbers; the full development of this evidence is in §6.2). (Financial Wisdom TV)

import re

def cifras_en_respuesta(texto: str) -> list[str]:
    """Extract numeric literals from an answer (supports scale and separators)."""
    return re.findall(r"\$?\d[\d,.]*\s*(?:million|billion|thousand|%)?", texto)

def verificacion_verbatim(respuesta: str, contextos: list[str]) -> dict:
    """Every cited figure must exist VERBATIM in some retrieved context."""
    corpus = " ".join(contextos).replace(",", "")
    no_soportadas = [
        c for c in cifras_en_respuesta(respuesta)
        if c.replace(",", "").strip() not in corpus
    ]
    return {
        "ok": not no_soportadas,
        "cifras_no_soportadas": no_soportadas,
        # In production: if ok=False -> abstain ("NOT_AVAILABLE")
        # or route to a human review queue, NEVER publish the figure.
    }

5.5 RAG evaluation

An unevaluated RAG pipeline is a pipeline that has already failed silently. Institutional evaluation is done separately for retrieval and generation, and then end-to-end over a golden dataset: (towardsai.net) retrieval is measured with ranking metrics (context precision/recall), generation with faithfulness and answer relevancy. (ActiveWizards: data science and engineering lab)

5.5.1 RAGAS, a proprietary golden dataset, and acceptance thresholds

The four RAGAS metrics (operational definitions from arXiv 2409.09046): (ActiveWizards: data science and engineering lab)

• Faithfulness: fraction of atomic claims in the answer that are inferable from the retrieved context — the hallucination detector. An LLM judge decomposes the answer into claims and verifies each one against the context.
• Answer Relevancy: mean cosine similarity between the original question and questions regenerated from the answer; it detects answers that do not address what was asked.
• Context Recall: fraction of ground-truth claims attributable to the retrieved context — it measures whether retrieval brought the necessary evidence.
• Context Precision: rank-weighted precision@k — it rewards relevant chunks appearing at the top of the list.

The retrieval metric underlying context recall is recall@k: given the set $R_q$ of relevant passages for query $q$ and the $k$ retrieved passages $D_k$,

That is, the fraction of the necessary evidence that appears within the top $k$ results; it is the metric that justifies hybrid search (91% vs. 78% reported recall@10) (LangChain) and the one a proprietary golden dataset measures against the annotated evidence passages.

RAGAS's key diagnostic value is failure separation: low faithfulness with high context recall indicates a generation problem (the model had the documents but distorted them); low context recall indicates a retrieval problem (the documents were never retrieved). Faithfulness and Answer Relevancy are reference-free (they require no golden answer), which makes them scalable to real traffic. (shafiqulai.github.io)

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    Faithfulness, AnswerRelevancy,
    LLMContextPrecisionWithReference, LLMContextRecall,
)
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

judge_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
judge_emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))

rows = {"question": [], "answer": [], "contexts": [], "reference": []}
for item in EVAL_QUESTIONS:  # e.g. FinanceBench-style questions
    result = rag.invoke({"input": item["q"]})
    rows["question"].append(item["q"])
    rows["answer"].append(result["answer"])
    rows["contexts"].append([d.page_content for d in result["context"]])
    rows["reference"].append(item["ref"])

result = evaluate(
    Dataset.from_dict(rows),
    metrics=[Faithfulness(), AnswerRelevancy(),
             LLMContextPrecisionWithReference(), LLMContextRecall()],
    llm=judge_llm, embeddings=judge_emb,
)
print(result.to_pandas().mean(numeric_only=True))

A proprietary FinanceBench-style golden dataset. The template: expert-written questions about the firm's own filings, each one with (a) a golden answer with the exact figure and tolerance (±2.5% on numeric answers, as in FinanceBench), (b) annotated evidence passages (doc, section, page) that define $R_q$ for recall@k, and (c) a question type (extractive, cross-period comparative, multi-hop, with calculation) to break down failures. (API EvangelistAPI Evangelist) Some 100-200 questions over 20-40 portfolio filings cover 80% of a desk's query modes. LangSmith complements this with a structured-output correctness grader ("grade ONLY for factual accuracy against the ground truth; additional information is acceptable if it does not contradict it") and per-trace drill-down. (matt-harrison.com)

Acceptance thresholds for production. The 2026 RAGAS guide with CI gating proposes as a starting point: faithfulness ≥ 0.85, answer relevancy ≥ 0.80, context precision ≥ 0.75, and context recall ≥ 0.80; (shafiqulai.github.io) real financial pipelines on FinanceBench report faithfulness around 0.86-0.87 as an achievable reference. (CallSphere) Two finance-specific gates are added to these thresholds: (1) verbatim verification rate = 100% of the figures in the evaluation sample (§5.4) — deterministic, non-negotiable; (2) adequate abstention rate: on questions with no evidence in the corpus, the correct answer is "NOT_AVAILABLE", and a fabrication rate >0% on that subset blocks deployment. Judge cost control is solved with a cheap model (gpt-4o-mini), temperature 0, and PR sampling. (shafiqulai.github.io)

Applied case: research desk on banking filings

A banking research team deploys the module's pipeline over the 10-K/10-Qs and earnings calls of its 40 covered banks: chunking by Items with section_path and turns with role; financial embedding; Qdrant with metadata filtering; EnsembleRetriever (0.6/0.4) → CohereRerank (top 5) → ParentDocumentRetriever; XBRL facts and ratios in SQL/calculator tools; mandatory verbatim verification. Golden dataset: 150 FinanceBench-style questions written by the analysts themselves.

The first RAGAS measurement shows context recall 0.62 — a retrieval problem: 40% of failures concentrate on queries with exact tickers and instrument codes. Raising the BM25 weight to 0.5 and adding the fiscal_year filter lifts recall to 0.84. The second measurement shows faithfulness 0.79 — now a generation problem: the LLM was deriving year-over-year changes in text. All calculation is moved to the ratio tool and verbatim verification is activated: faithfulness rises to 0.88 and unsupported figures drop to zero. The production gate (faithfulness ≥ 0.85, recall ≥ 0.80, verbatim 100%, 0% fabrication on no-evidence questions) is passed, and the system enters a pilot with human review of every figure feeding client notes. The transferable lesson is the diagnostic loop: each metric pointed to a different layer of the pipeline, and no improvement came from prompting.

Exercises

1. Chunking a 10-K by Items. Download JPMorgan's most recent 10-K from EDGAR (Module 4) and implement chunk_10k_by_items (§5.2.1) with a complete section_path. Verify that Item 1A and Item 7 end up separated, re-split Items >2,000 tokens propagating metadata, and write a test: no chunk without item and fiscal_year.
2. Measured hybrid search. On the exercise 1 corpus, build a dense-only retriever and a 0.6/0.4 EnsembleRetriever. With 20 questions mixing semantic queries ("liquidity risks") and exact ones ("Common Equity Tier 1 ratio", tickers), measure recall@10 against annotated evidence and compare with the 78% vs. 91% reference.
3. Reranking and parent context. Add CohereRerank (or a local cross-encoder ms-marco-MiniLM-L-6-v2) and ParentDocumentRetriever (child 400 / parent 2,000). Measure RAGAS context precision before and after on the same 20 questions and quantify the added latency per query.
4. Text/numbers split. Implement xbrl_lookup (SEC company facts) and the ratio calculator; rewrite five calculation questions ("year-over-year change in net margin") to be solved via tool calling. Apply verificacion_verbatim and report the unsupported-figure rate before and after.
5. RAGAS evaluation with a CI gate. Run evaluate() (§5.5.1) with a proprietary golden dataset of ≥30 FinanceBench-style questions (golden answer + evidence passages). Diagnose retrieval-vs-generation with the faithfulness × context recall matrix and set the 0.85/0.80/0.75/0.80 thresholds as a CI job that blocks the merge.
6. Controlled abstention. Add 10 questions with no answer in the corpus (another period or another company). Measure the fabrication rate and adjust prompt and verifier until 0% invented figures with "NOT_AVAILABLE" abstention. Discuss why this gate outweighs a point of accuracy for a risk manager.

Module 5 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; market prices and figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.