Production engineering for financial LLM systems

Learning objectives

• Instrument a LangChain/LangGraph agent with OpenTelemetry tracing in LangSmith, structuring versioned datasets with splits and offline/online evaluation.
• Select the right observability platform (LangSmith, Langfuse, Phoenix, Galileo, Braintrust) by license, data residency, and pricing model.
• Build a FinanceBench-style financial golden dataset and evaluate agents with numeric exact-match and trajectory metrics, not just LLM-as-judge.
• Integrate evaluation into CI/CD with quality gates that block merges on prompt regressions.
• Design a defense-in-depth guardrail stack: PII, fail-closed numeric grounding, rate limiting, circuit breakers, and kill switch.
• Budget tokens and latency per architecture (single-agent, debate, swarm) and delineate which trading decisions admit agents and which demand determinism.

Module 10 established observability as the defense against the governance vacuum: SR 26-2 explicitly excludes agentic AI from the model risk framework, so the firm remains fully responsible with no standard telling it how to govern it. This module implements that defense technically. The operating thesis is twofold: rigorous financial evaluation demands numeric and trajectory exact-match (course insight I6), and July 2026 token economics already allow production agents for research and risk, while the compounded latency of agentic loops delineates where they cannot operate (insight I7). Both ideas are developed with prices and cases verified as of July 2026.

11.1 Observability with LangSmith

11.1.1 Tracing, datasets, evaluators, experiments, dashboards, and alerts

LangSmith is LangChain's first-party platform and, as of July 2026, has consolidated as a complete "agent engineering" platform rather than a mere trace viewer (Divitae Assets) . In LangChain/LangGraph applications, tracing is enabled via environment variables (LANGSMITH_TRACING=true) with no additional instrumentation: every chain step, agent decision, and tool call is captured (Divitae Assets) . The platform is framework-agnostic —it traces applications built with the OpenAI, Anthropic, Vercel AI, LlamaIndex SDKs or custom implementations— and accepts OpenTelemetry traces via an OTLP endpoint, support added in March 2026 (SimTrade) . The SDK uses an asynchronous callback that ships traces to a distributed collector: application performance is unaffected and, if LangSmith suffers an incident, the agent keeps operating normally (SimTrade) . For a regulated desk, this property is not a detail: observability must never become a single point of failure for the decision pipeline.

The second piece is datasets: collections of examples (inputs, optional reference outputs, optional metadata) where reference outputs are not passed to the application, only used in evaluators, and metadata enables filtered views and versioned splits (questdb.com) . The official recommendation is to start with 5–10 hand-curated examples per critical component —for a financial agent, examples of correct tool selection, argument formatting, or reasoning trajectory— and grow from production (questdb.com) . Annotation queues close the loop: they turn production traces into dataset examples with rubrics, multiple reviewers, anti-conflict reservations, and direct export (questdb.com) .

LangSmith supports four evaluation techniques (questdb.com) : human (manual review with annotation queues; analyst sign-off on answers that feed reports), deterministic code (rule-based functions: numeric exact match of figures extracted from a 10-K, JSON schema validation), LLM-as-judge (reference-free or reference-based; useful for factual correctness against filing evidence or earnings-summary completeness), and pairwise (A/B comparison of two versions when absolute scoring is hard). The documentation warns that LLM-as-judge evaluators require careful score review and judge-prompt tuning, and that few-shot evaluators —with examples of inputs, outputs, and expected grades— usually improve performance (questdb.com) . An experiment is the result of evaluating a specific version against a dataset, capturing outputs, scores, and per-example traces, with comparison and pairwise experiments (questdb.com) . The recommended lifecycle: offline evaluation validates before deployment (benchmarking, regression testing, backtesting), online evaluation monitors live traffic, and online failures become permanent offline cases (questdb.com) .

In production, LangSmith generates prebuilt dashboards per project with trace sections (count, latency, error rate), LLM calls, Cost & Tokens (totals and per trace, by token type), most-used tools, and feedback scores; custom dashboards add p50/p99 latency metrics, time to first token (TTFT), and ratios with custom filters (Bing) . Alerts operate by threshold over five metrics (Run Count, Cost, Errors, Feedback Score, Latency), with 5- or 15-minute windows and Slack (native), PagerDuty, Dynatrace, or HTTP webhook destinations; the official best practice is to start with broad thresholds and refine (twopirllc.github.io) . The following snippet shows tracing activation, the creation of a dataset with a split, and the launch of an experiment with custom evaluators:

import os
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "research-agent-prod"

from langsmith import Client, evaluate

client = Client()

# Golden dataset with regression split (examples = inputs + reference outputs)
ds = client.create_dataset("golden-10k-extraction-v3",
                           description="Extracción de cifras de 10-K/10-Q, split regression")
client.create_examples(
    inputs=[{"question": "¿Cuál fue el EPS diluido de ACME en el FY25?"}],
    outputs=[{"answer": "$0.78", "evidence": "10-K FY25, p. 47"}],
    metadata=[{"split": "regression", "task": "extraccion_puntual"}],
    dataset_id=ds.id,
)

# Experiment: evaluates the current agent version against the dataset
results = evaluate(
    lambda inputs: agent.invoke({"messages": [("user", inputs["question"])]}),
    data="golden-10k-extraction-v3",
    evaluators=[exact_match_numeric, trajectory_exact_match],   # §11.2
    experiment_prefix="gpt56-sol-prompt-v3",
    metadata={"model": "gpt-5.6-sol", "prompt_version": "v3.2"},
)

On pricing, verified as of July 2026 (quantitativebrokers.com) : the Developer plan is free (1 seat, 5,000 base traces/month, 14-day retention); Plus costs $39 per seat/month with 10,000 traces included; Enterprise is custom (SSO, RBAC, configurable retention, self-hosted or hybrid). On July 21, 2026, LangChain restructured its pricing into two normalized currencies: 1 LCU = $1.50 (compute) and 1 LSU = $1.00 (traces and storage), with extra traces at 0.005 LSU each (Github) . A prudence signal for budgets: LangChain has changed its deployment billing meter three times in fourteen months (Github) — platform estimates should be made with the official calculator plus a contingency margin.

The relevant alternatives as of July 2026 are Langfuse (MIT open source, acquired by ClickHouse on January 16, 2026, free self-hosted with full feature parity on a ClickHouse backend) (Github) , Arize Phoenix (OTel-native via OpenInference, strong on span-level trajectory analysis) (Morioh) , Galileo (eval-first, proprietary groundedness metrics designed for RAG where hallucination is the dominant failure mode) (databento.com) and Braintrust (eval-first with CI/CD quality gates that block merges) (quantvps.com) .

Table 11.1 — LLM observability and evaluation platforms (as of July 2026)

Sources: (quantitativebrokers.com) .

The decision is rarely technical in the strict sense; it is a data-governance and operating-model decision. For a financial firm, the first question is residency: if traces —which contain filing excerpts, positions, and potentially PII— cannot leave the perimeter, self-hosted Langfuse under MIT offers feature parity at no license cost, and its acquisition by ClickHouse in January 2026 provides some corporate-continuity assurance (WilmerHale) . LangSmith wins when the stack is pure LangChain/LangGraph and the team values platform breadth (datasets with splits, annotation queues, Deployment, Sandboxes GA since May 2026) over data sovereignty (Divitae Assets) . Phoenix fits organizations that have already bet on OpenTelemetry as their cross-cutting observability standard. Galileo and Braintrust are eval-first bets: the former when the dominant risk is hallucination in document RAG (databento.com) ; the latter when prompt CI/CD discipline is the organizational bottleneck (quantvps.com) . A frequent pattern in mature teams is hybrid: tracing in Langfuse for residency, CI gates with promptfoo or Braintrust, and human evaluation in either platform's annotation queues. It is also wise to re-verify prices and licenses quarterly: this module's own evidence log detects discrepancies between sources on Langfuse's Pro tier and on Phoenix's exact license.

11.2 Rigorous financial evaluation

11.2.1 Golden datasets, exact-match, trajectories, replay, and prompt CI/CD

A financial golden dataset is a curated set of triplets (question → reference answer + evidence) built by domain experts, with ecological validity: it reflects the analyst's real tasks, not toy questions. The industry's reference model is FinanceBench (Patronus AI): 10,231 triplets over 40 public companies and 361 public filings from 2015–2023, each entry with an evidence string, document page, and company/sector/type labels (Coursiv) . Its operating lessons are directly transplantable: annotators with financial experience after screening, weekly review of 10–20% of cases per annotator, adjudication by a senior analyst, and an evaluation sample stratified by task type instead of pure random sampling (Coursiv) . A critical design finding: in long context, presenting the filing first and the question at the end drastically improved the evaluated models (GPT-4-Turbo: 78% vs 25%) — context-question ordering is a regression variable that must be pinned in the dataset and in CI gates (Coursiv) . Public seeds and baselines include FinQA (8,281 multi-step numeric reasoning pairs, execution accuracy metric), ConvFinQA (3,892 conversations), TAT-QA (16,552 hybrid table-text questions), and DocFinQA for ~123,000-word long context (BenchLM) .

The practical recipe for building your own dataset has five steps: (1) sample 100–1,000 real production traces and anonymize PII before curating —mandatory under GDPR—; (2) stratify by task type: single-figure extraction, derived calculation (growth, margins), multi-hop across sections, table vs. text and, critically, questions unanswerable from the document to measure abstention; (3) annotate with experts and 10–20% QC; (4) version with splits (base / regression / adversarial) and label difficulty and origin; (5) turn every confirmed production incident into a permanent regression case (Coursiv) .

Agent evaluation happens in three layers: final answer, trajectory (sequence of steps and tool calls), and per-turn in production (Price Per Token) . Mature programs track 5–8 trajectory metrics per agent (Morioh) : (1) trajectory accuracy against a golden trajectory; (2) tool-call precision and recall; (3) path efficiency (steps taken vs. shortest path); (4) cost efficiency (tokens and wall-clock per successful trajectory); (5) safety refusal rate; (6) recovery success rate under injected failures; (7) HITL gate adherence —the proportion of consequential actions escalated to human approval, an audit-critical metric in regulated banking—; and (8) loop detection rate. Replay testing complements goldens: capture real traces, anonymize them, curate a replay dataset of 100–1,000 traces, and re-run it against the new version and the baseline, flagging trajectories that diverge materially; goldens cover curated edge cases, replay covers the real distribution (Morioh) .

In institutional practice. Kensho (S&P Global) operates a multi-agent framework on LangGraph for financial data retrieval whose multi-stage evaluation relies on routing and tool-calling exact-match: the question is not "does the answer sound good?" but "did the agent select the right tool, with the right arguments, and does the final figure match the reference exactly?" (🦜️🔗 LangChain) . The reason is structural: an LLM judgment about a risk figure is circular —the evaluator shares the failure modes of the evaluated—. Semantic similarity does not work either: an output that changes "EPS $0.78" to "EPS $0.81" scores ~0.96 cosine while being a material defect (Price Per Token) . That is why the institutional standard reserves LLM-as-judge for non-numeric qualities (tone, completeness) and demands exact-match —after normalization— for routing, tool-calling, and figures.

Numeric exact-match requires prior normalization: "$1.85 billion" and "1,850 million" must be reduced to \(1{,}85 \times 10^{9}\) before comparing, with an explicit tolerance policy (exact equality, rounding, aliases, tolerance with qualifiers) (Price Per Token) . FinQA/ConvFinQA's execution accuracy runs the reasoning program and compares the result, separating extraction errors from arithmetic errors (BenchLM) . The following snippet implements a LangSmith numeric exact-match evaluator and a trajectory evaluator:

import re
from langsmith.schemas import Example, Run

_UNITS = {"billion": 1e9, "bn": 1e9, "million": 1e6, "mn": 1e6}

def normalize_number(text: str) -> float | None:
    """'$1.85 billion' and '1,850 million' -> 1.85e9."""
    m = re.search(r"\$?\s*([\d][\d.,]*)\s*(billion|bn|million|mn)?", text.lower())
    if not m:
        return None
    val = float(m.group(1).replace(",", ""))
    return val * _UNITS.get(m.group(2) or "", 1.0)

def exact_match_numeric(run: Run, example: Example) -> dict:
    got = normalize_number(run.outputs["answer"])
    ref = normalize_number(example.outputs["answer"])
    ok = got is not None and ref is not None and abs(got - ref) <= max(1e-9, 0.001 * abs(ref))
    return {"key": "exact_match_numeric", "score": float(ok)}

def trajectory_exact_match(run: Run, example: Example) -> dict:
    """Routing + tool-calling: expected vs. actual tool sequence."""
    expected = example.outputs["tool_sequence"]          # e.g. ["xbrl_facts", "calc_eps"]
    actual = [s.name for s in run.child_runs if s.run_type == "tool"]
    return {"key": "trajectory_exact_match", "score": float(actual == expected)}

The last piece is prompt CI/CD: the prompt is code —versioned, reviewed in PR, with an automatic quality gate and rollback (BenchLM) . promptfoo (OSS) keeps cases in YAML next to the code and returns a non-zero exit code if assertions fail, so the PR gate is two lines in GitHub Actions; --pass-rate-threshold 0.9 tolerates model noise without letting real regressions through, and saving the output as a CI artifact creates the audit trail for every prompt change (Price Per Token) . Braintrust offers a dedicated GitHub Action that runs experiments per PR, comments the breakdown (which cases improved or regressed and by how much), and blocks the merge below threshold (quantvps.com) . The minimum PR suite in finance: exact-match on the numeric golden, abstention rate on unanswerable questions, absence of PII in outputs, and schema validation of structured outputs (Price Per Token) .

# promptfooconfig.yaml — PR gate for the research agent
prompts:
  - file://prompts/research_v4.md
providers:
  - openai:gpt-5.6-sol
tests:
  - description: "EPS diluido FY25 ACME (regresión, incidente 2026-05)"
    vars: { question: "¿Cuál fue el EPS diluido de ACME en el FY25?" }
    assert:
      - type: javascript      # normalized numeric exact-match
        value: Math.abs(extractNumber(output) - 0.78) < 1e-6
      - type: not-contains-any   # PII guardrail
        value: ["SSN", "IBAN", "cuenta 4021"]

# .github/workflows/prompt-gate.yml (excerpt)
- name: Prompt regression gate
  run: npx promptfoo@latest eval --no-progress-bar --pass-rate-threshold 0.9
        # exit code != 0 => the merge is blocked  [(Price Per Token)](https://pricepertoken.com/pricing-page/model/openai-gpt-5) 

Human evaluation remains the final arbiter: FinanceBench evaluated 2,400 answers with manual expert review (Coursiv) , and LLM-as-judge evaluators must be calibrated against the human panel before any offline scoring is delegated to them (questdb.com) . Review sampling must be stratified by task type, with oversampling of high-impact cases —figures that feed decisions— and mandatory review of everything flagged by the numeric or PII guardrails in the next section.

11.3 Guardrails and operational security

11.3.1 Frameworks, numeric grounding, PII, rate limiting, circuit breakers, and sandboxing

The guardrail stack in finance is defense in depth with two reference frameworks, both Apache 2.0 (EvoLink) . Guardrails AI provides output validation and correction with more than 50 prebuilt validators (schema, ranges, format) and an added latency of 50–200 ms per validation, with parallel validator execution; NVIDIA NeMo Guardrails controls conversational flow with Colang scripting, stronger against jailbreak and prompt injection, at a typical 100–300 ms (50–150 ms on optimized NVIDIA infra) due to the extra LLM calls for intent recognition (EvoLink) . The pattern observed in 2026 is to use both: NeMo for topic control —the assistant does not opine on securities outside its mandate nor give personalized investment advice— and Guardrails AI for structural and semantic output validation, plus custom domain validators: units, magnitudes, signs, and temporal consistency (EvoLink) .

The most dangerous failure mode in finance is not semantic hallucination but numeric substitution: fluent, faithful text with one figure changed. The documented diagnostic case —source "EPS $0.78", output "EPS $0.81"— scores ~0.96 cosine similarity, invisible to any semantic check; a numeric grounding detector that extracts every numeric claim from the output and cross-checks it against the source does catch it (Price Per Token) . The World Bank's Proof-Carrying Numbers (PCN) protocol takes this idea to its strictest form: verification moves to the renderer, not the model; numeric spans are emitted bound to structured claims; the verifier applies declared policies (exact equality, rounding, aliases, tolerance) with fail-closed behavior —"trust is earned only by proof": only mechanically verified numbers are shown as verified, and the absence of a mark communicates uncertainty—; and it is anti-spoofing: the model cannot mark itself as verified (Vibe Engines) . Its explicit audience is financial reporting and regulatory compliance (Vibe Engines) . The known limitation of these detectors is that they flag the computed-correct-but-uncited as ungrounded; that is why the correct pipeline is detector → human review with the derivation shown → publication. One documented financial pipeline cut the hallucination rate from ~15% (LLM alone) to ~8% (RAG) and to ~2% by adding post-generation fact-checking (neuraltrust.ai) .

def numeric_grounding(output: str, source: str) -> dict:
    """Mechanical verification of figures against the source. Fail-closed policy."""
    claims = extract_numeric_claims(output)                    # regex + magnitude parser
    src = {normalize_number(t) for t in extract_numeric_claims(source)}
    grounded = [c for c in claims if c in src]
    ungrounded = [c for c in claims if c not in src]
    return {
        "grounding_rate": len(grounded) / max(1, len(claims)),
        "ungrounded": ungrounded,
        # Fail-closed: without full mechanical verification, nothing is published;
        # the output is marked "unverified" and goes to human review.
        "publish": len(ungrounded) == 0,
    }

For PII and sensitive data, the self-hosted standard is Microsoft Presidio (MIT): the Analyzer detects entities combining regex, Luhn-style checksums, NER, and context words; the Anonymizer applies replace/mask/hash/redact/encrypt operators (Meet Parse) . The reference strategy for financial pipelines is risk-tiered: Critical (SSN, cards, IBAN → strict regex + checksum, where a false negative carries the highest regulatory cost), High (phones, accounts → regex + contextual NER), Moderate (names → transformer NER), and Low (dates, generic IDs) (technolynx.com) . Custom recognizers —regex + context words for internal identifiers such as proprietary account numbers— are the most common customization (Meet Parse) . Two operating rules: anonymize before sending to the LLM with functional tags that preserve structure ([US_SSN], [IBAN]), and anonymize production traces before curating replay datasets (Morioh) . And one honest caveat: "fuzzy" entity detection is probabilistic; Presidio is "a strong risk-reduction control, not a compliance guarantee" — it demands tuned thresholds, custom recognizers, tests with real data, and human review in high-risk cases (Meet Parse) .

LLM rate limiting differs from classic API rate limiting because the cost per request is variable: a single request can cost from a fraction of a cent to several dollars (SumatoSoft) . Limiting requests alone is a mistake: you must count tokens (input + output + system prompt) and combine request, token, and cost limits, with a multi-level token bucket (RPM + TPM + budget), upfront estimation (tiktoken or a ~1.3 tokens-per-word heuristic), and logging of actual usage (SumatoSoft) . The documented safe rollout takes five weeks: audit-only → token bucket on the riskiest workload class → cost-velocity breaker at 10× the planned rate → loop-signature detector + fallback chains → steady state with quarterly re-tuning (AI Today Brief) .

Risk note. An agent can chain more than 12 tool calls per instruction; if the stop condition fails, billing grows without limit. Documented incident: a LangChain retry loop escalated from \(127/week to **\)47,000 in 11 days before anyone noticed (ZenML analysis of more than 1,200 production deployments); another publicly shared loop cost $30,000 (Dmytro Klymentiev) . Provider spend caps are not enough: they are monthly, they do not identify which agent caused the spike and, when they trip, they kill the entire account —including the healthy agents— (Dmytro Klymentiev) . The circuit-breaker taxonomy required in production: (1) step limits; (2) cost limits per session/day; (3) repetition/loop-signature detectors** (same tool with identical arguments N times in a short window); (4) cost-velocity breaker at 10× the planned rate, which catches the slow runaways the token bucket cannot see; (5) rate-threshold breaker —sustained 10,000 tokens/min caught a loop in under 60 seconds, because a healthy agent rarely sustains more than 3,000–4,000 tokens/min— (AI Today Brief) . Practical thresholds: measure baseline for two weeks; warning at 2.5× and halt at 5× average daily spend; on trip, alert with per-agent breakdown and human decision in under 5 minutes (Meet Parse) .

from langchain.agents.middleware import wrap_model_call

class CostCircuitBreaker:
    """Per-agent cost kill switch: halt at 100% of the daily budget."""

    def __init__(self, daily_limit_usd: float, p_in: float, p_out: float):
        self.limit, self.p_in, self.p_out = daily_limit_usd, p_in, p_out
        self.spent, self.recent_calls = 0.0, []

    @wrap_model_call
    def guard(self, request, handler):
        if self.spent >= self.limit:
            raise RuntimeError("Kill-switch: presupuesto diario agotado")
        sig = (request.messages[-1].content[:128])               # loop-signature
        self.recent_calls.append(sig)
        if len(self.recent_calls) >= 3 and len(set(self.recent_calls[-3:])) == 1:
            raise RuntimeError("Circuit breaker: bucle detectado (3 llamadas idénticas)")
        response = handler(request)
        u = response.usage_metadata or {}
        self.spent += (u.get("input_tokens", 0) * self.p_in
                       + u.get("output_tokens", 0) * self.p_out) / 1e6
        return response

For agents with access to financial execution (orders, payments), the list is stricter: per-agent tool allowlist, notional limits per operation and per session, dual human approval for irreversible actions, idempotent tool calls with a unique requestId to avoid duplicates on retries, and a gateway-level kill switch that revokes tool credentials, not just stops calling the LLM (Morioh) . In LangGraph this is implemented with interrupt_before on every node that executes an irreversible action (arXiv.org) . Finally, sandboxing of agent-generated code: Firecracker microVMs (E2B) or gVisor/Kata, never a shared container; network egress denied by default with an explicit allowlist; ephemeral filesystem; secrets injected by the host, never into the model's context; short TTLs and CPU/memory/time limits per execution (stockcharts.com) . LangSmith Sandboxes (GA May 2026) bills per second ($0.0576/vCPU-hr) (quantitativebrokers.com) ; E2B charges full wall-clock including idle — watch out when modeling costs at scale (portfoliometrics.net) .

11.4 Token economics and latency

11.4.1 Prices, cost levers, per-architecture budgets, and the latency frontier

API prices change frequently; every figure in this module is dated July 2026 with a source, and must be verified against the official pages before budgeting (Bajaj Finserv) .

Table 11.2 — Model prices per million tokens (as of July 2026)

Sources: BenchLM synced with the official OpenAI, Anthropic, and Google pages, sync Jul 28, 2026 (Bajaj Finserv) .

The reading of the table is an architecture lesson, not a purchasing one. At the flagship level, Gemini 3.1 Pro costs less than half of GPT-5.6 Sol and Claude Opus 5, which reshapes the design of high-volume research pipelines (Bajaj Finserv) . The intra-vendor spread matters even more: within OpenAI, from GPT-5 nano to GPT-5.5 Pro there is a 600× factor (Bajaj Finserv) ; sending everything to the flagship is, under the typical production difficulty distribution (60–75% simple queries, 15–25% medium, 5–15% hard), an overpayment of up to 50× on the simple bucket (ryanoconnellfinance.com) . Long context carries a premium rate: stuffing the entire filing "because it fits" doubles the price, and selective RAG is usually both cheaper and more accurate (Bajaj Finserv) . All three providers offer a Batch API with −50% on input and output for asynchronous work —evaluation suites, mass filing extraction, overnight generation— and prompt caching of up to −90% on repeated input tokens; at Anthropic both discounts are stackable (a batched cache hit on Opus 5 costs $0.25/M) (John Rothe, CMT | Investment Strategy) . A measurement caveat: the Claude 4.7+ tokenizer produces ~30% more tokens for the same text, per Anthropic — the math must be redone with tokens measured from the real model, not inherited from another provider (ryanoconnellfinance.com) .

Table 11.3 — Cost-reduction levers with measured effect

Sources: (metricgate.com) .

The levers are not mutually exclusive: the Singapore bank case —a compliance copilot that cut the monthly bill from $180,000 to $71,000 (−60%) in 90 days— combined caching on the system prompt and regulatory context (−47%), cascading to a mid-tier model for routine queries (−25%), and a re-ranker refining the RAG context (−12%), all with unchanged analyst acceptance (APILayer BlogAPILayer Blog) . It is the pattern to replicate: first measure the difficulty distribution of your own traffic, then apply the levers in order of increasing risk —caching and batch are conservative, routing touches perceived quality—. The key methodological caveat of routing: the threshold only makes sense calibrated against a quality bar measured on your own traffic, and quality must be reviewed by segment, because a small high-risk segment can silently degrade while the aggregate barely moves (DailyTickers) . In finance this translates into a hard rule: no regulatory, figure, or end-client query is routed to a cheap model without an exact-match eval backing it (DailyTickers) . Complementarily, the correct design metric for agentic strategies is accuracy-per-dollar: Tree-of-Thought reached top accuracy on TAT-QA compound arithmetic but cost 7.2× more tokens per query than Plan-and-Solve, which offered the best accuracy-per-dollar on purely numeric tasks (metricgate.com) .

The token budget turns these levers into engineering discipline: estimate before, log after, count all three token types (input, output, system), budget per session/agent/day with an alert at 80% and halt at 100% (SumatoSoft) . The cost per decision of an agentic architecture is modeled as:

where \(n_{\text{calls}}\) is the number of LLM calls per decision, \(T_{\text{in}}\) and \(T_{\text{out}}\) the average tokens per call, and \(p_{\text{in}}\), \(p_{\text{out}}\) the rates per million tokens. The critical factor is \(n_{\text{calls}}\): a 5-agent swarm can triple token spend versus a monolith on the same task, because each handoff adds context overhead (CSDN博客) , and a TradingAgents-style analysis consumes ~30–50 LLM calls per ticker (Global Database) . Reliability also decays with the chain: a 10-step pipeline with 90% per-step reliability delivers only 34.9% end-to-end success (MarketXLS) .

PRECIOS = {  # $/M tokens, as of Jul 2026  [(Bajaj Finserv)](https://www.bajajfinserv.in/investment/maximum-drawdown) 
    "gemini-3.1-pro": (2.0, 12.0),
    "claude-sonnet-5": (2.0, 10.0),   # intro rate until Aug 31, 2026
    "gpt-5.6-terra": (2.5, 15.0),
}

def coste_decision(n_calls: int, tok_in: float, tok_out: float, modelo: str) -> float:
    p_in, p_out = PRECIOS[modelo]
    return n_calls * (tok_in * p_in + tok_out * p_out) / 1e6

# Per-architecture budget (12k tok in / 2k tok out per call, Gemini 3.1 Pro)
ARQ = {"single": 5, "debate": 15, "swarm": 40}   # LLM calls per decision  [(CSDN博客)](https://blog.csdn.net/s060403072/article/details/160055663) 
for nombre, n in ARQ.items():
    c = coste_decision(n, 12_000, 2_000, "gemini-3.1-pro")
    print(f"{nombre:8s} {c:.3f} USD/decisión -> {c * 200 * 21:,.0f} USD/mes a 200 decisiones/día")
# single  0.240 USD/decisión -> 1,008 USD/mes
# debate  0.720 USD/decisión -> 3,024 USD/mes
# swarm   1.920 USD/decisión -> 8,064 USD/mes

The figure shows the estimated monthly inference cost for the three architectures at volumes from 50 to 2,000 decisions/day, with the assumptions marked on the chart itself (Gemini 3.1 Pro at Jul 2026 prices, 12k input tokens and 2k output tokens per call, 21 market days/month, no caching, batch, or routing). Three readings: first, at the daily research volume of a mid-size desk (50–200 decisions/day) even the full swarm is affordable in absolute terms —between $2{,}016 and $8{,}064 per month—; second, the multiplier between architectures (3× from single to debate, 8× from single to swarm, by number of calls) is structural and no price lever removes it, because it acts on \(n_{\text{calls}}\) and not on rates —and it is also conservative: it does not include the context overhead of up to ~3× tokens per call that a swarm's handoffs drag along (Module 3, §3.4); third, at 2,000 decisions/day the swarm approaches $81,000 per month, territory where routing and caching stop being optional.

Latency is the other frontier, and it is the one that delineates where agents can operate. A real agentic turn is 2–5 LLM hops plus tool calls, compounding a latency of 3–10× over single-model benchmarks (CNY終極指南：一篇搞懂香港開戶、增值、大灣區消費全攻略 – CHMFIA 中港澳金融資訊交流協會) :

The levers, ordered by impact: reduce hops; prompt caching (60–80% of an agent's input tokens are cacheable: system prompt + tool definitions); parallelize independent tool calls (cuts 40–60% of turn time); mix models per hop (fast for routing, strong for synthesis); stream the final hop; per-component timeouts; and measure P99, not P50 —a P50 of 2 s with a P99 of 12 s means 1 in 100 users has a bad experience— (CNY終極指南：一篇搞懂香港開戶、增值、大灣區消費全攻略 – CHMFIA 中港澳金融資訊交流協會) . The reference SLO framework places research agents on a budget of up to 30 seconds per turn, versus <3 s for interactive chat (CNY終極指南：一篇搞懂香港開戶、增值、大灣區消費全攻略 – CHMFIA 中港澳金融資訊交流協會) .

From this follows the course's practical frontier (insight I7): agents for research, daily risk, and reporting; determinism for intra-day execution. An agent that takes 8–30 seconds to compose its answer has no place on the critical path of a time-sensitive order, where execution demands a deterministic FSM with millisecond latencies; that same agent, by contrast, is perfectly viable —and affordable, per the figure above— for overnight watchlist analysis, the morning risk report, or a PM's ad-hoc research. Autonomy grows upward in the stack (research → decision), never toward execution; between decision and order there is always a deterministic FSM, HITL interrupt, and Postgres checkpointer as audit log.

c_route = (0.70 * (12_000 * 0.10 + 2_000 * 0.40)
         + 0.25 * (12_000 * 2.0  + 2_000 * 10.0)
         + 0.05 * (12_000 * 5.0  + 2_000 * 25.0)) / 1e6
# 0.70 × $0.002 + 0.25 × $0.044 + 0.05 × $0.110 = $0.0179 per call

EXPANDE

The practical frontier, in one map

The entire module converges on a single design decision — where the agent goes and where determinism goes (insight I7). This is the map of that frontier with the module's figures:

Three things the map does not show and worth keeping in mind while doing the exercises: compounded reliability punishes long chains (10 steps at 90% per step → 34.9% end-to-end success), latency is budgeted at P99, not P50 (a P50 of 2 s with a P99 of 12 s is a bad experience for 1 in 100 users), and autonomy grows upward in the stack —research → decision— never toward execution: between decision and order there is always a deterministic FSM, HITL interrupt, and checkpointer as audit log.

Exercises

1. Minimum golden dataset. Build a 20-query golden dataset over two real 10-Ks following the FinanceBench recipe: stratify into four buckets (single-figure extraction, derived calculation, multi-hop across sections, unanswerable from the document —at least 3 queries of this last type—), with reference answer, evidence (document + page), and calculation rationale. Upload it to LangSmith with base and regression splits and task-type metadata. (Coursiv)

2. Exact-match evaluator. Implement the exact_match_numeric evaluator from the §11.2 snippet with magnitude normalization and an explicit tolerance policy (exact / rounding / 0.1% tolerance). Validate it with adversarial cases: "$1.85 billion" vs "1,850 million" (must pass) and "EPS $0.81" vs "EPS $0.78" (must fail). Run it with evaluate() on the exercise 1 dataset and compare its verdict with an LLM-as-judge evaluator on the same 20 cases: document the discrepancies. (Price Per Token)

3. Prompt CI gate. Set up the promptfoo suite from §11.2 on the research agent: golden exact-match, abstention rate on unanswerables, PII absence, and schema validation. Integrate it into GitHub Actions with --pass-rate-threshold 0.9, deliberately introduce a regression in the prompt (e.g., reorder context-question), and demonstrate that the merge is blocked with the breakdown on the PR. Save the output as a CI artifact. (Coursiv)

4. Per-architecture token budget. With the cost-per-decision formula and the prices in Table 11.2, budget three architectures (single 5 calls, debate 15, swarm 40) for a desk with 200 decisions/day, under two model scenarios (all Gemini 3.1 Pro vs. 70/25/5 routing over Flash-Lite/Sonnet 5/Opus 5). Quantify the routing savings, then apply Batch API (−50%) to 40% of the overnight volume, and defend the chosen architecture with the accuracy-per-dollar metric. (metricgate.com)

5. Circuit breakers and kill switch. Implement the CostCircuitBreaker from §11.3 with all five breaker types (step, cost, loop-signature, cost-velocity, rate-threshold). Simulate a runaway (a tool that always returns the same partial result) and verify the loop-signature detector stops it in under 3 identical iterations. Set the warning/halt thresholds at 2.5×/5× over a two-week synthetic baseline and write the runbook: alert → per-agent breakdown → human decision in <5 minutes → tool credential revocation. (AI Today Brief)

Module 11 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; market prices and figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.