Prompt engineering and structured output for finance

Module 5 built the retrieval layer: the context that feeds the prompts. This module addresses the second half of the problem: how the model is instructed to turn that context into reliable, auditable outputs consumable by front-office systems. It closes Block II (understanding information) and sets up Block III (risk and portfolios). Two principles run through it, both with direct empirical backing: the LLM never calculates —all arithmetic is delegated to deterministic tools— and every cited figure is verified by exact-match against its source, not by another LLM's judgment (Financial Wisdom TV).

6.1 Techniques by task type

The central lesson of the 2023–2026 evidence is that there is no "best prompt": there is a technique × task correspondence, and choosing badly not only raises costs —in some cases it actively degrades accuracy (Day Trading).

6.1.1 Role-based system prompts: sell-side analyst and risk manager

Role prompting (assigning an expert role in the system message) is well supported in finance. OpenAI's official guide recommends structuring the developer message into four sections —Identity, Instructions, Examples, and Context— with the context at the end of the prompt and delimited with XML tags (towardsai.net). In financial sentiment, the role-playing format outperforms flat prompts (arXiv.org), and in multi-agent decision architectures the pattern is formalized into [ROLE], [GOAL], [RULES], [FORMAT] blocks, with a Strategist → Critic → Moderator separation that improves the detection of evidentiary gaps and numeric inconsistencies relative to a single agent (Learnsignal).

Template 1 — sell-side analyst system prompt (production):

# Identidad
Eres un analista sell-side senior de renta variable (sector tecnología) con 15 años
de experiencia. Escribes para inversores institucionales.

# Instrucciones
- Toda cifra que cites debe proceder EXCLUSIVAMENTE del bloque <filing_excerpt>.
- Si un dato no está en el contexto, responde "NO_DISPONIBLE" en ese campo.
- No calcules ratios mentalmente: para cualquier cálculo, invoca la herramienta
  `calculator`. La aritmética en texto está prohibida.
- Distingue siempre entre hechos (del filing) e inferencias (tu juicio); etiqueta
  cada párrafo como [HECHO] o [INFERENCIA].
- Horizonte del análisis: 12 meses. Indica siempre la fecha "as-of" de los datos y
  el period_end de cada cifra. No uses información posterior a {as_of}.

# Formato de salida
Devuelve ÚNICAMENTE un objeto JSON conforme al schema InvestmentThesis (adjunto).

# Contexto
<filing_excerpt source="10-K AAPL FY2025" as_of="2025-11-01" page="31">
...
</filing_excerpt>

Template 2 — risk manager system prompt:

# Identidad
Eres un risk manager de un fondo multi-estrategia. Tu prioridad es la preservación
de capital y el cumplimiento normativo; ante la duda, sé conservador.

# Instrucciones
- Evalúa cada propuesta con el esquema: evidencia → regla/política → conclusión.
- Verifica la coherencia aritmética (P5 ≤ P50 ≤ P95, rangos y totales) antes de
  aprobar; las comprobaciones mecánicas las ejecuta el código, no tú.
- Rechaza cualquier recomendación sin cuantificación de incertidumbre o sin citas.
- Nunca modifiques límites de riesgo; solo los evalúas contra la política adjunta.

Three details are not decorative. The negative instruction ("if it is not in the context → NO_DISPONIBLE") is the primary defense against hallucination: without it, the model fills gaps with potentially stale training knowledge (IDEAS/RePEc). The as_of date is a point-in-time defense: standard models exhibit look-ahead bias, with out-of-sample alpha decay of more than 15 percentage points, because they memorize outcomes instead of predicting them (metricgate.com). And the ban on in-text arithmetic anticipates the evidence in section 6.2.

Few-shot: three examples are enough

Few-shot prompting (including input-output examples in the prompt) is the technique with the most robust evidence for financial sentiment classification. The seminal study by Lopez-Lira and Tang (2023) showed that headline sentiment scores predict daily returns, and that predictability is an emergent capability of large models (GPT-3.5/4 yes; GPT-1/2 and BERT no), with a Sharpe of 3.8 for GPT-4 in a 2021–2022 long-short without costs (arXiv.org). On dosage: in lightweight LLMs (Qwen3-8B, Llama3-8B) the jump from 0-shot to 3-shot raises average accuracy from 0.67 to 0.74, but from 3-shot to 5-shot the improvement stops being substantial (arXiv.org). The additional lever is not more examples but verbose category descriptions and an escape "Other" class (metricgate.com).

Template 3 — 3-shot sentiment classifier (AD-FCoT format, with explicit causality and no look-ahead):

Eres un analista financiero. Lee la noticia y razona paso a paso sobre su impacto
en la acción; después emite Positive/Negative/Neutral.

<ejemplo>
Noticia: "ACME retira del mercado su producto estrella por defectos de seguridad."
Razonamiento: La retirada implica costes directos, riesgo de litigios y pérdida de
ingresos del producto principal → impacto negativo en próximos trimestres.
Sentimiento: Negative
</ejemplo>
<ejemplo>
Noticia: "ACME supera expectativas de beneficio y eleva la guía anual."
Razonamiento: Beat de EPS + guía al alza señalan momentum operativo;
históricamente el mercado reacciona con re-rating positivo.
Sentimiento: Positive
</ejemplo>
<ejemplo>
Noticia: "ACME anuncia que cambiará el auditor del ejercicio 2027."
Razonamiento: Cambio rutinario de auditor sin implicación directa en resultados.
Sentimiento: Neutral
</ejemplo>

Noticia objetivo: {titular}
Razonamiento:

Chain-of-thought: where it helps and where it destroys accuracy

Chain-of-thought (CoT; inducing the model to verbalize intermediate steps) is the standard for multi-step reasoning, and in finance its value is real but bounded: in a multimodal filings benchmark, adding CoT raised calculation accuracy from 60.0% to 76.9% and reduced hallucination from ~15% to ~5% (🦜️🔗 LangChain); on FinQA/TAT-QA (numeric questions over annual reports), decomposition takes GPT-3.5-turbo from 28.4 to 51.4–55.1 points (metricgate.com). Methodological warning: the verbalized chain is not necessarily faithful to the model's internal computation; it is not explainability (ryanoconnellfinance.com).

The counter-evidence is the most counterintuitive finding in the area. On tasks where "thinking" hurts even humans, CoT degrades the model: in implicit statistical learning an absolute drop of 36.3 percentage points is observed, and in classification with exceptions to generalizable rules CoT increased the required iterations by up to 331%, because it biases the model toward the general rule and makes it ignore the contextual tokens carrying the correct answer (Day Trading). In classification with human ground truth, enabling CoT improved no model and slightly degraded some; Chain-of-Verification reduced accuracy by 1–2 points by retracting correct classifications (metricgate.com). In the legal domain —a structural analogue of the financial one— CoT helps contractual entailment (+8.5 pp) but hurts holdings identification (−16.0 pp) and multi-label classification (−3.8 pp) (arXiv.org). Explicit CoT also degrades instruction following (A.L. Capital Advisory).

Self-consistency samples $k$ chains and takes the majority answer:

Combined with program generation it raised GSM8K from 72.0% to 80.4% with $k=40$ (Financial Wisdom TV). Its documented limit: if the prompt or the context induces the same bias, the samples agree on the error (majority hallucination); the technique presupposes that the model has access to the necessary information (arXiv.org). In financial practice it is used with $k=5$–$11$ over extractions of the same metric, sending the minority answers to an audit queue.

Least-to-most decomposes the problem into subproblems solved sequentially, reusing previous answers; it solved the SCAN benchmark with ≥99% accuracy versus CoT's 16% (arxiv.org). A 2025–2026 warning: LLM-assisted decomposition can omit subtasks or introduce irrelevant steps; the mitigation is hybrid —the human fixes the skeleton (revenue → margins → cash flow → debt → valuation) and the LLM fills in the sub-questions (arXiv.org).

Table 6.1 — Prompting techniques: when to use each one, and the evidence

6.2 Grounding and anti-numeric-hallucination

6.2.1 The "answer only with data from the context" instruction and its documented limits

The strict grounding pattern has five components observed in documented financial architectures: (1) context delimited with XML tags and metadata (source, as_of, page); (2) an explicit negative instruction ("if the answer is not in the context, answer NO_DISPONIBLE; never use training knowledge"); (3) a mandatory citation with provenance for every figure; (4) a confidence field and an evidence_quote per claim in the schema; (5) subsequent two-step verification (Learnsignal). The canonical control block from the multi-agent literature is verbatim: [CONTROL] Do not introduce new data; rely exclusively on the CONTEXT (Learnsignal). Citations with source and date reduce hallucinations and ease auditing (Learnsignal); in real pipelines over FinanceBench, the RAGAS Faithfulness metric —the fraction of claims supported by the retrieved context— sits at ~86–87% (CallSphere).

The limits are structural, not stylistic. NumericBench documents that even frontier models (GPT-4o, DeepSeek-V3) fail at simple arithmetic, number retrieval, and multi-step reasoning, because they treat numbers as discrete tokens without numeric semantics; trivial perturbations of figures significantly reduce accuracy (博客园). The paradigmatic case is 9.11 vs 9.9: "9.11" tokenizes as ["9", ".", "11"] and the model compares the 11 and 9 fragments as integers, reinforced by software-versioning patterns in the training data; the serious part is that Claude Sonnet 4 and GPT-4o articulate the decimal-comparison procedure with textbook precision while executing it wrong —the so-called computational split-brain (langchain.js). At domain scale, a 2024 study cited by FailSafeQA estimates hallucination in up to 41% of financial queries, and minor wording differences ("net income" vs "earnings") alter the outputs (博客园). In filing extraction, the dominant failures are: retrieval failure (42%), multi-state gap (23%), value-scale error (15%) —reading "millions" as "thousands"— and figure fabrication (10%) (🦜️🔗 LangChain). That is why scale and unit are mandatory schema fields. The difficulty context: GPT-4 with retrieval scored ~19% on FinanceBench in the original configuration, and GPT-4 sits at 62.87% on FinQA versus 91.16% for the human expert (the full benchmark figures, in §5.1) (arXiv.org).

PAL and tool calling as mitigation: the LLM never calculates

The mitigation with the most quantified evidence is delegating computation. PAL (Program-Aided Language Models) showed that the dominant failure mode of CoT in mathematics is not reasoning but arithmetic —in 16 of 25 cases, CoT generated nearly identical "thoughts" with different numbers— and that executing the reasoning as a program lifts GSM-Hard from CoT's ~20% to 61.2% (Financial Wisdom TV). Toolformer had already shown that with access to a calculator the model solves arithmetic that pure models consistently fail (稀土掘金). The tool calling contract is explicit: the LLM does not execute the function; it proposes which function to call and with what arguments, and the code validates and executes (supermemory.ai).

from langchain_core.tools import tool
import ast, operator as op

_OPS = {ast.Add: op.add, ast.Sub: op.sub, ast.Mult: op.mul,
        ast.Div: op.truediv, ast.Pow: op.pow, ast.USub: op.neg}

@tool
def calculator(expression: str) -> float:
    """Evaluates an arithmetic expression. The ONLY authorized path for ANY
    calculation; the LLM must not do arithmetic in text."""
    def ev(node):
        if isinstance(node, ast.Constant): return node.value
        if isinstance(node, ast.BinOp): return _OPS[type(node.op)](ev(node.left), ev(node.right))
        if isinstance(node, ast.UnaryOp): return _OPS[type(node.op)](ev(node.operand))
        raise ValueError("expresión no permitida")
    return ev(ast.parse(expression, mode="eval").body)  # restricted AST, never eval()

@tool
def xbrl_lookup(ticker: str, concept: str, period_end: str) -> dict:
    """Returns a point-in-time XBRL concept from a SEC filing with citation (doc, page)."""
    ...

Two-step verification with fail-closed

The second pillar is deterministic post-generation verification: a code verifier checks that every figure appears literally in the declared verbatim quote. If it fails, the record is not persisted: it retries, returning the error to the model, and once retries are exhausted, it rejects (fail-closed). It is the application of institutional insight I6: for figures, exact-match in code > LLM-as-judge, because asking another LLM to judge a figure is circular.

def extract_with_retry(prompt: str, max_retries: int = 2) -> FilingMetric:
    last_error = None
    for _ in range(max_retries + 1):
        out = extractor.invoke([
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt + (f"\nError previo: {last_error}" if last_error else "")},
        ])
        if out["parsing_error"] is None:
            metric = out["parsed"]
            # Two-step verification: the figure must appear literally in the quote
            if metric.value is None or str(metric.value).replace(",", "") in \
               metric.evidence_quote.replace(",", ""):
                return metric
            last_error = "evidence_quote no contiene la cifra literal"
        else:
            last_error = str(out["parsing_error"])
    raise ValueError(f"extracción fallida tras reintentos: {last_error}")

Note the deliberate asymmetry: value = null passes verification (the model abstains), but a figure that is not in the quote never passes. Abstention is a legitimate result; fabrication is a system failure.

6.3 Pydantic schemas for the front office

6.3.1 InvestmentThesis and TradingSignal ready for OMS/risk

The reliability hierarchy of output mechanisms is documented: JSON mode only guarantees syntactically valid JSON (truncation is its leading cause of failure (博客园) ); function/tool calling types the arguments with ~86% schema compliance; and Structured Outputs (strict json_schema) reaches ~100% via grammar-constrained decoding (A.L. Capital Advisory). In LangChain, with_structured_output is the integration point: in langchain-openai the default method moved from function_calling to json_schema in 0.3.0, and with strict=True Pydantic models cannot use Field metadata (min/max) or default values (oneuptime.com); in agents, create_agent chooses ProviderStrategy (native structured output) if the provider supports it and ToolStrategy otherwise, with handle_errors for retries and the result in structured_response (Red Hat Developer). Essential design fact: the class name, the docstring, and the field descriptions are injected into the prompt —the schema IS part of the prompt (Quant Memo).

Full schema — investment thesis:

from pydantic import BaseModel, Field
from typing import Literal, Optional
from datetime import date

class Evidence(BaseModel):
    """Mandatory citation for every factual claim."""
    doc_id: str = Field(description="Identificador del documento, p.ej. 'AAPL_10K_FY2025'")
    page: int = Field(description="Página del documento fuente")
    quote: str = Field(description="Extracto VERBATIM (<= 40 palabras) que soporta la afirmación")

class MetricValue(BaseModel):
    """A financial metric with explicit unit, scale, and provenance."""
    value: Optional[float] = Field(
        description="Valor numérico. null si NO aparece en el contexto. NUNCA inventar.")
    unit: Literal["USD", "EUR", "shares", "percent", "ratio", "other"]
    scale: Literal["units", "thousands", "millions", "billions"] = Field(
        description="Escala tal como aparece en el filing (evita errores 10^3/10^6)")
    period_end: date = Field(description="Fecha de cierre del periodo reportado")
    as_reported: bool = Field(description="True si la cifra es literal del documento; "
                                          "False si se derivó de una tool call")
    evidence: Optional[Evidence] = None

class InvestmentThesis(BaseModel):
    """Structured investment thesis generated exclusively from the context."""
    ticker: str
    as_of: date = Field(description="Fecha de corte de la información (point-in-time)")
    rating: Literal["strong_buy", "buy", "hold", "sell", "strong_sell"]
    horizon_months: Literal[3, 6, 12, 24]
    revenue_ltm: MetricValue
    eps_ltm: MetricValue
    net_debt: MetricValue
    bull_case: list[str] = Field(max_length=3)
    bear_case: list[str] = Field(max_length=3)
    key_risks: list[str]
    confidence: Literal["low", "medium", "high"]
    reasoning_summary: str = Field(description="<= 100 palabras; marca [HECHO]/[INFERENCIA]")

Full schema — trading signal for OMS/risk integration:

class TradingSignal(BaseModel):
    """Signal consumable by an OMS; no field computed by the LLM."""
    instrument_id: str = Field(description="Identificador canónico (ISIN/FIGI/ticker interno)")
    as_of: date
    direction: Literal["long", "short", "flat"]
    strength: float = Field(ge=-1.0, le=1.0, description="Intensidad normalizada [-1, 1]")
    signal_family: Literal["sentiment_news", "fundamental", "technical", "composite"]
    valid_from: date
    valid_until: date = Field(description="Caducidad de la señal (point-in-time safety)")
    max_position_pct_nav: float = Field(ge=0.0, le=100.0)
    rationale_codes: list[str] = Field(description="Códigos de la taxonomía interna, NO texto libre")
    model_version: str = Field(description="Snapshot del modelo + hash del prompt, para auditoría")
    compliance_flags: list[Literal["restricted_list", "mnpi_risk", "liquidity_low"]] = []

The design notes for consumption by an OMS or risk engine are five: closed enums and numeric ranges on the decisional fields; explicit expiry (valid_until) as a point-in-time defense; traceability (model_version, prompt hash) per record; typed compliance flags; and the separation of powers from insight I1 —the LLM proposes and the risk engine disposes: the signal never executes directly. For diagnostics, include_raw=True returns {'raw', 'parsed', 'parsing_error'} and allows capturing errors without an exception (oneuptime.com); self-repairing parsers in the OutputFixingParser style retry with an LLM until valid or until the limit (博客园); and it is wise to set max_tokens at 2–3× the expected length against truncation (博客园).

6.4 Reasoning models and technique routing in 2026

6.4.1 reasoning_effort / budget_tokens: when to pay for reasoning

Since 2025–2026, reasoning depth is a governable API parameter: OpenAI's o-series exposes reasoning_effort (low/medium/high; GPT-5 adds minimal) and Claude exposes thinking.budget_tokens (typically 1024, 4096, or 16000) (Github). OpenAI's official guide frames the distinction: reasoning models generate an internal chain of thought, excel at complex tasks and multi-step planning, and are slower and more expensive; the official analogy is senior colleague (high-level objective) versus junior (explicit instructions) (towardsai.net). The routing rule is symmetric: raise the effort for mathematics, code, and tool-call planning; lower it for simple classification, extraction, and formatting —"reasoning depth is not the higher the better" (Github). On multi-capability financial benchmarks reasoning models lead (o1: 67.3% on XFinBench), but on trading-opportunity judgments every LLM failed the buy component of FinTradeBench (Github).

Production details that break real systems: budget_tokens is a soft cap, but max_tokens must be strictly greater or the API returns a 400 error —a bug shipped to production by real teams—; the response is multi-block (thinking + text) and you must iterate response.content (arXiv.org). Interleaved thinking allows continued reasoning between tool calls, useful in long agentic chains (Github).

Table 6.2 — Technique × task routing with evidence

Prompting vs fine-tuning: decision criteria

The decision is not ideological. With good prompting you reach 80–90% of fine-tuning performance at zero training cost and instant iteration; fine-tuning is justified for strictly consistent format, domain vocabulary, cost at high volume, and latency (財經無界). In regulated domains there is an additional argument: "prompt engineering does not give full control over the model's output; in high-risk, regulated domains such as legal, finance, and health, fine-tuning must be used" (Elastic). Economic viability already exists: fine-tuned lightweight LLMs (Qwen3-8B, Llama3-8B) are competitive in financial sentiment even with 5% of the data (arXiv.org). The operational criteria are four: (1) if the bottleneck is format or vocabulary, fine-tuning; if it is access to information or reasoning, prompting with grounding and tools; (2) if the token cost of few-shot exceeds the amortized training cost, fine-tuning; (3) if the regulator demands demonstrable control of outputs, fine-tuning plus structured output; (4) in both cases, continuous evals —snapshot changes alter prompt behavior, and OpenAI recommends pinning snapshots in production with evaluation suites (towardsai.net).

There are, moreover, limits that neither technique resolves: look-ahead bias is not fixed with prompts (it requires point-in-time data and evaluation) (metricgate.com); numeric incompetence is representational and is mitigated with tools, not instructions (langchain.js); and Lopez-Lira-style sentiment predictability is arbitraged away as it spreads —prompting gives access to the capability, not a defensive moat (arXiv.org). Systematic edge requires proprietary data, execution, and risk management; prompting alone was never an investment strategy.

Exercises

1. Role-based system prompt. Write the complete system prompt for a credit analyst evaluating a 10-K for a risk committee, with the Identity → Instructions → Examples → Context structure. It must include an as_of date, a negative grounding instruction, [HECHO]/[INFERENCIA] tagging, and a ban on in-text arithmetic. Justify each line by citing which failure mode from section 6.2 it mitigates; remove those that mitigate none.

2. Schema with semantic validators. Extend InvestmentThesis with: (a) a target_price: MetricValue field whose scale and unit are validated against revenue_ltm; (b) a @model_validator that rejects theses with an extreme rating and confidence="low" simultaneously; (c) a validator requiring non-null evidence whenever value is non-null. Explain why (c) cannot be guaranteed by constrained decoding.

3. Routing experiment. Implement the Template 3 classifier in two variants —direct zero-shot and with CoT— over 200 labeled headlines. Measure accuracy, latency, and token cost; contrast with the evidence of −36.3 pp and +331% iterations (Day Trading) and identify on which subset of headlines CoT fails first.

4. Two-step verification pipeline. Build the pipeline from the Mermaid diagram: extractor with with_structured_output(..., include_raw=True), normalized exact-match verifier (punctuation, thousands separators, scale), and a fail-closed policy with max_retries=2. Inject a filing where the requested figure does not exist and document whether the system returns null (correct) or a fabricated figure (failure); report the confidently wrong rate before and after making null part of the schema.

5. Reasoning budget. Configure three routes of the same research agent —news classification (minimal effort), metric extraction (low effort + tools), multi-step thesis (high effort)— and measure cost and quality per route. Add the max_tokens > budget_tokens guardrail and a regression test that fails if it is violated (Github).

6. Prompting vs fine-tuning decision. Write a one-page memo recommending prompting or fine-tuning for a high-volume corporate-events classifier in a regulated domain, applying the four criteria from section 6.4 and quantifying the cost crossover point with your provider's current prices.

Module 6 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; prices and market figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.