LLM signals and rigorous backtesting

Modules 7 and 8 equipped the reader with the risk framework and portfolio construction with Black–Litterman views. This module closes Block III by walking the full path from signal to order: what the LLM trading-agent literature really says, how a natural-language output becomes a defensible quantitative signal, how it is validated to the standards an allocation committee would demand, and how it connects to execution without ceding control to a stochastic system. The guiding principle, already anticipated in earlier modules, remains intact: the LLM reasons, the math decides, the human vetoes; agent papers are treated here as an object of study, not as a deployable strategy (ProfitLogic).

9.1 The state of the art of trading agents

9.1.1 Verified landscape: architectures, headline numbers, and documented biases

Between 2023 and 2026 the academic literature produced a recognizable genealogy of LLM agents for trading: FinMem (arXiv 2311.13743, AAAI Symposium 2024) introduced memory layered by time horizon —shallow layers (daily news, high decay), intermediate (10-Q), and deep (10-K and reflections, low decay)— with a switchable risk profile, and reported Sharpe > 2.0 and cumulative returns > 0.35 on TSLA and NFLX across five mega-caps in 2021–2022 (pfolio.io). FinAgent (arXiv 2402.18485, KDD 2024) added multimodality (text, numerical series, K-line charts) and double reflection; its abstract headline —92.27% return— comes from one dataset out of six, the exact pattern of best-result selection (arXiv.org). TradingAgents (arXiv 2412.20138; repo with more than 65,000 stars) simulates a complete firm —four analysts, bull/bear researchers in debate, trader, risk committee, fund manager— and reports CR 26.62% and Sharpe 8.21 on AAPL over a three-month window in 2024, with maximum drawdowns of 0.91% (🦜️🔗 LangChain). The authors themselves add an unusually honest self-critical footnote: the Sharpe exceeds their expected empirical range (SR > 3 "excellent") and they attribute it to the absence of corrections in the window; with 11 LLM calls and more than 20 tool calls per prediction, they could only test three months (langchain.js).

StockAgent (arXiv 2407.18957, ACM TIST) makes the methodological contribution of naming the leakage right in the abstract: it is a simulation with an order book and investor personalities explicitly designed to prevent the model from exploiting prior knowledge of the test period (Springer). FinRobot (arXiv 2405.14767, AI4Finance) is a four-layer analysis platform with a multi-LLM Smart Scheduler, not a trading system; its value is architectural, and its authors criticize the field's obsession with backtesting over process-oriented approaches (rdrr.io). FinCon (arXiv 2407.06567, NeurIPS 2024 main track) implements a manager–analyst hierarchy with conceptual verbal reinforcement (CVRF) and risk gating with CVaR, reporting CR 113.84% and Sharpe 3.269 on a TSLA/MSFT/PFE portfolio, and being the only agent with a positive return in the TSLA-2022 stress; it also admits hallucinations in memory management (Github). MarketSenseAI (arXiv 2401.03737, ESWA) reports beating the S&P 100 by 13 points with returns of up to 40% in 2023 —a year inside GPT-4's knowledge window— and declares a conflict of interest: the authors market the framework as a startup (skfolio.org). HedgeAgents (arXiv 2502.13165, WWW 2025 Companion) reports 70% annualized and 400% over three years, and in the same abstract admits that conventional LLM agents lose ~−20% in fast drawdowns (arXiv.org).

Table 9.1 — Published LLM trading agents: architecture, reported result, and documented biases

9.2 Sentiment as a quantitative signal

9.2.1 Lopez-Lira & Tang, the PEAD evidence, and the integration pipeline

The LLM signal with the best evidence is not an agent architecture but a headline classifier. Lopez-Lira and Tang (arXiv 2304.07619) scored with ChatGPT every common-stock headline from CRSP between October 2021 and December 2022 —after ChatGPT's cutoff, with explicit contamination control— and built a self-financing long-short strategy: Sharpe 3.8 with GPT-4 in the paper's v4, degraded to 3.28 in v5 (and GPT-3.5 from 3.1 to 1.79), versus negative Sharpes for GPT-1/2 and BERT (arXiv.org). The cumulative return exceeds 500%, but without transaction costs, and with a revealing asymmetry: the short leg contributes 29 bps/day versus 9 bps/day for the long leg —the signal is above all a bad-news detector, concentrated in small-caps where limits to arbitrage make the correction expensive (arXiv.org). The fact that the numbers change between versions of the same paper is itself a methodological lesson on the fragility of backtests. The PEAD evidence is complementary and of solid pedigree: contextual features from earnings calls improve the per-trade return of a long-short portfolio by 53–354 bps out-of-sample on a point-in-time S&P 500 universe 2010–2022 (fgv.br), although honest counterexamples exist where text alone does not beat the baseline (Advisor Perspectives).

Professional integration treats the LLM score as one more factor, not as an order. The reference pipeline: (1) the LLM emits continuous scores (preferable to discrete labels, because they preserve intensity); (2) temporal aggregation with exponential decay —the documented predictability is 1–5 days— and novelty weighting, discounting duplicate or republished headlines; (3) winsorization and cross-sectional z-score by date; (4) neutralization against sector and risk factors (market, size, momentum), keeping the residual; (5) IC-weighted combination with the existing alpha; (6) execution with modeled ingestion lag, because news timestamps record publication, not availability to the pipeline (horizontrading.io).

where $s_{i,t}$ is the aggregated news score of asset $i$ and $h$ is the signal's half-life (1–3 days in documented practice). Neutralization is implemented as a daily cross-sectional regression of the z-score on sector dummies and factor exposures, using the residual as the final signal.

# --- LLM sentiment signal pipeline → quantitative factor (pandas) ---
import numpy as np, pandas as pd

def agregar_decay(scores: pd.DataFrame, halflife: float = 2.0) -> pd.DataFrame:
    """scores: DataFrame [dates x tickers] of daily LLM scores (NaN = no news).
    Exponential decay with `halflife`-day half-life; only days with news carry weight."""
    lam = np.log(2) / halflife
    out = scores.copy().astype(float)
    for t in range(1, len(out)):
        out.iloc[t] = out.iloc[t].fillna(0) + np.exp(-lam) * out.iloc[t-1].fillna(0)
    return out

def zscore_cross_sectional(sig: pd.DataFrame, win: float = 0.01) -> pd.DataFrame:
    lo, hi = sig.quantile(win, axis=1), sig.quantile(1 - win, axis=1)
    w = sig.clip(lo, hi, axis=0)                      # winsorized by date
    med = w.median(axis=1)                            # cross-sectional median
    mad = w.sub(med, axis=0).abs().median(axis=1)     # cross-sectional MAD
    return w.sub(med, axis=0).div(1.4826 * mad.replace(0, np.nan), axis=0)

def neutralizar(z: pd.DataFrame, sectores: pd.Series) -> pd.DataFrame:
    """Residual of z on sector dummies, by date (cross-sectional OLS)."""
    D = pd.get_dummies(sectores)
    def resid(fila):
        y = fila.dropna(); Xd = D.loc[y.index]
        beta, *_ = np.linalg.lstsq(Xd.values, y.values, rcond=None)
        return y - Xd.values @ beta
    return z.apply(resid, axis=1)

9.3 Backtesting to institutional standards

9.3.1 Frameworks 2026, traps, and statistical correction

The first architectural decision is not which library but which family: vectorized engines compute the signal over complete arrays and simulate approximate fills at bar close; event-driven engines process events ordered in time and the strategy reacts —high execution fidelity, lower speed (CSDN博客). The 2026 decision rule: massive signal discovery in vectorbt/vectorbtpro; microstructure validation and the research→live path in nautilus_trader or QuantConnect LEAN; cross-sectional factors in zipline-reloaded (Pipeline API); weight-based allocation in bt (ar5iv).

Table 9.2 — Backtesting frameworks 2026: architecture, speed, backtest→live parity, and maintenance

The four canonical traps each have their detector. Look-ahead: using information not available at decision time —including the LLM case, where the leak is in the model's weights— (臺灣博碩士論文知識加值系統). Survivorship: universes without delistings, with estimated annual distortions between 0.1% and 0.9% and pre-inclusion bias when using current index constituents (臺灣博碩士論文知識加值系統). Data snooping / multiple testing: trying many configurations on the same data and reporting the best; the reported Sharpe is almost never a single hypothesis but the maximum of many noisy estimates (RDocumentation). Selection/reporting: McLean and Pontiff documented average degradations of −26% out-of-sample and −58% post-publication in published predictors (Github).

The canonical statistical correction is the Probabilistic Sharpe Ratio, which adjusts the estimated Sharpe for non-normality and sample length with Lo's standard error:

where $T$ is the number of observations, $\hat{\gamma}_3$ the sample skewness, $\hat{\gamma}_4$ the kurtosis (3 = Normal), and $\Phi$ the Normal CDF. The Deflated Sharpe Ratio replaces the benchmark $SR^*$ with the expected maximum under the null hypothesis of $N$ attempted strategies without skill:

with $\gamma_{EM} \approx 0{,}5772$ (Euler–Mascheroni constant) and $\mathbb{V}[\cdot]$ the variance of the Sharpes of the $N$ tested configurations. Then $\widehat{DSR} = \widehat{PSR}(SR_0)$, with operational rule DSR > 0.95: even admitting how many strategies were tried, there is a 95%+ probability that the edge is real (PyPI). Complements: PBO (Probability of Backtest Overfitting) via CSCV —splitting the P&L matrix into $S$ blocks, taking the $\binom{S}{S/2}$ half-IS/half-OOS combinations, selecting the IS-optimal configuration and measuring its OOS rank— is the probability that the best in-sample configuration falls below the median out-of-sample, and tends to 1 as $N$ grows regardless of whether genuine predictive power exists (PyPI). The Harvey–Liu haircut converts the Sharpe to a t-ratio, adjusts the p-value for the number of tests (Bonferroni, Holm, BHY), and converts back; it is markedly non-linear —the "50% haircut" rule is an error; Sharpe 1.0 with 120 months and 100 tests becomes 0.438 (BHY) or 0.232 (Bonferroni)— (博客园):

# --- Deflated Sharpe Ratio (Bailey & López de Prado 2014) ---
import numpy as np
from scipy.stats import norm

def probabilistic_sharpe(sr, sr_star, n_obs, skew, kurt):
    se = np.sqrt(1 - skew * sr + (kurt - 1) / 4 * sr**2)   # Lo (2002) standard error
    return norm.cdf((sr - sr_star) * np.sqrt(n_obs - 1) / se)

def expected_max_sharpe(n_trials, var_sr, euler=0.5772156649):
    return np.sqrt(var_sr) * ((1 - euler) * norm.ppf(1 - 1 / n_trials)
                              + euler * norm.ppf(1 - 1 / (n_trials * np.e)))

def deflated_sharpe(sr, n_obs, skew, kurt, n_trials, var_sr):
    return probabilistic_sharpe(sr, expected_max_sharpe(n_trials, var_sr),
                                n_obs, skew, kurt)

# Sharpe 1.2 annual, 3 years daily, 100 trials, skew=-1, kurt=6
dsr = deflated_sharpe(1.2/np.sqrt(252), 756, -1.0, 6.0, 100, 0.10/252)
print(f"DSR = {dsr:.3f} -> {'edge plausible' if dsr > 0.95 else 'indistinguible de ruido'}")

# --- Backtest with costs in vectorbt (real API) ---
import vectorbt as vbt

data  = vbt.YFData.download("AAPL", start="2020-01-01", end="2023-01-01")
close = data.get("Close")
fast, slow = vbt.MA.run(close, 20), vbt.MA.run(close, 50)

pf = vbt.Portfolio.from_signals(
    close,
    entries=fast.ma_crossed_above(slow),
    exits=fast.ma_crossed_below(slow),
    init_cash=10_000,
    fees=0.001,        # 0.1% commission per side
    slippage=0.001,    # 0.1% slippage per side
    freq='D',
)
print(pf.stats())      # compare with fees=0, slippage=0 to quantify the drag

9.3.2 Realistic transaction costs and capacity

The canonical measure of execution cost is the implementation shortfall (Perold 1988): the gap between the price at order arrival and the price actually achieved (LangChain):

The most forgotten component in the LLM papers is borrow on shorts: the short seller pays an annual rate on the position's value, with daily cost

where $r_b$ is the annual borrow fee. Real magnitudes: a general collateral ETF like SMH was quoted at 0.7% annual (IBKR, Jul-2026); hard to borrow names exceed 10–100% annual and make high-turnover short strategies unviable (CSDN博客). Recalling that the short leg contributed triple the long leg in Lopez-Lira & Tang (arXiv.org), omitting the borrow is not a detail: it is omitting the signal's main cost.

The theory-practice bridge model is Almgren–Chriss (2001): liquidating $X$ shares in $[0,T]$ with $N$ steps of length $\tau = T/N$, price with permanent impact $\gamma$ and temporary $\eta$, minimizing $E(x) + \lambda V(x)$ with

The closed-form solution is a hyperbolic trajectory with urgency parameter $\kappa$ (Blockchain News):

Two instructive limits: with $\kappa T \ll 1$ the trajectory degenerates into linear TWAP (minimizes impact); with $\kappa T \gg 1$ the liquidation is exponential front-loaded with half-life $\theta = 1/\kappa$. And since the efficient frontier is differentiable at the minimum-cost point, the naive strategy ($\lambda = 0$) is never optimal for a risk-averse trader: significant variance reductions cost only marginal increments of expected cost (Github).

On impact, the dominant empirical law for metaorders is the square-root law (🦜️🔗 LangChain):

with $\sigma_D$ the daily volatility, $Q$ the metaorder volume, and $V_D$ the average daily volume —mnemonic rule: it costs approximately one day of volatility to trade one day of volume (🦜️🔗 LangChain). Impact is concave in $Q$: high-turnover strategies on small positions barely suffer it, but any institutional scaling turns it into the strategy's capacity limit. The aggregate arithmetic is lethal for high-turnover daily signals:

With full daily turnover, even 5 bps per side means ~25 percentage points of annual drag ($250 \times 2 \times 5\,\text{bps}$): a gross Sharpe of 2.0 collapses below 0.5. The following chart illustrates the effect on a synthetic sentiment signal: the gross version shows a Sharpe of 1.44; the net version at 6 bps per side with 35% daily turnover falls to 0.67.

# --- Almgren-Chriss: optimal liquidation trajectory ---
import numpy as np

def almgren_chriss_trajectory(X, T, N, sigma, eta, lam):
    tau = T / N
    kappa = np.sqrt(lam * sigma**2 / eta)           # urgency κ ≈ sqrt(λσ²/η)
    t = np.arange(N + 1) * tau
    x = X * np.sinh(kappa * (T - t)) / np.sinh(kappa * T)
    return t, x, -np.diff(x), kappa                 # sales n_j per interval

t, x, n, k = almgren_chriss_trajectory(1e6, T=1.0, N=390,
                                       sigma=0.02, eta=1e-7, lam=1e-6)
print(f"Vida media de la liquidación θ=1/κ = {1/k:.3f} días")

# --- Total cost in bps: spread + sqrt impact + borrow ---
def coste_total_bps(Q, V_dia, sigma_dia, spread_bps, comision_bps,
                    Y=0.6, r_borrow=0.0, dias_corto=0, direccion=1):
    impacto = Y * sigma_dia * np.sqrt(Q / V_dia) * 1e4
    borrow  = r_borrow / 360 * dias_corto * 1e4 if direccion < 0 else 0.0
    return 0.5 * spread_bps + impacto + comision_bps + borrow

FINSABER provides the empirical verification at scale: in systematic backtests over two decades and more than 100 symbols, the advantages of timing-based LLM strategies deteriorate significantly relative to the reported narrow windows, and the regime analysis shows the perverse pattern of excess caution in bull markets and excess aggression in bear markets (arXiv.org). The following table condenses the validation checklist that an institutional allocation committee demands before real capital.

Table 9.3 — Institutional strategy-validation checklist before allocating real capital

9.4 From signal to governed execution

9.4.1 Brokers, protocols, and the RiskGateway + HITL architecture

In the retail-prosumer world, Alpaca offers a commission-free REST/WebSocket API with a paper environment identical to live, 4x intraday / 2x overnight buying power, short selling, and advanced orders (bracket, OCO, trailing stop); the standard idempotency practice is the unique client_order_id, which prevents duplicating orders if the agent retries after a timeout (LangChain). Interactive Brokers is the institutional-retail reference (global multi-asset access, public borrow feed (CSDN博客) ), and the current Python stack is ib_async, the maintained successor of ib_insync (archived) (LangChain):

from ib_async import *

ib = IB()
ib.connect('127.0.0.1', 7497, clientId=1)          # 7497 = TWS paper

contract = Stock('TSLA', 'SMART', 'USD')
parent = LimitOrder('BUY', 100, 250.00)
parent.orderId = ib.client.getReqId(); parent.transmit = False
stop = StopOrder('SELL', 100, 240.00)
stop.orderId = ib.client.getReqId(); stop.parentId = parent.orderId; stop.transmit = False
tp = LimitOrder('SELL', 100, 260.00)
tp.orderId = ib.client.getReqId(); tp.parentId = parent.orderId; tp.transmit = True

for o in (parent, stop, tp):
    ib.placeOrder(contract, o)

In the institutional world the strategy does not talk to the broker: it talks to the OMS (order lifecycle, compliance, allocation, audit —the system of record), which routes to the EMS (smart order routing, VWAP/TWAP/IS algos, TCA), which executes against venues via FIX (langchain.com). FIX 4.2 remains massively deployed for equities: SOH-delimited tag=value messages, with 35=D NewOrderSingle, 35=8 ExecutionReport, 35=F OrderCancelRequest and 35=G Cancel/Replace, session with Logon (35=A) and Heartbeat (35=0), and strictly monotonic sequence numbers (tag 34) —a gap triggers ResendRequest (35=2)— (arXiv.org):

8=FIX.4.2|49=CLIENT|56=EXCHANGE|34=1|52=20260729-13:00:00|35=D|11=LLM-7F3A|55=AAPL|54=1|38=500|40=2|44=212.50|10=123|

The regulatory floor is SEC Rule 15c3-5 (Market Access Rule): after the Flash Crash, broker-dealers must apply automated and non-delegable pre-trade controls —capital and credit limits, maximum order size, prevention of erroneous orders and, for DMA, real-time monitoring and a kill switch— (skywork.ai). The lesson for LLM systems is direct: the regulation assumes the algorithm will fail and mandates that the guardrail be independent of the order generator. The architecture taught in this course implements exactly that principle: the agent never has trading credentials; it has a submit_order tool wrapped in a deterministic RiskGateway (limits in code, never in the prompt), a HITL interrupt for orders above a threshold, and a kill switch that never asks for permission (🦜️🔗 LangChain).

class RiskGateway:
    """Deterministic pre-trade risk layer (15c3-5 style)."""
    def __init__(self, api, max_notional=25_000, max_daily_loss=5_000, max_orders_min=10):
        self.api, self.max_notional = api, max_notional
        self.max_daily_loss, self.max_orders_min = max_daily_loss, max_orders_min
        self.kill = False                      # global kill switch

    def check(self, symbol, qty, price, side):
        if self.kill:
            raise RiskBlock("KILL-SWITCH ACTIVO: toda nueva orden bloqueada")
        if qty * price > self.max_notional:
            raise RiskBlock(f"Notional {qty*price:.0f} > límite {self.max_notional}")
        if self.daily_realized_pnl() < -self.max_daily_loss:
            self.kill = True                   # auto-kill on daily loss
            self.cancel_and_flatten()
            raise RiskBlock("Límite de pérdida diaria: posiciones cerradas")
        if self.orders_last_minute() >= self.max_orders_min:
            raise RiskBlock("Rate limit interno excedido (anti runaway-loop)")
        if side == "sell" and not self.is_shortable(symbol):
            raise RiskBlock(f"{symbol} no shortable / borrow no disponible")
        return True

class RiskBlock(Exception): pass

The LangChain integration uses HumanInTheLoopMiddleware: it interrupts execution at configured tool calls, persists state with a checkpointer, and admits approve/edit/reject/respond decisions, with conditional interrupts (when) to require human approval only above a given notional (🦜️🔗 LangChain):

from langchain.agents import create_agent
from langchain.agents.middleware import HumanInTheLoopMiddleware, ToolCallRequest

def big_order(request: ToolCallRequest) -> bool:
    a = request.tool_call["args"]
    return a.get("qty", 0) * a.get("limit_price", 0) > 10_000   # conditional HITL

agent = create_agent(
    model="gpt-5.5",
    tools=[get_market_data, compute_signal, submit_order, cancel_all_orders],
    middleware=[HumanInTheLoopMiddleware(
        interrupt_on={
            "submit_order": {"allowed_decisions": ["approve", "reject"], "when": big_order},
            "cancel_all_orders": False,      # the kill switch NEVER asks for permission
            "get_market_data": False,
        },
        description_prefix="ORDEN PENDIENTE DE APROBACIÓN",
    )],
    checkpointer=AsyncPostgresSaver(...),      # mandatory for HITL
)

The kill-switch triggers are concrete and monitorable: daily loss, deviation of realized slippage from that assumed in the backtest, anomalous latency, data errors (gaps, future timestamps), LULD halts, or the operator's manual button. The FIX equivalent is a mass 35=F followed by session logout (35=5). The module's honest closing metric is therefore unique: the Sharpe net of costs, deflated by the N trials, validated post-cutoff, and executed under 15c3-5-style controls. Everything else is backtest marketing.

Exercises

1. Sentiment-signal backtest with vectorbt. Build a synthetic signal of LLM scores over 50 tickers (or download real headlines and score them with a model via batch API), aggregate it with a 2-day half-life decay, and turn it into long-only entries/exits of the top quintile. Run Portfolio.from_signals with fees=0, slippage=0 and with fees=0.001, slippage=0.001; report gross Sharpe, net Sharpe, and implied turnover. Repeat with fees=0.003 and discuss at what friction level the signal ceases to be viable.
2. DSR calculation. Using the module's probabilistic_sharpe, expected_max_sharpe, and deflated_sharpe functions, take the net Sharpe from exercise 1 and compute the DSR assuming N ∈ {1, 10, 100, 1,000} trials, with the sample skewness and kurtosis of your returns. From what N does the DSR fall below 0.95? Compare with the Harvey–Liu haircut for 100 tests.
3. Almgren–Chriss cost simulation. Liquidate 1,000,000 shares in one day (390 1-minute intervals) with σ = 0.02, η = 10⁻⁷ and λ ∈ {10⁻⁷, 10⁻⁶, 10⁻⁵}. Plot the three trajectories $x_j$, compute the half-life θ = 1/κ of each, and numerically verify the TWAP limit when κT ≪ 1. Then estimate the impact cost with the square-root law for Q/V_D ∈ {1%, 5%, 25%} and Y = 0.6.
4. Turnover arithmetic. A sentiment strategy turns over 100% of the portfolio daily, half in shorts on small-caps with a 12% annual borrow. Compute the total annual drag with 8 bps per side of friction (spread + commission) and the daily borrow of the short leg. What minimum gross Sharpe does the signal need for a net Sharpe of 1.0, assuming 15% annual volatility?
5. RiskGateway design. Extend the module's RiskGateway class with: (a) a per-sector concentration limit; (b) a volume participation limit (≤ 10% of the symbol's ADV, estimated with the square-root law); (c) a symbol whitelist; (d) auto-kill if the average realized slippage of the last 50 orders exceeds 2× that assumed in the backtest. Write the unit tests proving that no order can reach the broker bypassing a limit, including the case of an agent in a loop issuing 200 orders per minute.
6. Paper audit. Choose one of the papers from the table in section 9.1.1 and evaluate it against the 10 points of the institutional checklist: identify which points it satisfies, which it violates, and what minimum additional experiment would be needed for its results to be admissible before an allocation committee. Present your conclusions in one page in the format evidence → rule → conclusion.

Module 9 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; prices and market figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.