Market and documentary data for LLM systems

Module 3 gave agents orchestration: state graphs, tools, persistence and human interrupts. But an agent is only as good as the data its tools serve it. This module opens Block II (understand) by building that foundation: which market and documentary sources exist in 2026, which license is needed to feed an LLM with them, how they are consumed programmatically, and how they are organized in a point-in-time architecture that the RAG of Module 5 and the calculations of Modules 7 and 8 can audit.

4.1 The source landscape in 2026

4.1.1 Institutional vs accessible

The financial data market in 2026 is stratified into two worlds that barely touch. In the first, the institutional terminals: as of July 2026 Bloomberg Terminal lists at around $31,980 per seat per year ($28,320 in multi-seat configuration), after the 6.5% increase applied to renewals since January 2025 (Github). LSEG Workspace (successor to Refinitiv Eikon) sits above $22,000 per seat in its full version, with a reduced edition close to $4,000 per year (Github). FactSet, with a historical list price of $12,000, shows real contracts averaging close to $45,000 per year according to buyer transactional data (arXiv.org). S&P Capital IQ Pro moves in the $12,000–$20,000 per seat band (arXiv.org). These figures are industry estimates —none of the four vendors publishes price lists—, but they converge across two independent sources and are accepted as a budgetary reference (arXiv.org).

The decisive fact for an LLM systems engineer is different: the terminal subscription does not include programmatic access. Feeding an automated pipeline —and therefore a LangChain agent— with Bloomberg data requires B-PIPE ($2,000–$3,000/month) or Data License ($5,000–$50,000+/year); extracting terminal content into a vector store or a prompt without that license breaches the contract (arXiv.org).

In the second world, developer-first APIs have reached a coverage that five years ago was exclusive to the buy-side. The structural milestone is the renaming of Polygon.io as Massive.com, effective October 30, 2025: the api.polygon.io keys and endpoints remain operational in parallel with api.massive.com, and CME/CBOT/COMEX/NYMEX futures, Benzinga data and an llms.txt file in the documentation for consumption by AI assistants were added (Quant Memo). Its plans run from free EOD to $199/month real time and $2,000/month Business with a commercial license (Picture Perfect Portfolios). Alpha Vantage gives 25 free requests per day and premium tiers from $49.99 to $249.99/month scaled by req/min, with the caveat that U.S. real time requires an additional entitlement (MDPI). Tiingo covers 30+ years of EOD for $30/month ($50/month for the commercial-use Business plan) (A.L. Capital Advisory). Databento distributes direct exchange data (not SIP) with 6.1-microsecond normalization and MBO/MBP-10 schemas in nanoseconds, from $199/month (pfolio.io). Nasdaq Data Link operates as a marketplace with free economic datasets and à-la-carte premium ones ($50–$500+/month) (Entropy Method: Solving RL Without Gradients).

Table 4.1 — Institutional vendors vs accessible APIs (prices as of July 2026)

4.1.2 Licenses for LLM use: non-display, AI addenda and the yfinance case

Three contractual mechanisms determine whether a data point may enter a production LLM system.

The first is the exchange non-display fee: the consumption of market data by machines —algorithms, servers and, explicitly, LLM agents that generate signals— rather than by a human looking at a screen. The official NYSE guide sets $3,000/month for category 1 non-display on NYSE Trades and $5,000/month on the NYSE National Integrated Feed, with a category 3 that covers trading signals and ATSs (alanhou.org). A LangChain agent whose tool queries a real-time feed falls, in the exchange's eyes, within this definition: it is the most frequent budget surprise in AI projects with market data (alanhou.org).

The second mechanism is the vendors' AI addenda. Bloomberg, LSEG, FactSet and S&P contracts license data for internal use by licensed users; sending that content to a third-party LLM (where it may remain in logs) or to a vector store accessible by unlicensed staff enters prohibited territory absent an explicit "AI/derived data use" addendum (arXiv.org). The counterpart: the accessible vendors formalize the reverse channel —official MCP servers from Alpha Vantage and Benzinga, llms.txt at Massive— which turns their APIs into agent-discoverable sources (Quant Memo).

The third is the negative litmus test: yfinance. Yahoo shut down its public API in 2017; the library scrapes unofficial endpoints, saturates under load (~2,000 estimated requests/hour before the 429 error), its terms of service limit it to personal use and it lacks any SLA (Ask Everything).

4.2 SEC EDGAR as a programmatic source

4.2.1 Real endpoints: submissions, companyfacts, companyconcept, frames and full-text search

EDGAR is the only major financial source that combines three properties: free, no API key, and direct legal value (filings are regulatory documents). In return, the SEC asks for a descriptive User-Agent with a contact email and a maximum of 10 requests per second; for massive backfills it recommends the nightly ZIPs rather than hammering per-company endpoints (arXiv.org). The first step of any pipeline is to cache https://www.sec.gov/files/company_tickers.json, which resolves ticker → CIK locally (ResearchGate).

Table 4.2 — SEC EDGAR endpoints (verified as of July 2026)

The following snippet implements access to submissions while respecting SEC policy:

import time
import requests

SEC_HEADERS = {"User-Agent": "QuantResearch [email protected]"}  # mandatory
RATE_LIMIT_S = 0.12  # ~8 req/s, margin below the official 10 req/s maximum

def get_submissions(cik: int) -> dict:
    """Complete filing history of a company (columnar parallel arrays)."""
    url = f"https://data.sec.gov/submissions/CIK{cik:010d}.json"
    resp = requests.get(url, headers=SEC_HEADERS, timeout=20)
    resp.raise_for_status()
    time.sleep(RATE_LIMIT_S)
    return resp.json()

subs = get_submissions(320193)  # Apple
recent = subs["filings"]["recent"]
# form[i], filingDate[i] and accessionNumber[i] share an index (columnar format)
primer_10k = next(
    i for i, f in enumerate(recent["form"]) if f == "10-K"
)
print(recent["form"][primer_10k], recent["filingDate"][primer_10k],
      recent["accessionNumber"][primer_10k])
# Document URL: https://www.sec.gov/Archives/edgar/data/
#   {cik_no_zeros}/{accession_no_dashes}/{primaryDocument}

And this is the LangChain tool pattern over companyconcept —the first real tool of the course, which the agents of modules 3 and 7 will consume directly:

from langchain.tools import tool
import requests

@tool
def get_xbrl_concept(ticker: str, concept: str = "NetIncomeLoss") -> str:
    """Returns the history of a us-gaap XBRL concept for a company (SEC EDGAR).

    Each fact includes period (end), value (val), form (form) and,
    critically, the publication date (filed) for point-in-time joins.
    """
    cik = resolve_cik(ticker)  # cached lookup from company_tickers.json
    url = (f"https://data.sec.gov/api/xbrl/companyconcept/"
           f"CIK{cik:010d}/us-gaap/{concept}.json")
    r = requests.get(url, headers={"User-Agent": "QuantApp [email protected]"},
                     timeout=20)
    r.raise_for_status()
    units = r.json()["units"]
    return str(units)[:4000]  # truncate: token and cost control

Best practices separate a production tool from a demo: a precise docstring (the LLM chooses the tool by its description), input validation with pydantic, output truncation, compact JSON instead of HTML, actionable errors and retries with backoff in the face of 429s from GDELT, Yahoo or Alpha Vantage (Apify).

4.3 News, sentiment and alternative data

4.3.1 GDELT, Benzinga, Alpha Vantage; MCP; Form 4 and 13F; deduplication and alpha decay

The news ecosystem has a well-known price cliff: NewsAPI.org is free only in development (100 req/day, 24 h delay) and jumps to $449/month on Business —and even paid tiers do not deliver full article bodies (Github). The structural alternative is GDELT, free and keyless: the DOC 2.0 API (https://api.gdeltproject.org/api/v2/doc/doc) monitors press, web and broadcast in more than 100 languages at a 15-minute cadence, with ArtList, TimelineVol (breaking news detection by volume) and ToneChart modes, 250 records per query and aggressive rate limits that demand caching (Fundamentals Hub). In affordable paid territory, Benzinga distributes its real-time wire, ratings with history since 2012, earnings and guidance via Massive for $99/month per package, and launched its MCP server in November 2025 (templates). In the institutional segment, RavenPack processes more than 40,000 sources with more than twenty sentiment indicators and 20+ years of archive for an estimated $50,000–$100,000+ per year (medium confidence: no public price), without covering social media (TL;DR Filing).

Table 4.3 — News and sentiment sources (as of July 2026)

Real GDELT query:

import requests

def gdelt_artlist(query: str, timespan: str = "1week", maxrecords: int = 50) -> list[dict]:
    """Recent articles from GDELT DOC 2.0 (no API key; respect ~6 s between calls)."""
    params = {"query": query, "mode": "artlist", "format": "json",
              "timespan": timespan, "maxrecords": maxrecords}
    r = requests.get("https://api.gdeltproject.org/api/v2/doc/doc",
                     params=params, timeout=30)
    r.raise_for_status()
    arts = r.json().get("articles", [])
    seen, out = set(), []
    for a in arts:  # dedupe by URL/title: wire agencies republish the same news item
        key = a.get("url", a.get("title"))
        if key not in seen:
            seen.add(key)
            out.append(a)
    return out

Two regulatory sources deserve separate mention. The insider Form 4 is filed within two business days of the transaction: it is the freshest legally mandated signal in the U.S. market —insider behavior appears there before it does in 13F, research notes or the press—, with free XML on EDGAR and APIs such as sec-api.io that structure it (Apify). The 13F, by contrast, reflects quarter-end positions published with a 45-day delay, long-only in 13(f) securities —no shorts and most derivatives excluded—: historical context, never current positioning (TL;DR Filing).

On exploiting sentiment, two empirical caveats. The first is deduplication: wire agencies republish the same story dozens of times, and counting volume without URL/title dedupe inflates the signal (orthogonal.info). The second is alpha decay: the performance of a sentiment signal decays with its adoption, forcing continuous regeneration of variants (orthogonal.info). A common working model is exponential decay

where $\lambda$ grows with the diffusion of the signal; the half-life $t_{1/2}$ of public sentiment signals is measured in months, not years (orthogonal.info).

4.4 Point-in-time pipeline architecture

4.4.1 The golden rule of data: "text to the vector store, numbers to SQL"

Everything above converges on a single design decision, the one that organizes the course's reference architecture (structural insight I5) —the golden rule of data, sister of the course's golden rule from §1.2.2, applied to the storage layer—: unstructured text goes to the vector store; numbers go to relational/columnar databases and are queried with deterministic tools. Embeddings are semantic retrieval, not a store of exact figures: an LLM that "remembers" a revenue figure via vector similarity may retrieve the wrong period or distort the number, whereas a SQL query over XBRL returns the value with its filed and its accession number (Apify). The concrete partition: to the vector store go 10-K/10-Q sections (Item 1A, Item 7), transcripts and full-text news, with {ticker, form, filing_date, section} metadata for pre-retrieval filters; to SQL/columnar go OHLCV, XBRL facts, 13F holdings, Form 4 transactions and estimates; to Redis, the hot state (Apify). The stack is simplified if the vector store is pgvector inside Postgres/TimescaleDB itself (Apify).

For numeric transport, the standard pattern is: vendor WebSocket → Python producer → Kafka topics partitioned by symbol (preserves per-asset ordering) → consumers → stores, using the exchange timestamp as event-time so that jitter does not reorder calculations; Kafka's replayability allows regenerating features without re-downloading from the vendor (博客园). In serving, TimescaleDB (Postgres) provides hypertables and continuous aggregates that pre-compute OHLCV roughly 100 times faster than views over raw ticks, with 90–95% compression (Apify):

CREATE TABLE tick_data (time TIMESTAMPTZ NOT NULL, symbol TEXT NOT NULL,
                        bid NUMERIC(20,8) NOT NULL, ask NUMERIC(20,8) NOT NULL);
SELECT create_hypertable('tick_data', 'time');

CREATE MATERIALIZED VIEW tick_1m_view WITH (timescaledb.continuous,
  timescaledb.materialized_only = false) AS
SELECT time_bucket('1 minute', time) AS time, symbol,
  FIRST((bid+ask)/2, time) AS open, MAX((bid+ask)/2) AS high,
  MIN((bid+ask)/2) AS low,  LAST((bid+ask)/2, time) AS close
FROM tick_data GROUP BY time_bucket('1 minute', time), symbol;

SELECT add_retention_policy('tick_data', INTERVAL '30 days');

For research and backtesting, the de facto pattern is partitioned Parquet (year=/month=/symbol=) queried with in-process DuckDB —columnar reads with predicate pushdown at zero marginal cost—, reserving TimescaleDB for serving (Entropy Method: Solving RL Without Gradients). The piece that closes the temporal loop is Feast, a feature store with an offline store (training) and an online store (serving) whose point-in-time join generates datasets with exactly the anti-look-ahead semantics a backtest requires: each feature is joined with the latest value available at the label's instant, not the latest known today (Accuweather Api).

4.4.2 Data traps: look-ahead, survivorship and point-in-time universes

The availability rule that defines point-in-time (PIT) is simple to state and costly to implement: a fact $f$ may only feed the decision at simulation time $t$ if its effective publication date is earlier than or equal to $t$:

There are three common violations, with distinct antidotes (orthogonal.info). First, restatements: using the corrected value of a fundamental before the correction was public. The canonical case is Plug Power: diluted loss for fiscal year 2018 was originally reported at $0,36 per share and was restated to $0,39 in a filing dated May 14, 2021; any 2019 or 2020 simulation using $0,39 has leaked two years of future (orthogonal.info). The SEC's XBRL API gives filed per fact, but shows the latest version of each period —it does not preserve the pre-restatement value—; strict PIT requires Compustat Point-in-Time, Bloomberg's PIT mode, Norgate, engines such as QuantConnect LEAN (fundamentals "as of" the publication date) or a custom versioning layer over accn + filed (orthogonal.info).

The third violation is survivorship bias: a universe built with today's living tickers has removed the bankrupt, acquired and delisted ones, and never sees the worst outcomes. The magnitude is not minor: in a Morningstar study, 58% of the funds existing in 1999 no longer existed in 2019 (Accuweather Api). And there is a structural limitation of the cheap stack: APIs such as yfinance or Alpha Vantage do not serve delisted stocks, so a serious long-history backtest requires sources with delistings (Norgate, CRSP) and point-in-time universes that reconstruct the S&P 500 constituents as of each date, not today's (Accuweather Api). Checklist complements: shift every signal by one bar (anything computed with bar $N$ cannot be traded until the open of $N+1$), exchange versus reception timestamps, halt-gap detection, XBRL null validation and news dedupe (orthogonal.info).

With this module, the reader has the sources (4.1–4.3) and the temporally honest architecture (4.4) that the later agents will consume: Module 5 (RAG) will be built on the documentary branch of the diagram, and Modules 7 and 8 on the numeric branch and its PIT joins.

Exercises

1. EDGAR explorer. Write a script that, given a ticker, resolves the CIK with company_tickers.json, downloads submissions and lists the last ten 10-K/10-Q filings with the document URL built from accessionNumber and primaryDocument. Respect 10 req/s and add exponential backoff on a 429. Extend it to measure the observed reporting lag: days between periodOfReport and filingDate for the 10-Ks of five companies.

2. Point-in-time XBRL tool. Turn get_xbrl_concept into get_concept_asof(ticker, concept, as_of_date), filtering facts with filed <= as_of_date and returning the latest available one. Verify the Plug Power trap: query EarningsPerShareDiluted for PLUG and show that a naive query (latest FY2018 value) and the PIT query with as_of_date = 2020-01-01 return different values.

3. Pipeline design. Draw (Mermaid or pseudocode) the pipeline of a strategy that trades on Benzinga news and confirms with EDGAR fundamentals: assign each component to Kafka, TimescaleDB, Parquet/DuckDB, Feast or vector store; justify for each data element whether it is "text" or "numbers"; annotate where an error would introduce look-ahead. Estimate the monthly cost as of July 2026 using Table 4.1, including the applicable non-display fee.

4. License analysis. A PM proposes feeding a research agent with (a) yfinance headlines, (b) quotes from the Massive Starter plan at $29/month and (c) transcripts obtained by scraping. For each source, identify the license or ToS conflict, propose the minimum compliant alternative (with price as of July 2026) and draft the justification paragraph for compliance.

5. Deduplication and alpha decay. Using gdelt_artlist, collect three days of articles on five semiconductor tickers. Quantify the duplicate rate before and after dedupe and discuss how it contaminates a "news volume" signal. Estimate the half-life under $\alpha_t = \alpha_0 e^{-\lambda t}$ if ex-ante Sharpe falls 50% in eight months.

Module 4 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; market prices and figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.