Market and documentary data for LLM systems

Learning objectives

Upon completing this module, the reader will be able to:

• Distinguish the families of financial data sources in 2026 (institutional terminals, enterprise feeds, developer-first APIs, public regulatory sources) and size their real cost, including programmatic access and exchange non-display fees.
• Assess whether a source is licensed to feed prompts and vector stores in commercial production, and rule out those that are not (the yfinance case).
• Consume SEC EDGAR through its real JSON endpoints (submissions, companyfacts, companyconcept, frames, full-text search), respecting rate limits and the User-Agent policy.
• Build LangChain tools on top of those sources, ready for the agents of the later modules.
• Design a point-in-time pipeline —WebSocket → Kafka → TimescaleDB (serving) + Parquet/DuckDB (backtest) + Feast— applying the rule "text to the vector store, numbers to SQL".
• Detect and neutralize the traps that invalidate backtests with fundamentals: look-ahead from restatements, reporting lag and survivorship bias.

Module 3 gave agents orchestration: state graphs, tools, persistence and human interrupts. But an agent is only as good as the data its tools serve it. This module opens Block II (understand) by building that foundation: which market and documentary sources exist in 2026, which license is needed to feed an LLM with them, how they are consumed programmatically, and how they are organized in a point-in-time architecture that the RAG of Module 5 and the calculations of Modules 7 and 8 can audit.

4.1 The source landscape in 2026

4.1.1 Institutional vs accessible

The financial data market in 2026 is stratified into two worlds that barely touch. In the first, the institutional terminals: as of July 2026 Bloomberg Terminal lists at around \(31{,}980 per seat per year (\)28{,}320 in multi-seat configuration), after the 6.5% increase applied to renewals since January 2025 (Github) . LSEG Workspace (successor to Refinitiv Eikon) sits above $22{,}000 per seat in its full version, with a reduced edition close to $4{,}000 per year (Github) . FactSet, with a historical list price of $12{,}000, shows real contracts averaging close to $45{,}000 per year according to buyer transactional data (arXiv.org) . S&P Capital IQ Pro moves in the \(12{,}000–\)20{,}000 per seat band (arXiv.org) . These figures are industry estimates —none of the four vendors publishes price lists—, but they converge across two independent sources and are accepted as a budgetary reference (arXiv.org) .

The decisive fact for an LLM systems engineer is different: the terminal subscription does not include programmatic access. Feeding an automated pipeline —and therefore a LangChain agent— with Bloomberg data requires B-PIPE (\(2{,}000–\)3{,}000/month) or Data License (\(5{,}000–\)50{,}000+/year); extracting terminal content into a vector store or a prompt without that license breaches the contract (arXiv.org) .

In the second world, developer-first APIs have reached a coverage that five years ago was exclusive to the buy-side. The structural milestone is the renaming of Polygon.io as Massive.com, effective October 30, 2025: the api.polygon.io keys and endpoints remain operational in parallel with api.massive.com, and CME/CBOT/COMEX/NYMEX futures, Benzinga data and an llms.txt file in the documentation for consumption by AI assistants were added (Quant Memo) . Its plans run from free EOD to $199/month real time and $2{,}000/month Business with a commercial license (Picture Perfect Portfolios) . Alpha Vantage gives 25 free requests per day and premium tiers from $49.99 to $249.99/month scaled by req/min, with the caveat that U.S. real time requires an additional entitlement (MDPI) . Tiingo covers 30+ years of EOD for \(30/month (\)50/month for the commercial-use Business plan) (A.L. Capital Advisory) . Databento distributes direct exchange data (not SIP) with 6.1-microsecond normalization and MBO/MBP-10 schemas in nanoseconds, from \(199/month [(pfolio.io)](https://www.pfolio.io/academy/tangency-portfolio) . Nasdaq Data Link operates as a marketplace with free economic datasets and à-la-carte premium ones (\)50–$500+/month) (Entropy Method: Solving RL Without Gradients) .

Table 4.1 — Institutional vendors vs accessible APIs (prices as of July 2026)

Interpretation. The table reveals three asymmetries. First: the scale factor between the two worlds is two to three orders of magnitude —a desk with Tiingo + Massive Business costs less per year than a single Bloomberg seat—, but what the institutional premium buys is not the data but the license, support and consolidated coverage. Second: the "LLM license" column decides the viability of an agentic project, not the price: every institutional row demands explicit negotiation before a single token enters a prompt, and only some accessible APIs (Massive Business, Tiingo Business) include it out of the box. Third: latency at developer prices already covers any strategy below aggressive intra-day; Databento even opens the door to microstructure. Operational conclusion: the pipelines of modules 5 to 8 can be built on the accessible half of the table, keeping the institutional vendors as the scaling path once the license is approved.

4.1.2 Licenses for LLM use: non-display, AI addenda and the yfinance case

Three contractual mechanisms determine whether a data point may enter a production LLM system.

The first is the exchange non-display fee: the consumption of market data by machines —algorithms, servers and, explicitly, LLM agents that generate signals— rather than by a human looking at a screen. The official NYSE guide sets $3{,}000/month for category 1 non-display on NYSE Trades and $5{,}000/month on the NYSE National Integrated Feed, with a category 3 that covers trading signals and ATSs (alanhou.org) . A LangChain agent whose tool queries a real-time feed falls, in the exchange's eyes, within this definition: it is the most frequent budget surprise in AI projects with market data (alanhou.org) .

The second mechanism is the vendors' AI addenda. Bloomberg, LSEG, FactSet and S&P contracts license data for internal use by licensed users; sending that content to a third-party LLM (where it may remain in logs) or to a vector store accessible by unlicensed staff enters prohibited territory absent an explicit "AI/derived data use" addendum (arXiv.org) . The counterpart: the accessible vendors formalize the reverse channel —official MCP servers from Alpha Vantage and Benzinga, llms.txt at Massive— which turns their APIs into agent-discoverable sources (Quant Memo) .

The third is the negative litmus test: yfinance. Yahoo shut down its public API in 2017; the library scrapes unofficial endpoints, saturates under load (~2{,}000 estimated requests/hour before the 429 error), its terms of service limit it to personal use and it lacks any SLA (Ask Everything) .

Risk note. Using yfinance as a tool of an agent in commercial production violates Yahoo's terms of service (personal use only) and is structurally fragile: the February 2025 web redesign broke production scripts overnight, and 1-minute intraday only covers 7 days (Ask Everything) . For teaching and prototyping it is valid; for systems feeding real decisions, the minimum defensible source is an API with a commercial license (Massive Business, Tiingo Business, Databento) or EDGAR, public by law. Migrating to Tiingo Power costs $30/month: there is no economic justification for taking on the contractual risk.

In institutional practice. On the desks that have industrialized LLMs, license review happens before architecture review. The approval checklist for an agentic pipeline includes: (1) entitlement and non-display category declared to the exchange; (2) signed AI addendum or confirmation that the plan covers internal redistribution; (3) storage segregation —the vector store holding licensed data is accessible only to licensed users—; (4) lineage: every agent answer must be traceable to its source and its license. A technically perfect pipeline on poorly licensed data is a regulatory liability, not an asset.

4.2 SEC EDGAR as a programmatic source

4.2.1 Real endpoints: submissions, companyfacts, companyconcept, frames and full-text search

EDGAR is the only major financial source that combines three properties: free, no API key, and direct legal value (filings are regulatory documents). In return, the SEC asks for a descriptive User-Agent with a contact email and a maximum of 10 requests per second; for massive backfills it recommends the nightly ZIPs rather than hammering per-company endpoints (arXiv.org) . The first step of any pipeline is to cache https://www.sec.gov/files/company_tickers.json, which resolves ticker → CIK locally (ResearchGate) .

Table 4.2 — SEC EDGAR endpoints (verified as of July 2026)

Interpretation. The XBRL family (XBRL: eXtensible Business Reporting Language, the financial-fact tagging standard mandated by the SEC) makes EDGAR a first-order source for LLM systems: every figure arrives structured with taxonomy, unit, period and —critical for section 4.4— the filed date on which it became public, the basis of any point-in-time reconstruction (ResearchGate) . companyfacts is the workhorse for a company's fundamentals; companyconcept minimizes tokens when the agent only needs one metric (a full companyfacts can exceed the context window); frames enables cross-sectional screening —comparing NetIncomeLoss across the whole universe for a quarter with one call—, the exact pattern of quantitative queries (ResearchGate) . EFTS is the discovery index of the documentary pipeline in Module 5: locating every filing that mentions "material weakness" in risk factors costs one HTTP request (Apify) .

The following snippet implements access to submissions while respecting SEC policy:

import time
import requests

SEC_HEADERS = {"User-Agent": "QuantResearch [email protected]"}  # mandatory
RATE_LIMIT_S = 0.12  # ~8 req/s, margin below the official 10 req/s maximum

def get_submissions(cik: int) -> dict:
    """Complete filing history of a company (columnar parallel arrays)."""
    url = f"https://data.sec.gov/submissions/CIK{cik:010d}.json"
    resp = requests.get(url, headers=SEC_HEADERS, timeout=20)
    resp.raise_for_status()
    time.sleep(RATE_LIMIT_S)
    return resp.json()

subs = get_submissions(320193)  # Apple
recent = subs["filings"]["recent"]
# form[i], filingDate[i] and accessionNumber[i] share an index (columnar format)
primer_10k = next(
    i for i, f in enumerate(recent["form"]) if f == "10-K"
)
print(recent["form"][primer_10k], recent["filingDate"][primer_10k],
      recent["accessionNumber"][primer_10k])
# Document URL: https://www.sec.gov/Archives/edgar/data/
#   {cik_no_zeros}/{accession_no_dashes}/{primaryDocument}

And this is the LangChain tool pattern over companyconcept —the first real tool of the course, which the agents of modules 3 and 7 will consume directly:

from langchain.tools import tool
import requests

@tool
def get_xbrl_concept(ticker: str, concept: str = "NetIncomeLoss") -> str:
    """Returns the history of a us-gaap XBRL concept for a company (SEC EDGAR).

    Each fact includes period (end), value (val), form (form) and,
    critically, the publication date (filed) for point-in-time joins.
    """
    cik = resolve_cik(ticker)  # cached lookup from company_tickers.json
    url = (f"https://data.sec.gov/api/xbrl/companyconcept/"
           f"CIK{cik:010d}/us-gaap/{concept}.json")
    r = requests.get(url, headers={"User-Agent": "QuantApp [email protected]"},
                     timeout=20)
    r.raise_for_status()
    units = r.json()["units"]
    return str(units)[:4000]  # truncate: token and cost control

Best practices separate a production tool from a demo: a precise docstring (the LLM chooses the tool by its description), input validation with pydantic, output truncation, compact JSON instead of HTML, actionable errors and retries with backoff in the face of 429s from GDELT, Yahoo or Alpha Vantage (Apify) .

import time
import requests

SEC_HEADERS = {"User-Agent": "QuantResearch [email protected]"}  # mandatory
RATE_LIMIT_S = 0.12  # ~8 req/s, margin below the official 10 req/s maximum

# STEP 1 — ticker → CIK with the cached file: one download, a thousand lookups
tickers = requests.get("https://www.sec.gov/files/company_tickers.json",
                       headers=SEC_HEADERS, timeout=20).json()
cik = next(v["cik_str"] for v in tickers.values() if v["ticker"] == "AAPL")
# cik == 320193 (integer, no leading zeros)

# STEP 2 — submissions: the CIK is formatted to 10 digits with leading zeros
subs = requests.get(f"https://data.sec.gov/submissions/CIK{cik:010d}.json",
                    headers=SEC_HEADERS, timeout=20).json()
time.sleep(RATE_LIMIT_S)

# STEP 3 — columnar format: form[i], filingDate[i], accessionNumber[i]
# and primaryDocument[i] are parallel arrays traversed with the same index
recent = subs["filings"]["recent"]
i = next(j for j, f in enumerate(recent["form"]) if f == "10-K")

# STEP 4 — document URL: CIK WITHOUT zeros and accession WITHOUT dashes
accn = recent["accessionNumber"][i].replace("-", "")
url = (f"https://www.sec.gov/Archives/edgar/data/{cik}/"
       f"{accn}/{recent['primaryDocument'][i]}")

4.3 News, sentiment and alternative data

4.3.1 GDELT, Benzinga, Alpha Vantage; MCP; Form 4 and 13F; deduplication and alpha decay

The news ecosystem has a well-known price cliff: NewsAPI.org is free only in development (100 req/day, 24 h delay) and jumps to $449/month on Business —and even paid tiers do not deliver full article bodies (Github) . The structural alternative is GDELT, free and keyless: the DOC 2.0 API (https://api.gdeltproject.org/api/v2/doc/doc) monitors press, web and broadcast in more than 100 languages at a 15-minute cadence, with ArtList, TimelineVol (breaking news detection by volume) and ToneChart modes, 250 records per query and aggressive rate limits that demand caching (Fundamentals Hub) . In affordable paid territory, Benzinga distributes its real-time wire, ratings with history since 2012, earnings and guidance via Massive for $99/month per package, and launched its MCP server in November 2025 (templates) . In the institutional segment, RavenPack processes more than 40{,}000 sources with more than twenty sentiment indicators and 20+ years of archive for an estimated \(50{,}000–\)100{,}000+ per year (medium confidence: no public price), without covering social media (TL;DR Filing) .

Table 4.3 — News and sentiment sources (as of July 2026)

Interpretation. The reading of the table is that the cost of news is not in the API but in the downstream processing: GDELT is free but returns noise and republications that must be deduplicated before an LLM counts volume as signal; Benzinga charges to deliver the event already structured (rating action, price target, firm) (templates) . The availability of official MCP servers in three accessible rows is the 2026 change with the greatest architectural impact: a LangGraph agent discovers GDELT, Benzinga and Alpha Vantage as tools without custom wrappers, although the underlying quota and entitlement still apply (MDPI) . Finally, the GDELT-vs-Benzinga-vs-RavenPack choice is not about quality but about budget and latency: for daily research, GDELT with dedupe is enough; for intra-day signal with time value, the structured wire justifies its price.

Real GDELT query:

import requests

def gdelt_artlist(query: str, timespan: str = "1week", maxrecords: int = 50) -> list[dict]:
    """Recent articles from GDELT DOC 2.0 (no API key; respect ~6 s between calls)."""
    params = {"query": query, "mode": "artlist", "format": "json",
              "timespan": timespan, "maxrecords": maxrecords}
    r = requests.get("https://api.gdeltproject.org/api/v2/doc/doc",
                     params=params, timeout=30)
    r.raise_for_status()
    arts = r.json().get("articles", [])
    seen, out = set(), []
    for a in arts:  # dedupe by URL/title: wire agencies republish the same news item
        key = a.get("url", a.get("title"))
        if key not in seen:
            seen.add(key)
            out.append(a)
    return out

Two regulatory sources deserve separate mention. The insider Form 4 is filed within two business days of the transaction: it is the freshest legally mandated signal in the U.S. market —insider behavior appears there before it does in 13F, research notes or the press—, with free XML on EDGAR and APIs such as sec-api.io that structure it (Apify) . The 13F, by contrast, reflects quarter-end positions published with a 45-day delay, long-only in 13(f) securities —no shorts and most derivatives excluded—: historical context, never current positioning (TL;DR Filing) .

On exploiting sentiment, two empirical caveats. The first is deduplication: wire agencies republish the same story dozens of times, and counting volume without URL/title dedupe inflates the signal (orthogonal.info) . The second is alpha decay: the performance of a sentiment signal decays with its adoption, forcing continuous regeneration of variants (orthogonal.info) . A common working model is exponential decay

where \(\lambda\) grows with the diffusion of the signal; the half-life \(t_{1/2}\) of public sentiment signals is measured in months, not years (orthogonal.info) .

4.4 Point-in-time pipeline architecture

4.4.1 The golden rule of data: "text to the vector store, numbers to SQL"

Everything above converges on a single design decision, the one that organizes the course's reference architecture (structural insight I5) —the golden rule of data, sister of the course's golden rule from §1.2.2, applied to the storage layer—: unstructured text goes to the vector store; numbers go to relational/columnar databases and are queried with deterministic tools. Embeddings are semantic retrieval, not a store of exact figures: an LLM that "remembers" a revenue figure via vector similarity may retrieve the wrong period or distort the number, whereas a SQL query over XBRL returns the value with its filed and its accession number (Apify) . The concrete partition: to the vector store go 10-K/10-Q sections (Item 1A, Item 7), transcripts and full-text news, with {ticker, form, filing_date, section} metadata for pre-retrieval filters; to SQL/columnar go OHLCV, XBRL facts, 13F holdings, Form 4 transactions and estimates; to Redis, the hot state (Apify) . The stack is simplified if the vector store is pgvector inside Postgres/TimescaleDB itself (Apify) .

For numeric transport, the standard pattern is: vendor WebSocket → Python producer → Kafka topics partitioned by symbol (preserves per-asset ordering) → consumers → stores, using the exchange timestamp as event-time so that jitter does not reorder calculations; Kafka's replayability allows regenerating features without re-downloading from the vendor (博客园) . In serving, TimescaleDB (Postgres) provides hypertables and continuous aggregates that pre-compute OHLCV roughly 100 times faster than views over raw ticks, with 90–95% compression (Apify) :

CREATE TABLE tick_data (time TIMESTAMPTZ NOT NULL, symbol TEXT NOT NULL,
                        bid NUMERIC(20,8) NOT NULL, ask NUMERIC(20,8) NOT NULL);
SELECT create_hypertable('tick_data', 'time');

CREATE MATERIALIZED VIEW tick_1m_view WITH (timescaledb.continuous,
  timescaledb.materialized_only = false) AS
SELECT time_bucket('1 minute', time) AS time, symbol,
  FIRST((bid+ask)/2, time) AS open, MAX((bid+ask)/2) AS high,
  MIN((bid+ask)/2) AS low,  LAST((bid+ask)/2, time) AS close
FROM tick_data GROUP BY time_bucket('1 minute', time), symbol;

SELECT add_retention_policy('tick_data', INTERVAL '30 days');

For research and backtesting, the de facto pattern is partitioned Parquet (year=/month=/symbol=) queried with in-process DuckDB —columnar reads with predicate pushdown at zero marginal cost—, reserving TimescaleDB for serving (Entropy Method: Solving RL Without Gradients) . The piece that closes the temporal loop is Feast, a feature store with an offline store (training) and an online store (serving) whose point-in-time join generates datasets with exactly the anti-look-ahead semantics a backtest requires: each feature is joined with the latest value available at the label's instant, not the latest known today (Accuweather Api) .

EXPANDE

In plain terms: why numbers do not go in the vector store

An embedding is a mathematical summary of a text's meaning, not a photocopy of its figures. If you store "revenue was 394.3 billion" as a vector, what you retrieve later is similar text —perhaps the wrong quarter, perhaps the figure of another company in the same sector— and the LLM rewrites it with its usual fluency. It is the mechanism of numerical hallucination, now with expensive infrastructure around it. By contrast, a SQL query over the XBRL table returns the exact value with its filed and its accession number: auditable, repeatable and interpretation-free.

The module's partition sums up in one sentence what a desk takes months to learn through incidents: text goes to the vector store with metadata for filtering, numbers to SQL/columnar queried with deterministic tools, and hot state to Redis.

Note the intermediate arrow in the diagram: the agent never "remembers" figures, it asks a tool that executes SQL for them. That is why pgvector inside Postgres itself simplifies the stack so much —a single database holds both halves of the golden rule—.

4.4.2 Data traps: look-ahead, survivorship and point-in-time universes

The availability rule that defines point-in-time (PIT) is simple to state and costly to implement: a fact \(f\) may only feed the decision at simulation time \(t\) if its effective publication date is earlier than or equal to \(t\):

There are three common violations, with distinct antidotes (orthogonal.info) . First, restatements: using the corrected value of a fundamental before the correction was public. The canonical case is Plug Power: diluted loss for fiscal year 2018 was originally reported at $0{,}36 per share and was restated to $0{,}39 in a filing dated May 14, 2021; any 2019 or 2020 simulation using $0{,}39 has leaked two years of future (orthogonal.info) . The SEC's XBRL API gives filed per fact, but shows the latest version of each period —it does not preserve the pre-restatement value—; strict PIT requires Compustat Point-in-Time, Bloomberg's PIT mode, Norgate, engines such as QuantConnect LEAN (fundamentals "as of" the publication date) or a custom versioning layer over accn + filed (orthogonal.info) .

Risk note. Look-ahead in fundamentals is the most silent overestimation in backtests with LLM data: it leaves no exceptions in the logs, the pipeline runs clean and the Sharpe comes out inflated. The Plug Power case ($0{,}36 → $0{,}39) looks harmless until it is multiplied by thousands of XBRL facts restated every year. Sister violation: reporting lag —joining the annual figure to the period close when the 10-K arrives 25–60 days later—. Correct temporal auditing asks one question per input: "at the exact instant this signal fired, could a real operator have known this number?" (orthogonal.info) . If answering requires looking at the filing calendar, the answer is no.

The third violation is survivorship bias: a universe built with today's living tickers has removed the bankrupt, acquired and delisted ones, and never sees the worst outcomes. The magnitude is not minor: in a Morningstar study, 58% of the funds existing in 1999 no longer existed in 2019 (Accuweather Api) . And there is a structural limitation of the cheap stack: APIs such as yfinance or Alpha Vantage do not serve delisted stocks, so a serious long-history backtest requires sources with delistings (Norgate, CRSP) and point-in-time universes that reconstruct the S&P 500 constituents as of each date, not today's (Accuweather Api) . Checklist complements: shift every signal by one bar (anything computed with bar \(N\) cannot be traded until the open of \(N+1\)), exchange versus reception timestamps, halt-gap detection, XBRL null validation and news dedupe (orthogonal.info) .

With this module, the reader has the sources (4.1–4.3) and the temporally honest architecture (4.4) that the later agents will consume: Module 5 (RAG) will be built on the documentary branch of the diagram, and Modules 7 and 8 on the numeric branch and its PIT joins.

Exercises

1. EDGAR explorer. Write a script that, given a ticker, resolves the CIK with company_tickers.json, downloads submissions and lists the last ten 10-K/10-Q filings with the document URL built from accessionNumber and primaryDocument. Respect 10 req/s and add exponential backoff on a 429. Extend it to measure the observed reporting lag: days between periodOfReport and filingDate for the 10-Ks of five companies.

2. Point-in-time XBRL tool. Turn get_xbrl_concept into get_concept_asof(ticker, concept, as_of_date), filtering facts with filed <= as_of_date and returning the latest available one. Verify the Plug Power trap: query EarningsPerShareDiluted for PLUG and show that a naive query (latest FY2018 value) and the PIT query with as_of_date = 2020-01-01 return different values.

3. Pipeline design. Draw (Mermaid or pseudocode) the pipeline of a strategy that trades on Benzinga news and confirms with EDGAR fundamentals: assign each component to Kafka, TimescaleDB, Parquet/DuckDB, Feast or vector store; justify for each data element whether it is "text" or "numbers"; annotate where an error would introduce look-ahead. Estimate the monthly cost as of July 2026 using Table 4.1, including the applicable non-display fee.

4. License analysis. A PM proposes feeding a research agent with (a) yfinance headlines, (b) quotes from the Massive Starter plan at $29/month and (c) transcripts obtained by scraping. For each source, identify the license or ToS conflict, propose the minimum compliant alternative (with price as of July 2026) and draft the justification paragraph for compliance.

5. Deduplication and alpha decay. Using gdelt_artlist, collect three days of articles on five semiconductor tickers. Quantify the duplicate rate before and after dedupe and discuss how it contaminates a "news volume" signal. Estimate the half-life under \(\alpha_t = \alpha_0 e^{-\lambda t}\) if ex-ante Sharpe falls 50% in eight months.

Module 4 of 15 — LangChain for Quantitative Trading. Data and versions verified as of July 2026; market prices and figures subject to change. LangChain 1.3.14 / langchain-core 1.5.2.