Portfolio
FinFacts-Eval: LLM financial-QA benchmarkSEC EDGAR XBRL · 36 S&P 500 companies · computed gold answers

FinFacts-Eval

LLMs rarely refuse finance questions. They state wrong figures confidently: the wrong units, the wrong fiscal year, a restated value given as the original, or a number for a period that hasn't been reported. This benchmark targets exactly those failures. Every gold answer is computed from SEC filings and traceable to its source.

Items by task family and split

Split by company, so no company's figures leak from train into test

DPO rejected answers by failure mode

Each rejected answer is verified to be graded as its labeled failure

Task families

Explore items

Select an item to see its gold answer, rationale, provenance (links to the filing on sec.gov) and the exact prompts.

Annotator review queue

Human QA on generated items, the way an AI-training team runs it. Verdicts are saved in your browser only (nothing is sent to a server). Export them as JSONL when you're done.

Review summary

Grader playground

Type the answer a model might give and see the deterministic grader's label. This is a JavaScript port of the Python grader, verified to reproduce its labels on all 1,174 graded answers.

Error taxonomy

  • scale_error: off by a power of 1,000 (thousands vs millions)
  • percent_format: 0.125 given instead of 12.5
  • wrong_period: matches the same metric in a different fiscal year
  • restated_value: gave the restated figure when the original was asked for
  • rounding_or_arith: within 5% but outside tolerance
  • hallucinated_answer: gave a number where the right answer is "not available"
  • false_abstain: refused an answerable question

Model results

Accuracy by task and mode

Percent correct; hover for counts

Error taxonomy

Incorrect responses by label and mode

Misses

How it's built

Download source code (.zip)EDGAR client, DuckDB warehouse, item generator, grader, Claude runner, Airflow DAG, datasheet and tests

Pipeline

SEC EDGAR APIcompanyfacts, rate-limited
→
DuckDB warehousefiscal calendar, concept resolution
→
Item generator7 task families + provenance
→
Validationschema, provenance, oracle = 100%
→
Eval / SFT / DPOJSONL by company split
→
Claude runner + graderstructured outputs, taxonomy

Data engineering details

  • Fiscal years aren't the year of the period end. Home Depot's FY2025 ends 2026-02-01, and a 10-K tags its comparative years with the filing's fy. Each period end is mapped to the 10-K in which it was the current period.
  • Concept drift. Revenue moved from Revenues to RevenueFromContractWithCustomer… after ASC 606. Each metric resolves through a priority list, and the chosen concept is kept as provenance.
  • Ticker drift. In 2026 SEC's ticker map pointed XOM to a new holding company with no 10-K, so its CIK is pinned.

Quality story

Porting the grader to JavaScript for this page meant testing it against every Python label. That test found four DPO "rejected" answers that were actually correct: Apple's diluted EPS was $6.11 in FY2022 and $6.13 in FY2023, inside the original 0.5% tolerance.

Tolerances for reported figures are now cent-level for EPS and 0.05% for dollar amounts. Every rejection is checked to be graded as its labeled failure, and the case has a regression test.