BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Data agents over structured sources must fit database schema into the model's context window. At realistic scale, however, a pooled catalog spans many databases and thousands of columns, so schema must be selected and serialized under a token budget that binds on cost and coverage well before any fixed window fills. We contribute BudgetSchemaBench, a reusable, execution-grounded diagnostic for this regime and the methodology that builds it. Over a fixed 80-database union catalog, the diagnostic sweeps an explicit schema-context budget $K$ and isolates the schema representation. Its construction is derivation-first: relevance labels come mechanically from gold SQL (no human or LLM ground truth), and a source-namespace validity contract rejects result-equivalent-but-wrong-source answers. Three controlled regimes sit behind explicit construction-validation gates: end-to-end, frozen-gold, and a remove-the-needed-table probe. On a held-out-complement read with bootstrap CIs, the diagnostic yields three patterns. The budget axis discriminates when retrieval is coverage-limited: a lexical retriever improves with $K$, while a strong dense retriever saturates recall and largely flattens the curve. Removing the needed tables exposes a schema-memorization floor, a weight-time-memory signal; schema representation is a bounded, generation-side null. The same three patterns reproduce on a single off-family reasoning model. Where coverage is the bottleneck, tokens buy accuracy through the schema budget rather than the serialization choice; the released artifact lets benchmark builders and method developers measure where a schema-context budget is actually spent.