Databricks made OfficeQA harder with a second edition. We kept the Energent agent fixed and compared the same models on both editions. The arrows in the chart above show what each model lost between editions.
TL;DR
- Two models held: Fable 5.1 (72.2% → 70.0%) and GPT-5.6 Sol (71.4% → 68.9%). Five fell by 6.5 to 28.8 points.
- The ranking reshuffled. Opus 5 led v1 at 74.4% and dropped to third; Sonnet 5 went from 67.7% to 38.9%, fifth to last.
- "Harder" is measurable: 2× the corpus (697 → 1,435 documents), 3.7× the documents cited per question (1.8 → 6.7), and about 9× the text to read per question (≈1 MB → ≈9 MB).
- For Energent users, nothing to configure: the agent is model-agnostic, so as your tasks grow we measure, pick the model that holds, and keep the best performance and speed for you.
Background: OfficeQA, v1 to v2
OfficeQA is Databricks' benchmark for grounded reasoning over U.S. Treasury financial reports. v1 here is OfficeQA Pro, introduced in the first announcement: the 133 questions labelled hard in the original benchmark, over 697 Treasury Bulletins, written by hand by human annotators.
v2 is OfficeQA Pro V2, introduced in the second announcement: 90 questions over 1,435 scanned ledgers of federal receipts and expenditures from 1793 to 2024.
Side by side, v2 is harder on every axis that matters for an agent:
| v1 | v2 | Change | |
|---|---|---|---|
| Documents in the corpus | 697 | 1,435 | 2× |
| Documents cited per question (mean) | 1.8 | 6.7 | 3.7× |
| Text to read per question (mean) | ≈1 MB | ≈9 MB | ≈9× |
Few benchmarks get a second edition of the same task. OfficeQA did, which makes difficulty an axis you can measure along.
How we tested
- Why OfficeQA. It looks like the work our users bring to Energent: find the right documents, pull figures out of dense tables, combine them, and return an exact answer that can be checked.
- Same agent, only the model swapped. The Energent agent is model-agnostic. The same tools, prompts, and step budget ran seven frontier models from Anthropic, OpenAI, and Google on both editions. The harder data is what drives the drop; with the agent fixed, the differences between models in how far they fell come down to the model, not the agent.
- Same rules for every model. Every answer is scored by the benchmark's official scorer, one run per model.
Results
Accuracy, v1 to v2
Answer accuracy under the official scorer, sorted by the v2 score.
| Model | v1 (133 q) | v2 (90 q) | Change |
|---|---|---|---|
| Fable 5.1 | 72.2% | 70.0% | −2.2 |
| GPT-5.6 Sol | 71.4% | 68.9% | −2.5 |
| Opus 5 | 74.4% | 63.3% | −11.1 |
| Fable 5 | 72.2% | 57.8% | −14.4 |
| GPT-5.6 Terra | 60.9% | 54.4% | −6.5 |
| Gemini 3.1 Pro | 56.4% | 44.4% | −12.0 |
| Sonnet 5 | 67.7% | 38.9% | −28.8 |
- On v1, five of the seven models sat inside a 7-point band; on v2 the same seven span 31 points.
- The two that held, Fable 5.1 and GPT-5.6 Sol, were mid-pack on v1 and finish first and second on v2.
v1: cost and inference profile
| Model | Cost / question | Avg duration | Model calls / q | Tool calls / q | Tools per call |
|---|---|---|---|---|---|
| Fable 5.1 | $0.58 | 1.7 min | 11.6 | 10.7 | 0.93 |
| GPT-5.6 Sol | $0.73 | 2.3 min | 13.6 | 23.9 | 1.76 |
| Opus 5 | $0.49 | 1.4 min | 9.7 | 11.6 | 1.20 |
| Fable 5 | $0.94 | 1.6 min | 8.5 | 11.0 | 1.29 |
| GPT-5.6 Terra | $0.26 | 1.7 min | 13.7 | 21.6 | 1.58 |
| Gemini 3.1 Pro | $0.30 | 1.3 min | 17.8 | 16.2 | 0.91 |
| Sonnet 5 | $0.28 | 2.2 min | 12.3 | 14.0 | 1.14 |
v2: cost and inference profile
| Model | Cost / question | Avg duration | Model calls / q | Tool calls / q | Tools per call |
|---|---|---|---|---|---|
| Fable 5.1 | $1.24 | 3.2 min | 18.1 | 17.6 | 0.97 |
| GPT-5.6 Sol | $2.12 | 6.9 min | 24.3 | 61.9 | 2.55 |
| Opus 5 | $0.98 | 2.7 min | 17.0 | 19.4 | 1.14 |
| Fable 5 | $2.00 | 4.0 min | 14.8 | 19.5 | 1.31 |
| GPT-5.6 Terra | $0.63 | 3.8 min | 20.7 | 43.4 | 2.10 |
| Gemini 3.1 Pro | $1.01 | 5.3 min | 38.4 | 36.9 | 0.96 |
| Sonnet 5 | $0.83 | 7.0 min | 25.9 | 33.0 | 1.27 |
- Every model worked harder on v2: 2 to 3.4 times the cost per question, 1.5 to 2.6 times the model and tool calls, and 1.9 to 4.1 times as long.
- The two that held did it differently: Fable 5.1 with the fewest tool calls of any model and 3.2 minutes per question, GPT-5.6 Sol with the most tool calls and 6.9 minutes.
Conclusion
OfficeQA's two editions offer something rare: the same task at two levels of difficulty, so you can see which models keep performing as the task gets harder and which ones drop. On v1 the seven models were bunched together; on v2 they spread far apart. Easy data hides the differences between models, hard data exposes them.
For Energent users, that choice is ours to keep making: as your tasks get harder, we measure the models on your kind of work, pick the one that holds, and balance accuracy against speed and cost, with nothing for you to configure.
Thanks
Special thanks to the Databricks team for OfficeQA: a raw, high-quality, and deliberately harder dataset, built on real documents with carefully verified questions, and released in two editions of the same task. That continuity is what let us see this trend at all. A detailed paper on these results is in preparation.
References
- Databricks, Introducing OfficeQA, the first-edition announcement
- Databricks, Introducing OfficeQA Pro V2, the second-edition announcement
- OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning, arXiv 2603.08655
- OfficeQA repository on GitHub
- OfficeQA Pro V2 dataset on Hugging Face