QandyEval: real financial work, graded.
QandyEval is QANDY's benchmark for financial agents. It measures whether a model can do the financial work people actually need done, under the rules in force on the day, in five domains and nine jurisdictions.
The financial work people actually need done.
-
Personal tax
Payroll withholding, annual returns, eligibility, residency and cross-border income, investment income, pensions, and the return forms themselves.
-
Corporate tax & accounting
Taxable income, rates and regimes, indirect tax returns, invoice compliance, withholding and treaties, and company bookkeeping through to the period-end close.
-
Save
Every contract and invoice in a household: what each costs, when it can end, the best offer for real usage, and cancelling or switching on the user’s behalf.
-
Trading
Order and market rules, costs and after-tax profit and loss, product access and suitability, disclosures, and margin and risk maths.
-
Insurance
Application and claim forms, policy extraction, coverage decisions, settlement calculations and public health reimbursement, across 11 lines of insurance.
- United States
- European Union
- United Kingdom
- Mainland China
- Japan
- South Korea
- Hong Kong SAR
- Taiwan
- Canada
The EU is covered at EU level and per member state, in tiers, across all 27 member states.
Right numbers, right forms, right law, every run.
-
Numbers and real forms
Every domain is tested on two things: getting the numbers right, and filling in the official forms field by field, consistent across forms.
-
The law on the day
Every item states the date it applies to. The hardest tier is built on 2026 rule changes and refreshed every quarter, so stale knowledge shows.
-
Audited answer keys
Answer keys come from calculation engines audited against primary sources.
-
Right every time
An item only counts if the model gets it right in every run (pass^k). Scores are 0–100 with a 95% range.
-
Simulated, never real
Portals, inboxes and drives are simulated. Nothing is ever submitted to a real tax authority or insurer, and no personal data enters the eval set.
Acting for the user, only with their authority.
Most agent evals fail an agent for taking any outward action. In Save, cancelling and switching is the job. So the agent gets written authority: cancel these, switch these if the saving is over a set amount, ask about the rest.
You ask. You approve. QANDY SAVE executes.
- 01The final state of every simulated provider portal.
- 02Every action stayed within the authority the user gave, and was sent in time.
- 03The effective date is right, with no gap in cover or service.
- 04Nothing else was touched.
- 05Decisions outside that authority went back to the user as a question.
Claude Fable 5.2 against Claude Opus 5.5 and Qwen 3.8 27B.
Claude Fable 5.2 and Claude Opus 5.5 answered the same frozen item sets under the same rules, scored 0–100 in every domain. Qwen 3.8 27B has not been run on the rebuilt hard and expert tiers yet. QANDY 2.0 27B results will follow.
- Claude Fable 5.2
- Claude Opus 5.5
- Qwen 3.8 27B
- Each tier counts 60% agentic tasks and 40% text questions. A hard or expert item counts only when it is fully right in all three runs. The core tier is one run.
- The hard and expert tiers were rebuilt on fresh cases (20 cases and 60 questions) and replace the pilot set of the first results, where Claude Fable 5.2 had 67 and Claude Opus 5.5 had 37. Their cases were hardened on rules both Claude models got wrong in earlier rounds; the cases and questions measured here had not been seen in any run.
- Qwen 3.8 27B has not been run on the rebuilt hard and expert tiers. Its scores are its core tier and its result on the pilot set, where it had no hard or expert item right in all three runs. It was run through a hosted API that accepts at most four page images per request, with reasoning off.
Show as table
| Domain | Claude Fable 5.2 | Claude Opus 5.5 | Qwen 3.8 27B |
|---|---|---|---|
| Personal tax | 52 | 50 | 4 |
| Corporate tax & accounting | 58 | 42 | 4 |
| Save | 35 | 34 | 4 |
| Trading | 74 | 66 | 7 |
| Insurance | 59 | 34 | 6 |
| Overall | 56 | 45 | 5 |
QANDY 2.0 27B runs on your own computer.
It powers QANDY SAVE, a personal financial agent that finds where you overpay and handles the switch, but only after you approve it.