QANDY 2.0 27B is now on Hugging Face
QANDY QANDY Join the waitlist
QandyEval · Benchmark

QandyEval: real financial work, graded.

QandyEval is QANDY's benchmark for financial agents. It measures whether a model can do the financial work people actually need done, under the rules in force on the day, in five domains and nine jurisdictions.

Five domains

The financial work people actually need done.

  • Personal tax

    Payroll withholding, annual returns, eligibility, residency and cross-border income, investment income, pensions, and the return forms themselves.

  • Corporate tax & accounting

    Taxable income, rates and regimes, indirect tax returns, invoice compliance, withholding and treaties, and company bookkeeping through to the period-end close.

  • Save

    Every contract and invoice in a household: what each costs, when it can end, the best offer for real usage, and cancelling or switching on the user’s behalf.

  • Trading

    Order and market rules, costs and after-tax profit and loss, product access and suitability, disclosures, and margin and risk maths.

  • Insurance

    Application and claim forms, policy extraction, coverage decisions, settlement calculations and public health reimbursement, across 11 lines of insurance.

Nine jurisdictions
  • United States
  • European Union
  • United Kingdom
  • Mainland China
  • Japan
  • South Korea
  • Hong Kong SAR
  • Taiwan
  • Canada

The EU is covered at EU level and per member state, in tiers, across all 27 member states.

How it's graded

Right numbers, right forms, right law, every run.

  • Numbers and real forms

    Every domain is tested on two things: getting the numbers right, and filling in the official forms field by field, consistent across forms.

  • The law on the day

    Every item states the date it applies to. The hardest tier is built on 2026 rule changes and refreshed every quarter, so stale knowledge shows.

  • Audited answer keys

    Answer keys come from calculation engines audited against primary sources.

  • Right every time

    An item only counts if the model gets it right in every run (pass^k). Scores are 0–100 with a 95% range.

  • Simulated, never real

    Portals, inboxes and drives are simulated. Nothing is ever submitted to a real tax authority or insurer, and no personal data enters the eval set.

The permission test

Acting for the user, only with their authority.

Most agent evals fail an agent for taking any outward action. In Save, cancelling and switching is the job. So the agent gets written authority: cancel these, switch these if the saving is over a set amount, ask about the rest.

You ask. You approve. QANDY SAVE executes.

What the grader checks
  1. 01The final state of every simulated provider portal.
  2. 02Every action stayed within the authority the user gave, and was sent in time.
  3. 03The effective date is right, with no gap in cover or service.
  4. 04Nothing else was touched.
  5. 05Decisions outside that authority went back to the user as a question.
Head-to-head

Claude Fable 5.2 against Claude Opus 5.5 and Qwen 3.8 27B.

Claude Fable 5.2 and Claude Opus 5.5 answered the same frozen item sets under the same rules, scored 0–100 in every domain. Qwen 3.8 27B has not been run on the rebuilt hard and expert tiers yet. QANDY 2.0 27B results will follow.

QandyEval scores by domain Overall score, 0-100 · core 20%, hard 50%, expert 30% · as of 2026-11-17
  • Claude Fable 5.2
  • Claude Opus 5.5
  • Qwen 3.8 27B
Personal tax 52 50 4
Corporate tax & accounting 58 42 4
Save 35 34 4
Trading 74 66 7
Insurance 59 34 6
Overall 56 45 5
  • Each tier counts 60% agentic tasks and 40% text questions. A hard or expert item counts only when it is fully right in all three runs. The core tier is one run.
  • The hard and expert tiers were rebuilt on fresh cases (20 cases and 60 questions) and replace the pilot set of the first results, where Claude Fable 5.2 had 67 and Claude Opus 5.5 had 37. Their cases were hardened on rules both Claude models got wrong in earlier rounds; the cases and questions measured here had not been seen in any run.
  • Qwen 3.8 27B has not been run on the rebuilt hard and expert tiers. Its scores are its core tier and its result on the pilot set, where it had no hard or expert item right in all three runs. It was run through a hosted API that accepts at most four page images per request, with reasoning off.
Show as table
Domain Claude Fable 5.2Claude Opus 5.5Qwen 3.8 27B
Personal tax 52504
Corporate tax & accounting 58424
Save 35344
Trading 74667
Insurance 59346
Overall 56455

QANDY 2.0 27B runs on your own computer.

It powers QANDY SAVE, a personal financial agent that finds where you overpay and handles the switch, but only after you approve it.