Benchmark Almanac

FinEval

Claude 3.5-Sonnet (snapshot unspecified) 72.9% tops its leaderboard Qwen2.5-72B 69.4% leads the models we track
Not in model cards

Models answer 8,351 Chinese finance questions in four areas: academic knowledge in 34 subjects, industry practice, financial security and financial agent tasks. Most questions are multiple choice, and the agent part tests tool use and multi-step reasoning. The makers run zero-shot and five-shot chain-of-thought settings.

The task set grew after the first release and now also has multimodal and rigor sections. Test answers are not public, so test scores come from submissions to the makers' leaderboard. About 290 GitHub stars.

Released
August 2023
Kind
Finance
Level
University
Scored
Checked automatically
Answers
Multiple choice
Tasks
Public, plus a private test set
Strongest model tested
Qwen2.5-72B #165 on the capability index
Last model added
September 2024
Scores refreshed
November 12th, 2026

In the authors’ words

Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs' financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Sonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain.

Abstract of FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (arXiv:2308.09975, CC0)
19
Entries on its own leaderboard, 7 models we track

Claude 3.5-Sonnet (snapshot unspecified) tops the leaderboard with 72.9%

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2.5-72B, at 69.4%.

The newest model with a score here came out in September 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

19 entries. Scroll sideways for the rest.

  1. 72.9 Claude 3.5-Sonnet (snapshot unspecified)
  2. 71.9 GPT-4o (snapshot unspecified)
  3. 69.4 Qwen2.5-72B
  4. 69.2 Gemini1.5-Pro (snapshot unspecified)
  5. 66.2 GPT-4o mini
  6. 65.6 Gemini1.5-Flash (snapshot unspecified)
  7. 62.3 Qwen2.5-7B-Instruct
  8. 61.5 Yi1.5-34B-Chat
  9. 59.1 XuanYuan3-70B-Chat
  10. 58.9 InternLM2.5-20B-Chat
  11. 58.4 GLM-4-9B-Chat
  12. 58.0 InternLM2-20B-Chat
  13. 56.9 Yi-1.5-9B-Chat
  14. 55.4 XuanYuan2-70B-Chat
  15. 55.3 CFGPT2-7B
  16. 47.8 Baichuan2-13B-Chat
  17. 43.2 ChatGLM3-6B
  18. 37.8 DISC-FinLLM
  19. 27.1 FinGPTv3.1
  1. 1 Claude 3.5-Sonnet (snapshot unspecified) zero-shot 72.9%
  2. 2 GPT-4o (snapshot unspecified) zero-shot 71.9%
  3. 3 Qwen2.5-72B zero-shot 69.4%
  4. 4 Gemini1.5-Pro (snapshot unspecified) zero-shot 69.2%
  5. 5 GPT-4o mini zero-shot 66.2%
  6. 6 Gemini1.5-Flash (snapshot unspecified) zero-shot 65.6%
  7. 7 Qwen2.5-7B-Instruct zero-shot 62.3%
  8. 8 Yi1.5-34B-Chat zero-shot 61.5%
  9. 9 XuanYuan3-70B-Chat zero-shot 59.1%
  10. 10 InternLM2.5-20B-Chat zero-shot 58.9%
  11. 11 GLM-4-9B-Chat zero-shot 58.4%
  12. 12 InternLM2-20B-Chat zero-shot 58%
  13. 13 Yi-1.5-9B-Chat zero-shot 56.9%
  14. 14 XuanYuan2-70B-Chat zero-shot 55.4%
  15. 15 CFGPT2-7B zero-shot 55.3%
  16. 16 Baichuan2-13B-Chat zero-shot 47.8%
  17. 17 ChatGLM3-6B zero-shot 43.2%
  18. 18 DISC-FinLLM zero-shot 37.8%
  19. 19 FinGPTv3.1 zero-shot 27.1%

Source: Official FinEval leaderboard

#EntryScoreSettingDateSource
1 Claude 3.5-Sonnet (snapshot unspecified) 72.9% zero-shot Official leaderboard
2 GPT-4o (snapshot unspecified) 71.9% zero-shot Official leaderboard
3 Qwen2.5-72B 69.4% zero-shot Sep 2024 Official leaderboard
4 Gemini1.5-Pro (snapshot unspecified) 69.2% zero-shot Official leaderboard
5 GPT-4o mini 66.2% zero-shot Jul 2024 Official leaderboard
6 Gemini1.5-Flash (snapshot unspecified) 65.6% zero-shot Official leaderboard
7 Qwen2.5-7B-Instruct 62.3% zero-shot Official leaderboard
8 Yi1.5-34B-Chat 61.5% zero-shot Official leaderboard
9 XuanYuan3-70B-Chat 59.1% zero-shot Official leaderboard
10 InternLM2.5-20B-Chat 58.9% zero-shot Aug 2024 Official leaderboard
11 GLM-4-9B-Chat 58.4% zero-shot Jun 2024 Official leaderboard
12 InternLM2-20B-Chat 58% zero-shot Official leaderboard
13 Yi-1.5-9B-Chat 56.9% zero-shot May 2024 Official leaderboard
14 XuanYuan2-70B-Chat 55.4% zero-shot Official leaderboard
15 CFGPT2-7B 55.3% zero-shot Official leaderboard
16 Baichuan2-13B-Chat 47.8% zero-shot Sep 2023 Official leaderboard
17 ChatGLM3-6B 43.2% zero-shot Oct 2023 Official leaderboard
18 DISC-FinLLM 37.8% zero-shot Official leaderboard
19 FinGPTv3.1 27.1% zero-shot Official leaderboard

5 models have held number one on the official leaderboard since its release in August 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

Other labs 01.AI Z.ai OpenAI Qwen
Own leaderboard Not tested since Sep 2024
Qwen2.5-72B 69.4%

Source: Official FinEval leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) Qwen2.5-72B GLM-4-9B-Chat Baichuan2-13B-Chat 0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) Qwen2.5-72B GLM-4-9B-Chat

Source: Official FinEval leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Law, finance and medicine

All 179 in Law, finance and medicine
  • MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. Qwen 3.8 Max 80.4%
  • HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
  • HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
  • FinSearchComp In use Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. Ling-3.1-flash 81.7%
  • KBL In use Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. Solar Pro 4 77.5%
  • KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Solar Pro 4 93.2%