LexGLUE
Models solve a suite of English legal text tasks drawn from court cases, EU law, contracts and terms of service. Most tasks are classification, and one asks for the holding of a US case. Each task reports micro and macro F1.
The suite was built for fine-tuned encoder models such as Legal-BERT, and each task ships with training data. About 270 GitHub stars, and the repo was last pushed in 2025-07.
- Made by
- University of Copenhagen and collaborators
- Released
- October 2021
- Kind
- Law
- Level
- Expert
- Scored
- Checked automatically
- Answers
- Multiple choice
- Tasks
- All public
In the authors’ words
Laws and their interpretations, legal arguments and agreements are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.
Legal-BERT tops the leaderboard with 78.9%
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.
- 1 Legal-BERT 78.9%
- 2 CaseLaw-BERT 78.5%
- 3 RoBERTa (Large) 78.4%
- 4 Longformer 77.5%
- 5 DeBERTa 77.4%
- 6 BigBird 77.2%
- 7 RoBERTa 76.8%
- 8 BERT 76.7%
Source: Official LexGLUE leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Legal-BERT | 78.9% | Official leaderboard | ||
| 2 | CaseLaw-BERT | 78.5% | Official leaderboard | ||
| 3 | RoBERTa (Large) | 78.4% | Official leaderboard | ||
| 4 | Longformer | 77.5% | Official leaderboard | ||
| 5 | DeBERTa | 77.4% | Official leaderboard | ||
| 6 | BigBird | 77.2% | Official leaderboard | ||
| 7 | RoBERTa | 76.8% | Official leaderboard | ||
| 8 | BERT | 76.7% | Official leaderboard |
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
More in Law, finance and medicine
- MedXpertQA-MM In use Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy.
Qwen 3.8 Max 80.4% - HealthBench Professional In use 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. Claude Sonnet 5.5 69.2%
- HealthBench In use 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. GPT-5 67.2%
- FinSearchComp In use Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy.
Ling-3.1-flash 81.7% - KBL In use Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items.
Solar Pro 4 77.5% - KorMedMCQA In use Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. Solar Pro 4 93.2%