MMLU: A 57-Subject Benchmark for LLM Evaluation

MMLU: A 57-Subject Benchmark for LLM Evaluation

TL;DR

MMLU is a multiple-choice benchmark designed to evaluate language models across 57 academic and professional subjects. Its 15,858 core questions span humanities, social sciences, STEM, and other domains, providing a shared baseline for measuring broad knowledge and problem-solving performance.

What MMLU Is

MMLU stands for Measuring Massive Multitask Language Understanding.

The benchmark brings questions from practice exams, standardized test preparation materials, and textbooks into one consistent evaluation format.

Its 57 subjects range from abstract algebra and computer science to law, medicine, history, economics, and world religions.

Every question contains four answer choices with one correct answer.

How the Benchmark Is Organized

Each MMLU record contains three fields:

  • question: Multiple-choice question text
  • choices: Four possible answers
  • answer: Correct answer label from A to D

The core benchmark contains 15,858 questions across three splits:

  • dev: 285 questions, with five examples per subject for few-shot prompting
  • val: 1,531 questions for validation
  • test: 14,042 questions for primary evaluation

MMLU also includes an auxiliary_train split containing 99,842 questions drawn from ARC, MC_TEST, OBQA, and RACE. These questions are not part of the original MMLU benchmark, bringing the complete distribution to approximately 115,700 records.

What MMLU Measures

Teams can use subject-level scores to:

  • Compare general-purpose language models on the same material
  • Identify strengths and weaknesses across domains
  • Measure changes before and after fine-tuning
  • Check whether domain adaptation reduces performance elsewhere
  • Establish a general knowledge baseline before deployment

MMLU and MMLU-Pro

MMLU remains a common baseline for general knowledge evaluation, while MMLU-Pro raises the difficulty with ten answer choices per question and a stronger focus on reasoning. Teams can use it alongside MMLU when they need clearer separation between more capable models.

Key Dataset Details

  • Dataset: MMLU
  • Full name: Measuring Massive Multitask Language Understanding
  • Dataset type: Language model evaluation benchmark
  • Core questions: 15,858
  • Total records with auxiliary_train: Approximately 115,700
  • Subjects: 57
  • Domains: 4
  • Question format: Four-option multiple choice
  • Splits: auxiliary_train, dev, val, and test
  • Human expert baseline: Approximately 89.8%
  • Language: English
  • Compilation license: MIT License

What Builders Should Know

MMLU is an English-language benchmark with largely US and Western-centric coverage, so multilingual and region-specific evaluations should complement it.

Because the benchmark is public and widely used, teams should also check for possible training-data contamination.

Access MMLU on AIOZ AI

57 subjects, 15,858 core questions, and one shared evaluation format: MMLU provides a practical baseline for measuring broad language model knowledge.

Access MMLU on AIOZ AI and start evaluating performance subject by subject.

FAQ

Q1: What is MMLU used for?

It is used to evaluate language models and compare their general knowledge performance.

Q2: How many subjects and questions does MMLU contain?

The benchmark covers 57 subjects and contains 15,858 core questions across the dev, validation, and test splits.

Q3: Is the auxiliary training split part of the original MMLU benchmark?

No. The auxiliary_train split contains 99,842 questions aggregated from ARC, MC_TEST, OBQA, and RACE. The original MMLU questions are contained in the dev, validation, and test splits.