LongBench: A Long Context Evaluation Benchmark

LongBench: A Long Context Evaluation Benchmark

TL;DR

LongBench is a long context evaluation benchmark designed to test how well large language models handle extended inputs across 21 tasks and six task categories.

Now listed on AIOZ AI, LongBench gives developers and researchers 4,750 test samples across English, Chinese, and code tasks for evaluating document QA, summarization, few-shot learning, synthetic reasoning, and code completion.

What LongBench Is

LongBench is an LLM evaluation dataset built to measure long context understanding.

It focuses on tasks where the model must process long documents, multi-document inputs, long-form summaries, few-shot examples, synthetic retrieval tasks, or code context.

For builders working with long context LLMs, LongBench provides one structured benchmark to evaluate where a model performs well and where it starts to break down.

How the Dataset Is Organized

LongBench covers 21 tasks across six categories:

  • Single-doc QA: question answering over one long document
  • Multi-doc QA: question answering across multiple documents
  • Summarization: condensing long reports, meetings, and news inputs
  • Few-shot: classification or QA with long in-context examples
  • Synthetic: retrieval and counting tasks designed to test context use
  • Code: code completion with long repository or file context

The benchmark draws from 16 established NLP datasets, with some tasks created or resampled by the LongBench authors to better test long context behavior.

Key Dataset Details

  • Dataset: LongBench
  • Dataset type: long context LLM evaluation benchmark
  • Total samples: 4,750
  • Splits: test only
  • Tasks: 21
  • Task categories: 6
  • English tasks: 14
  • Chinese tasks: 5
  • Code tasks: 2
  • Average English length: 6,711 words
  • Average Chinese length: 13,386 characters
  • Evaluation metrics: F1, ROUGE-L, accuracy, edit similarity
  • License: MIT for the compilation; source datasets retain original licenses

Where It Fits Best

  • Long context model selection
  • Context window regression testing
  • Bilingual QA system evaluation
  • Summarization benchmark workflows
  • Code completion over long contexts
  • Comparing context handling across model versions

Access LongBench on AIOZ AI

LongBench gives builders a structured way to evaluate long context understanding across documents, languages, tasks, and code.

Twenty-one tasks, one benchmark for finding where long context models hold up.

Access LongBench on AIOZ AI and start evaluating long context performance without building your own benchmark from scratch.

FAQ

Q1: What is LongBench used for?

It is used to evaluate how well LLMs handle long context inputs across QA, summarization, few-shot, synthetic reasoning, and code tasks.

Q2: How many samples does LongBench include?

LongBench includes 4,750 test samples across 21 tasks.

Q3: Is LongBench for training or evaluation?

LongBench is designed for evaluation. It includes test data only, with no train or validation split.