MiMo-7B-Base: A Foundation Model for Math & Code Reasoning

MiMo-7B-Base: A Foundation Model for Math & Code Reasoning

TL;DR

MiMo-7B-Base is a 7B-class decoder-only language model pretrained from scratch for mathematical and code reasoning.

It combines a 32,768-token context window, 36 transformer layers, Grouped-Query Attention, RoPE, SwiGLU, and Multi-Token Prediction. Trained on approximately 25 trillion tokens through a three-stage data-mixture strategy, it provides a compact base for supervised fine-tuning, reinforcement learning, and domain-specific adaptation.

What the Model Is

MiMo-7B-Base, now listed on AIOZ AI, is the pretrained base checkpoint in XiaomiMiMo's reasoning-focused MiMo-7B family.

Its training data increases the density of reasoning patterns and incorporates synthetic reasoning data, with a strong emphasis on mathematics and code.

As a base model, MiMo-7B-Base is designed for downstream adaptation rather than instruction-following out of the box.

How the Architecture Works

MiMo-7B-Base uses a 36-layer decoder-only transformer with a 32,768-token context window.

Its core components include:

  • Grouped-Query Attention: Shares key-value representations across attention heads for more efficient inference
  • Rotary Positional Embeddings: Encodes token position across the context window
  • SwiGLU: Supports the feed-forward layers within each transformer block
  • Multi-Token Prediction: Trains the model to predict multiple future tokens instead of only the next token

Multi-Token Prediction

Multi-Token Prediction serves both training and inference.

During pretraining, MTP asks the model to predict several upcoming tokens, providing an additional learning signal beyond standard next-token prediction.

At inference time, an MTP layer can support speculative decoding by proposing future tokens for verification by the main model. The MiMo team reports an acceptance rate of approximately 90% with one MTP layer, helping accelerate generation when proposed tokens are accepted.

Pretraining for Reasoning

MiMo-7B-Base was pretrained on approximately 25 trillion tokens using a three-stage data-mixture strategy.

Key elements include:

  • Enhanced text extraction and multidimensional filtering
  • Higher reasoning-pattern density
  • Large-scale synthetic reasoning data
  • Progressive adjustment of the training mixture
  • Math and code scaled to as much as 70% of the mixture
  • Multi-Token Prediction as an additional training objective

MiMo-7B-Base serves as the starting checkpoint for other models in the family:

  • MiMo-7B-Base: Pretrained base model
  • MiMo-7B-RL-Zero: Reinforcement-learning model trained from the base checkpoint
  • MiMo-7B-SFT: Supervised fine-tuned model trained from the base checkpoint
  • MiMo-7B-RL: Reinforcement-learning model trained from the SFT checkpoint

Key Technical Details

  • Model: MiMo-7B-Base
  • Developer: XiaomiMiMo / Xiaomi LLM Core Team
  • Architecture: Decoder-only transformer
  • Scale: 7B class
  • Layers: 36
  • Context window: 32,768 tokens
  • Attention: Grouped-Query Attention
  • Positional encoding: Rotary Positional Embeddings
  • Activation: SwiGLU
  • Additional objective: Multi-Token Prediction
  • Pretraining data: Approximately 25 trillion tokens
  • Training strategy: Three-stage data mixture
  • Maximum math-and-code share: 70%
  • MTP speculative-decoding acceptance: Approximately 90% with one MTP layer

Download It on AIOZ AI

With 7B-class scale, approximately 25 trillion pretraining tokens, reasoning-focused data, and Multi-Token Prediction, MiMo-7B-Base provides a compact foundation for custom math and code models.

Download MiMo-7B-Base on AIOZ AI and use it as a starting point for your own fine-tuning or reinforcement-learning workflow.

FAQ

Q1: What is MiMo-7B-Base?

It is a 7B-class decoder-only language model pretrained from scratch with an emphasis on mathematical and code reasoning.

Q2: What does Multi-Token Prediction do?

Multi-Token Prediction trains the model to predict several future tokens and can also support speculative decoding during inference.

Q3: Is MiMo-7B-Base instruction-tuned?

No. It is the pretrained base checkpoint. MiMo-7B-SFT and the reinforcement-learning variants are separate models built from the base.