EAGLE-2

Where EAGLE Fits#

EAGLE-2 builds on the EAGLE family of speculative-decoding methods, which observe that predicting hidden states (features) is easier than predicting tokens directly because features are smoother and lower entropy. A small autoregressive head — typically a one- or two-layer Transformer — predicts the next feature from the target model's penultimate hidden state, and that predicted feature is fed back through the target's LM head to produce a draft token.

Compared to drafting with an entirely separate small LLM, the feature-level approach gives higher acceptance rates because the draft head shares the target's representation space.

Dynamic Draft Trees#

EAGLE-1 generated a fixed draft tree — a tree of candidate continuations with a static fanout. EAGLE-2 makes the tree dynamic: at each step the draft head's confidence determines how wide the tree grows. Confident steps grow narrow; uncertain steps explore more candidates. The target model verifies the whole tree in one parallel forward pass.

The result is higher average accepted-token count per verification step at the same verification cost.

Integration#

• vLLM supports EAGLE-2 via the speculative-config interface.
• TensorRT-LLM ships an EAGLE-2 plugin in the speculative-decoding builder.
• SGLang supports EAGLE-2 natively.
• Pre-trained heads exist for Llama 2/3, Vicuna, Mistral and Qwen families.

Measured Speedup#

Published numbers for EAGLE-2 cluster in the 3-5x end-to-end speedup range on chat workloads with batch size 1-8, distribution-preserving. At higher batch sizes the speedup falls toward the 1.5-2x range as the target forward pass becomes compute-bound and parallel verification stops being cheap.

The technique is therefore most effective at moderate concurrency — a typical interactive chat service rather than a high-throughput batch endpoint.

Trade-offs#

Requires training and shipping an auxiliary head per (base model, fine-tune) combination, which adds operational overhead. Quality is unaffected — the rejection-sampling step preserves the target distribution exactly — so the trade-off is purely engineering.