Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models

About the computation cost of IAP sampling

Computation cost does not necessarily increase. Our IAP, which maintains the same performance as traditional tiling for regular images, does not increase the actual number of visual tokens. IAP primarily targets long-context sampling of images, particularly for images with long-tail aspect ratios similar to those found in InfoVQA. For example, for an image with a height and width of (1366, 6350) in InfoVQA, the number of visual tokens generated by Eagle2.5, InternVL3, and Qwen2.5VL (these models use 28x28 compression ratio for each image) is as follows.

Compared to traditional tiling methods like InternVL3 and LLaVA, IAP preserves more contextual information for large images, resulting in better results (as shown in Table 6 in the main paper), while only slightly increasing computational overhead (0.45s → 0.47s). Compared to Qwen2.5VL's anyres sampling strategy, Eagle2.5-8B saves about 50% vision tokens, yet its performance is only slightly lower (qwen2.5vl: 82.6).

About the data quality

Multiple works[1][2] have proved that integrating or collaborating multiple VLM models can reduce the response illusion and accuracy of queries. Hence, to curb hallucination in automatically generated captions and QA pairs, we performed a mutual review cycle in which GPT-4o and Gemini-1.5-Pro iteratively scored and revised each other’s annotations until consensus was reached. This procedure markedly improved caption fidelity, but it also introduced substantial time and money costs. In pilot studies with a 7B-parameter LLM, the marginal gain in downstream performance did not justify the extra cost. Consequently, we prioritized incorporating a broader range of newly rollout of GPT4o/Gemini 1.5 Pro—after basic sanity checks—into the full training set, deferring more intensive refinement to future work.

Why use CLIP for video clip retrieval?

We initially considered several models as potential options, including CLIP, SSCD, InternVideo, and SigLIP. However, after careful evaluation, we ultimately selected CLIP for our deployment. Our decision was based on several factors. First, although SSCD is effective for capturing detailed visual textures, we observed that the queries in existing video benchmarks are more semantic in nature, rather than focusing on fine-grained visual details. InternVideo and SigLIP are also strong candidates, but they come with higher computational overhead, which requires further consideration in our deployment context. In contrast, CLIP offers a good balance between semantic understanding and computational efficiency, and its ONNX format allows for fast and easy inference. In summary, while we considered multiple alternatives, we chose CLIP because it best meets the semantic requirements of our tasks while ensuring efficient inference.

About result highlighting in table 2 and 3

We appreciate the reviewer’s suggestion regarding the presentation of experimental results. We will revise the tables in the final version to highlight the best and second-best performance values for better clarity.

About the performance comparison with Qwen 2.5 VL

Qwen2.5 VL differs from Eagle 2.5 in many aspects. It uses a larger vision encoder than Eagle 2.5 (Qwen vision encoder 600m vs. siglip-400m used in Eagle2.5) for visual encoding. In addition, and most importantly, it samples much more pre-training data than Eagle 2.5 and does not publish the training data list. Qwen2.5VL trains on 18T vision-language tokens. Eagle 2.5 trains on a total of 0.67T vision-language tokens across its four stages. Consequently, the gap we observe on DocVQA and the overall average metric is likely driven less by architectural shortcomings in Eagle 2.5 but more by this stark disparity in encoder capacity and pre-training scale; we expect that scaling Eagle 2.5 to comparable data volumes and a higher-capacity vision backbone would substantially narrow the performance difference.

[1] Enhancing LLMs with an Ensemble of Critics for Mitigating Hallucination

[2] LVAgent: Long Video Understanding by Multi‑Round Dynamical Collaboration of MLLM Agents

Risk of Dropping Task-Critical Visual Cues

Before being fed into the LLM, all visual operations—such as sampling, slicing, visual encoding, visual token compression, and the visual-to-LLM projection—are query-agnostic. Consequently, two key factors determine the effectiveness of visual cue extraction: (1) preserving the integrity of the original input source, which directly sets the upper bound of visual feature information available to the LLM; and (2) providing complete textual semantic supervision, ensuring the visual encoder can extract universal visual features adaptable to any query. Our IAP ensures the upper limit of visual information, while ADS complements it by guaranteeing textual semantic supervision. Through end-to-end training, the visual encoder becomes task-agnostic and capable of extracting general visual features for query-aware decoding in the LLM.

How to prove heuristic method actually preserve more information.

We show this through practical experimental results. For IAP, we scan all images in the training set, compare the results of IAP and traditional tiling methods from InternVL and LLaVA, and remove results that did not match the tiling. We calculate the average coverage of the original image after tiling, as shown in the table below.

The results show that IAP can retain more pixels input to the model.

For ADS strategy, we scan all video data in the training data, compare the results of ADS and non-ADS sampling strategies. non-ADS strategy samples frames at fixed FPS and is limited by the maximum number of frames. We calculate the label truncation rate and sample discard rate, as shown in the table below. We set max_len = 16k. ADS minimizes both the label truncation rate and the sample dropout rate by adaptively calculating the number of sampling frames.

In this table, the label truncation rate counts samples with partially truncated labels, and the sample discard rate counts samples with completely truncated labels.

Need for sampling strategy at training and testing time

IAP/ADS can be applied post-hoc to any pre-trained VLM because it is a pure input-layer transformation; no model parameters are modified. Nevertheless, matching the sampling distribution during training yields the best performance. We therefore recommend "train-time + test-time" for new models. ADS ensures that, given any input of multiple rounds of text-and-image dialogues, the processor automatically performs heuristic sampling of visual tokens according to the set max_len, thereby ensuring that labels are not truncated and reducing the cost of hyper-parameter tuning.

Visualization of IAP

As is shown in the Figure 3 of the paper, when the input image size is 1300×2000 and the maximum number of tiles 12, the target aspect ratios include:[(1, 1), (1, 2), (2, 1), (3, 1), (1, 3), (2, 2), (4, 1), (1, 4), (5, 1), (1, 5), (1, 6), (6, 1), (3, 2), (2, 3), (7, 1), (1, 7), (4, 2), (2, 4), (1, 8), (8, 1), (1, 9), (3, 3), (9, 1), (2, 5), (5, 2), (10, 1), (1, 10), (11, 1), (1, 11), (12, 1), (3, 4), (4, 3), (1, 12), (6, 2), (2, 6)].

In InternVL's tiling method, the closest ratio to the image aspect ratio of 13:20 is 2:3, and thus the final tiles cover an area of 896×1344 pixels.

In contrast, IAP (our method) considers not only aspect ratio proximity but also the matching degree of tile area to the original image, which helps avoid generating excessive redundant tiles or overly small tiles that lead to information loss. According to Eq. (1), compared to a score of 0.4516 for the 2:3 configuration, the 3:4 configuration achieves a higher score of 0.5200. As a result, our method selects the 3:4 layout and tiles an area of 1344 × 1792, which better preserves visual information and reduces unnecessary padding or information loss.

About release plan

The model weight and inference code are open to the public now. The training code and dataset are under legal review. They will be released when the legal review is done.

About reproducibility & release plan

The model weight and inference code are open to the public now. The training code and dataset are under legal review. They will be released when the legal review is done.

About the clarity of information-first sampling

In brief, Information-First Sampling is a dynamic token-allocation strategy that, under a fixed context budget, selects the most informative text spans and image regions so that critical content remains intact. It has two complementary components:

Image Area Preservation (IAP) – fully preserves spatial image information.

Automatic Degradation Sampling (ADS) – ensures temporal inputs and their supervision signals are retained in full.

Together, ADS and IAP let the model devote its limited tokens to the meaningful patches.

About standard deviations missing

Existing closed-source model APIs enable sampling during inference, which introduces uncertainty and can lead to bias in multiple tests. However, for open-source VLMs, the common practice is to directly use greedy decoding. Our experiments are run with decoding temperature τ = 0 that is common practice, thus we employ greedy decoding for every evaluation. Because this setup is fully deterministic, repeated evaluations yield identical predictions and standard deviations are not applicable.

In addition, we fix the same random seed throughout the entire training pipeline—covering Python’s random, NumPy, PyTorch, and data-loader workers—so that initialization, data shuffling, and gradient updates are fully reproducible.

We will make these settings explicit in the revised manuscript to enhance transparency and avoid any potential confusion.

About the missing related work

We acknowledge that the current draft requires more comprehensive coverage of prior work in this domain and will address this in the final manuscript.

Do you think long-form video understanding is fundamentally a long-context modelling problem?

Not entirely. For images, visual understanding can largely be regarded as a short-context spatial modeling task (e.g., modeling objects, scenes, and spatial relationships). Videos, however, introduce the temporal dimension. As a result, video understanding requires both spatial modeling of individual frames and temporal modeling across frames (e.g., modeling events, motion, and changes over time). Because the temporal span can grow without bound, understanding long-form videos within general multimodal large language models ultimately manifests as a long-context modeling problem. Moreover, the information density of video is typically much lower than that of text, leading to substantial variability in sparsity when processed by general-purpose models. Various sparsity-oriented methods—some designed specifically for video and others applicable to long-context modeling more broadly—can be leveraged to address this issue. Exploring these directions is an important avenue for future work, though it lies beyond the scope of this paper.

We found that the results of hourvideo breakdown ware missing in the second submission. Here we append the results:

About HourVideo breakdown

Here is the per-category results for HourVideo:

Thank you again for your thoughtful review. We would appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. If there are any remaining issues, we would be glad to further clarify.

About the novelty of this work

The success of large-scale multimodal models stems from a combination of factors, including—but not limited to—model design, large-scale multimodal data, and effective training strategies. In this work, we focus on these holistic innovations and improvements. Specifically, we treat data sampling and tiling methods as part of model preprocessing, as they are distinct from the raw data itself or the training method. As noted by both Reviewer 4qXs and Reviewer YT9W, our contributions also include a novel training strategy and a valuable video dataset enriched with dual annotations, which together enhance the capabilities and robustness of models tackling the challenging task of long-form video understanding.

About the base model

Eagle2.5-8B is built on SigLIP2-400M and Qwen2.5-7B, without relying on any existing VLM base model. Compared with Eagle2, which has three training stages (Stage 1, Stage 1.5, Stage 2), Eagle2.5 adopts five stages (Stage 1, Stage 1.5, Stage 2, Stage 3, Stage 4). The first two stages are consistent between the two models, focusing on pretraining with large-scale image–language data.

In Eagle2, Stage 2 trains the model on high-quality image-language data. In contrast, Stages 2–4 of Eagle2.5 progressively train the model on long-context data. A slight difference exists in Stage 1.5, resulting in Eagle2.5-Stage 1.5 achieving better performance than Eagle2-Stage 1.5 and Eagle2-stage2.

To isolate the impact of our improvements, we present the following key benchmark comparisons.

These results show that Eagle2.5-8B substantially outperforms Eagle 2 and Eagle2.5-stage1.5 model on video understanding tasks, while maintaining competitive performance on image-language tasks. This confirms that the introduced extensions are effective and generalizable.

Performance Improvement of Eagle-Video-110K

Overall, incorporating Eagle-Video-110K yields only a modest overall improvement because gains on long-form videos are partially counterbalanced by small drops on short clips, in some video benchmarks, especially Video-MME.

Video-MME includes three subsets: long, medium, and short. Therefore, when adding long video data to the original data, the evaluation results on different subsets will be offset. This will lead to limited improvement in overall performance. For example, on the setting in Table 7 (testing up to 64 frames), the performance of different subsets with or without eagle-video-100k is as follows:

Adding Eagle-Video-110K yields a sizeable +2.3-point gain on the long subset (and +1.2 on medium), but a –1.3-point drop on short, translating to a modest +0.7 overall. These findings suggest that jointly tuning for both long- and short-context videos remains an open, worthwhile direction for future work.

Thank you again for your thoughtful review. We would appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. If there are any remaining issues, we would be glad to further clarify.