Can Test-Time Scaling Improve World Foundation Model?

Thank you for acknowledging our work “timely discussion of WFMs” with an “extendable” evaluation framework. We sincerely appreciate the detailed feedback and have addressed your comments as follows.

Q1: Generalization to Other WFMs

A1: COSMOS represents the recent SOTA world foundation models, trained on over 10⁸ video clips with 10,000 H100 GPUs over three months. It goes beyond single-task domains like autonomous driving, making it a strong and representative model series for our study. Furthermore, COSMOS utilizes a standard autoregressive transformer architecture, a common backbone in many large-scale models. Precedent in LLM research indicates that test-time scaling principles generalize across different model families sharing similar foundational architectures. Consequently, the methods we investigate (e.g., test-time sampling, verifier designs) are inherently model-agnostic and readily extendable to other WFMs built upon the same autoregressive transformer paradigm.

---

Q2: Fast Tokenizer

A2: We agree that the tokenizer decoder originates from the COSMOS work. However, it is typically not used in COSMOS for final output due to lower quality (e.g., artifacts, blurriness); instead, COSMOS relies on a much slower diffusion decoder for high-quality generation.
Our key insight is that verifiers in TTS only need to reliably distinguish between better and worse candidates, and do not require to give a precise score. This insight drives us to use this fast tokenizer decoder into SWIFT's search and pruning stages to make TTS practical, as diffusion decoding is over 9000× slower.
We also prove the strong correlation between scores derived from the tokenizer decoder and those from the diffusion decoder in Figure 5. This confirms that the tokenizer decoder can serve as a reliable and highly efficient proxy for candidate evaluation during TTS.

---

Q3: Potential for WFMs in Other Modalities (e.g., 3D)

A3: Thank you for the insightful comment. We agree that extending evaluation and discussion to other modalities such as 3D meshes and 3D videos is a promising direction. In fact, our evaluation toolkit was designed with this in mind, including metrics such as 3D consistency and spatial relationship awareness.
While current WFMs like COSMOS primarily generate videos and do not directly produce 3D meshes or scenes, we took a step toward broader applicability by evaluating 3D video generation, following the protocol in the COSMOS paper. Specifically, we tested COSMOS-4B and COSMOS-4B+best-of-2 on 500 videos from RealEstate10K test set, using rewards including temporal consistency, perceptual quality, and 3D consistency (Due to time limit, this evaluation focused on COSMOS-4B, and we omitted spatial relationship metrics in this case since no text prompts were used). As shown in the accompanying table, our test-time scaling approach significantly improved performance in this 3D-aware setting. We will incorporate a brief summary of these findings in the revised discussion section.

---

Q4: Comparison of Rule-Based and Preference-Based Rewards

A4: Thank you for this insightful question. To ensure a fair comparison in Figure 3, we used the same aspects (visual quality, temporal consistency) as reward for both Rule-Based and Preference-Based Rewards. The key difference is in how these rewards are derived:

• Rule-based rewards are computed from feature statistics (e.g., DINO similarity, X-CLIPScore),
• Preference-based rewards are learned via regression from human-annotated video preference pairs (e.g., using the VideoScore model).

Moreover, to avoid reward hacking, the evaluation metrics, geometric mean of FVD, FID, IQ(imaging quality), MS(motion smoothness), have no overlap with reward functions used for TTS.

Regarding your hypothesis about FVD: we conducted ablations focusing on FVD and FID in the following table. While preference-based rewards occasionally achieved slightly better FVD, this often coincided with notably worse FID. Overall, our rule-based approach yielded a more consistent and favorable balance across these metrics, supporting our selection of rule-based reward in our study.

• n: n in Best-of-n
• R:: Rule-Based Reward
• P:: Preference-Based Reward
• G: Geometric mean of normalized FVD and FID ↑ | n | R:FVD/FID | R:G | P:FVD/FID | P:G | |----|-----------|-----|-----------|-----| | 1 | 637.08/67.75 | 0.000 | 637.08/67.75 | 0.000 | | 2 | 622.82/58.93 | 0.054 | 619.06/62.09 | 0.049 | | 4 | 613.77/52.01 | 0.092 | 628.69/59.19 | 0.041 | | 8 | 599.10/49.27 | 0.128 | 602.98/56.50 | 0.094 | | 16 | 599.08/45.75 | 0.139 | 585.11/54.99 | 0.124 | | 32 | 596.38/42.48 | 0.154 | 584.36/54.69 | 0.126 |

Thank you for acknowledging our work’s “interesting insights” on “test‑time computation of world foundation models.” We have addressed your concerns below.

---

Q1: Regarding the Scope and Venue Fit

A1: Thank you for the thoughtful feedback. We believe our work aligns well with COLM’s scope:

• Multimodal foundation model, not just a video generation model: COSMOS itself is a multimodal model that models the physical world with both human-provided descriptions (text domain) and visual inputs (image or video domain). Video generation is the evaluation protocol to assess a WFM's ability to model the complex physical world, which does not mean we are studying a video generation model. We will revise our writing to avoid this confusion.
• LLM-inspired modeling: COSMOS is an autoregressive model that predicts the physical world via next-token prediction, the paradigm inspired by the success of LLMs. We believe it is very relevant to the new application of LLMs, aligning with Call for Papers #18 (LMs on diverse domains and novel applications).
• Aligned with COLM’s multimodal and world modeling tracks: COLM Calls for Papers #14 and #13 explicitly invite work on Multimodal LMs and LMs and the World, respectively, which our contributions directly fit, making us choose COLM.
• Precedent multimodality foundation model works in COLM: COLM has recently accepted several multimodal works [1–5], including on text-to-image generation [2,3], adversarial image reasoning [1], and compositional visual tasks [4,5].

---

Q2: Regarding the Obviousness of Observation

A2: We would like to argue that it is an open question (non-obvious) whether similar test-time scaling (TTS) trends hold for world foundation models (WFMs) as LLMs [6] and text-to-image generation [7], because (1) WFMs solve a hard task that models the physical world and involves more modalities; (2) Unlike LLMs, which often study tasks like math and puzzles with well-defined, rule-based rewards, WFMs need to operate under much weaker reward signals like visual fidelity and temporal consistency.

In this paper, we bridge this gap by presenting the first systematic study of test‑time scaling in WFMs. Though concluding that TTS exists may appear trivial from the experiments, we rigorously validate it through non‑trivial efforts: evaluating multiple model sizes, varying N in best‑of‑N sampling, and collecting thousands of video clips. The first WFM evaluation toolkit is established to fill the assessment gap, and practical TTS strategies are explored to reach a conclusion.

---

Q3: Generalization of Results

A3: Why COSMOS? COSMOS represents the recent SOTA world foundation models, trained on over 10⁸ video clips with 10,000 H100 GPUs over three months. It goes beyond single-task domains like autonomous driving, making it a strong and representative model series for our study. To the best of our knowledge, COSMOS was the only publicly accessible and fundamental WFM available at the time of our work. If other suitable WFMs are suggested, we are happy to include them.

Furthermore, COSMOS utilizes a standard autoregressive transformer architecture, a common backbone in many large-scale models. Precedent in LLM research indicates that test-time scaling principles generalize across different model families sharing similar foundational architectures. Consequently, the methods we investigate (e.g., test-time sampling, verifier designs) are inherently model-agnostic and readily extendable to other WFMs built upon the same autoregressive transformer paradigm.

To bolster the robustness and generalizability of our findings concerning scaling laws, we conducted a comprehensive evaluation:

• Evaluate across all available model sizes in the autoregressive COSMOS family (4B, 5B, 12B, 13B)
• Sweep N from 1 to 32 in best-of-N settings
• Compare multiple verifier strategies, including ORM, PRM, and our proposed probability-based beam search.

Regarding the "best-of-2" observation, we concur that the precisely compute-optimal N may vary on other WFMs. However, the critical insight is the consistent self-evolving trend: performance robustly improves with increasing N (as illustrated in Figure 3). This mirrors observations in LLM scaling literature, where general scaling laws hold broadly, even if specific numerical parameters differ across individual models.

---

Q4: Typos

A4: Thank you so much for pointing out the typos in our manuscript, which is important to further improve our quality. We have revised the paper.

Dear Reviewer qtYN:

We sincerely appreciate the time and effort you have dedicated to reviewing our work! Your insightful suggestions have been invaluable in improving the quality of our work. We have carefully considered all your feedback and have addressed all your comments in our response above, especially why we believe our work aligns well with COLM’s scope.

As the discussion period deadline approaches, we would be grateful if you could review our responses. We will be online over the next few days and are happy to address any further questions or concerns you may have. Thanks!

Thank you for acknowledging our work “addresses significant practical challenges” and provides “much-needed evaluation toolkit”. We sincerely appreciate the detailed feedback and have addressed your comments as follows:

---

Q1: Ablation of Temperature in SoftMax of Top-K Pruning

A1: Thank you for highlighting this. In our framework, the temperature τ in the softmax-based Top-K pruning is not a key hyperparameter. We heuristically select 0.1 since we find the reward scores are close in magnitude. So a smaller τ helps sharpen the distribution to better differentiate the probabilities to prune out unsatisfactory paths.
Following your suggestion, we conducted an ablation study by sweeping τ ∈ {0.01, 0.1, 0.5, 1.0} using our probability-based beam search (N = 4) on a nuScenes subset. We found that slightly varying t in range [0.1, 0.5] does not change the performance much.

We will include this ablation and other hyperparameter studies in the full revision for completeness.

---

Q2: Human Evaluation

A2: Thank you for the valuable feedback. To address your concern, we conducted an additional human study with another 12 new independent participants. Despite natural variation, the results remained consistent with our initial study:

• The 4B model with test-time scaling continued to outperform the 12B baseline, with preferences increasing by ~5% (ORM) and ~8% (probability-based beam search).
• Beam search remained favored over best-of-N ORM sampling.

Due to the time limitation, we only added 12 participants to the rebuttal. For future revisions, we are committed to further expanding participant numbers for human studies.

---

Q3: Deployment in Resource-Constrained Case

A3: We agree that test-time scaling (TTS) involves multiple inference passes. However, we respectfully argue that deployment cost should be considered alongside the training cost of larger models. Even from a deployment-only perspective, we observe a compute-optimality finding that multiple inferences with a small model can beat a single inference with a large model in terms of quality with similar cost. Similar trends have also been reported in the LLM literature [1]. Moreover, parallel inference can alleviate runtime concerns: because autoregressive models are typically memory‑bound rather than compute‑bound, it is feasible in practical systems to decode multiple paths concurrently.

We emphasize that test-time scaling (TTS) is fundamentally a method for trading inference compute for improved performance. One key motivation for studying TTS is to explore whether we can enhance a base model’s capability without any additional training or scaling. Our results show that small models with TTS can beat larger models, offering a practical alternative when training/assessing a larger model is infeasible. This is particularly valuable in two scenarios:

• When large models are unavailable—for example, the current 13B COSMOS model is not the upper limit by design, but a result of limited data and the high cost of WFM training;
• When large models are closed-source.

In such settings, computation is not the primary constraint. TTS offers a flexible, training-free pathway to scale performance under these limitations.

---

[1] Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

Dear Reviewer oo4B:

We sincerely appreciate the time and effort you have dedicated to reviewing our work! Your insightful suggestions have been invaluable in improving the quality of our work. We have carefully considered all your feedback and have addressed all your comments in our response above.

As the discussion period deadline approaches, we would be grateful if you could review our responses. We will be online over the next few days and are happy to address any further questions or concerns you may have. Thanks!