WISA: World simulator assistant for physics-aware text-to-video generation

Dear Reviewer 1tcW,

Thank you very much for your valuable and insightful comments, and for your praise of our paper’s clear figures, straightforward motivation, valuable finely annotated dataset, excellent writing and explanations, as well as significant performance improvements. Below, we address and clarify each of your comments (including identified weaknesses and questions) in detail.

---

W1: Name

W1: The naming of things is confusing and unintuitive at times ....

A1: Thank you for this valuable feedback on the naming conventions. We agree that the dual use of "WISA" could cause confusion.

Our original intent for the name "World Simulator Assistant" was to encapsulate its dual role: the dataset acts as a data assistant, while the framework acts as a model assistant. However, we recognize your point about the potential for ambiguity.

To improve clarity, we will rename the dataset to WISA-DATA-80K throughout the revised manuscript. This clearly distinguishes the dataset from the WISA framework and resolves the ambiguity.

---

W2: Related Works

W2: I think it's very important to have a discussion of related works in the body of the paper ...

A2: Thank you for your suggestion. We will adjust the paper format and include the Related Work section in the main manuscript as much as possible.

---

W3: Physical Classifier

W3: I had a hard time understanding Section 4.3, the Physical Classifier. ...

A3: We apologize that our description of the Physical Classifier was unclear. Thank you for giving us the opportunity to clarify. Here is a detailed breakdown following your A/B/C structure:

A) Input:

• At Training: The input is a learnable embedding vector, which we call the [PHYSICS_TOKEN]. This token is concatenated with the noisy visual tokens and text prompt tokens and is processed by the entire MM-DiT and MoPA architecture.
• At Inference: The input is identical to training—the same trained [PHYSICS_TOKEN] is used.

B) Output:

• At Training: After being processed by the network, the final hidden state of the [PHYSICS_TOKEN] is fed into a simple MLP classification head. This head outputs logits for the 29 qualitative physical categories. This output is used only to compute the multi-label binary cross-entropy loss for training.
• At Inference: The output from the classifier head is completely discarded. It does not influence the video generation process in any way. Its sole purpose is to provide an auxiliary training signal to help the model learn to represent physical concepts.

C) What it is "classifying":

• It is performing a multi-label classification task on the 29 qualitative physical categories that we define (e.g., "Collision", "Melting", "No obvious dynamic phenomenon", etc., as detailed in Appendix A.6). For each video, it predicts which of these 29 phenomena are present.

We will replace the current text in Section 4.3 with this detailed, structured explanation to ensure clarity and reproducibility.

---

W4, W7, and Q2: Ablations on Major Components

W4: ... W7: ... Q2: As I understand it, WISA includes three different ways of baking ....

A4: Thank you for this critical question. This gets to the heart of our contribution, and we are happy to provide a detailed ablation study to clarify the impact of each component.

First, your understanding is perfectly correct. WISA integrates three types of physical information. To test your hypothesis and isolate the contribution of each, we conducted a new set of ablation experiments on CogVideoX-5B using our WISA-DATA-80K. The results are summarized below:

Analysis: These results clearly demonstrate that:

1. The Qualitative Physics Categories, guided by our MoPA and Physical Classifier, provide the most significant performance boost in isolation.
2. Appending Textual Physical Descriptions offers a moderate but clear improvement.
3. The full WISA framework, integrating all three components, achieves the best performance, indicating a synergistic effect.

Regarding your final point on "test-time LLM-guided prompt-engineering," this approach was explored by PhyT2V. As shown in Table 1, while it provides some benefit, its performance is limited and it incurs extremely high inference costs. Our results show that directly integrating these concepts into the model architecture via our proposed modules is a much more effective and efficient approach than relying on iterative prompting.

---

W5 and W6: Minor Typo

A5: Thank you for your suggestion. We will revise the example images accordingly and correct the errors, including the model names in the paper entries such as Table 1.

---

Q1: Data Sources

The paper says (approx line 126) "we manually collected videos from the Internet ...

A6: First, we defined 17 categories of physical phenomena. Based on this categorization, annotators were tasked with searching for relevant videos of these physical phenomena from publicly available YouTube channels, without involving any exclusive or private data sources. They also annotated the time segments during which the physical phenomena occur.

Finally, our data collection and release strictly comply with YouTube’s Data Privacy Policy and Fair Use Policy. The released dataset is intended solely for research purposes.

---

Q3: Noise in WISA-80K

How accurate were the automatically generated annotations (see lines 161-167) ...

A7: This is an excellent and crucial question regarding the reliability of our dataset annotations. To quantify this, we performed a manual evaluation on a randomly sampled subset of 500 videos from our dataset.

Our evaluation focused on the three types of AI-generated annotations:

1. Textual Physical Descriptions: We measured human rater satisfaction (i.e., does the description accurately reflect the video's physics?). Result: 95% satisfaction rate.
2. Qualitative Physics Categories: We compared the AI-generated labels against the ground-truth labels assigned during initial video collection. Result: 76% accuracy.
3. Quantitative Physical Properties: We again measured human rater satisfaction (i.e., are the estimated density/time/temperature values plausible?). Result: 86% satisfaction rate.

While some label noise is inherent in any large-scale, automatically annotated dataset, these results demonstrate that the overall quality of WISA-DATA-80K's annotations is high. The data is sufficiently reliable to provide a strong learning signal, as evidenced by the performance gains in our main experiments.

---

Q4: Wan2.1 vs. CogVideoX

It is well-established that Wan 2.1 has better video generation ...

A8: We agree that if the base model already possesses strong physical understanding capabilities, the gains introduced by WISA may be less pronounced. The reason Wan2.1's scores are not substantially higher than CogVideoX's is that the VideoCon-Physics [1] evaluation model was trained on videos generated by CogVideoX, but not on those generated by Wan2.1. A detailed explanation please refer to the manuscript(lines 255–258).

[1] VideoPhy: Evaluating Physical Commonsense for Video Generation

---

Q5: Why are there missing entries in Table 1?

A9: Thank you for your question. The results for VideoCrafter2, OpenSora, and HunyuanVideo are sourced from the VideoPhy paper. Since these models were evaluated exclusively on the VideoPhy prompt set, inference times and performance on our PhyGenBench benchmark are not available. Additionally, as the VideoPhy paper provides output videos for OpenSora, we were able to evaluate its IS and CLIP-SIM scores accordingly.

The results for PhyT2V are taken directly from its original paper. As PhyT2V is open-sourced, we measured its inference time ourselves. However, since PhyT2V focuses specifically on physical reasoning tasks, we only compare our method with its SA and PC components.

If you have further concerns or suggestions regarding this comparison, we would be happy to discuss and incorporate them.

---

Q6: Why is the PhyT2V in the same section in Table 1 as CogVideoX? Is it built using the same base model? Line 251 implies it's built on top of Tarsier-34B, a different video generation model.

A10: PhyT2V is a comprehensive test-time prompt-engineering framework that integrates a diffusion-based video generation model (CogVideoX-5B), a vision-language model for video captioning, and a large language model (Tarsier-34B) for prompt refinement. As all methods in Table 1 adopt CogVideoX-5B as the underlying video generation model, the comparison is fair and consistent.

---

Q7: Best Seed only for WISA?

To what extent did cherry picking of results ...

A11: The test results of WISA do not involve cherry-picking. We used the same random seed (42) as the baseline during evaluation to ensure a fair comparison.

---

Q8: What is the "open source video dataset" mentioned in lines 289-290?

A12: The "open source video dataset" mentioned in lines 289–290 refers to Koala36M. We will add a clarification about this in the manuscript.

---

Q9: Can you please provide more info about the User Preference study (section 5.4)? E.g. how many videos, which videos.

A13: The manual evaluation was conducted on 343 test samples from the VideoPhy dataset.

---

Q10: In Section A.5 ... but not for Wan2.1, how many are there?

A14: The number of trainable parameters for Wan2.1 is 587 million. We will include this information in the revised manuscript.

Dear Reviewer kzVn,

Thank you very much for your valuable and insightful comments, as well as for recognizing the novelty of our approach and the value of our carefully annotated dataset. Below, we address and clarify each of your comments (including identified weaknesses and questions) in detail.

---

W1 and Q1: Handling Prompts Without Annotations

W1: It is unclear how WISA model handle input prompts without detailed annotation, such as textual physical description, qualitative physical categories, and quantitative physical properties. According to the training pipeline, this information is required and integrated into the framework, but in-the-wild prompt data do not have such detailed annotation. The paper should provide more details about how the model obtains such information for inference. Otherwise, the comparison with baseline methods might be unfair.
Q1: How does the proposed method handle input prompts without detailed annotation?

A1: Thank you for this critical question regarding the handling of in-the-wild prompts. This is essential for the practical application and fair evaluation of our method.

Inference-time Annotation: For any given user prompt, we use a large language model (GPT-4o) with a set of predefined instructions to generate the required physical annotations (textual description, qualitative categories, quantitative properties). Crucially, this process uses only the input text prompt, with no access to any visual information, thus preventing any unfair information leakage. The cost is also minimal (approx. 2000 tokens per prompt). The detailed instructions are provided in the appendix (Figures 11-13).

Fairness and Effectiveness of WISA: To prove that WISA's performance gain comes from our structured information injection framework rather than just having more textual information, we conducted a new experiment. We concatenated all the generated physical annotations directly into the text prompt and fed it to the original CogVideoX-5B base model (without our WISA modules). The results on VideoPhy are shown below:

As the results show, simply adding the physical information as text did not improve performance; it slightly hurt semantic consistency. This strongly suggests that the base model cannot effectively utilize this information without the specialized modules and training provided by WISA. Therefore, WISA's superiority stems from its core design, not from an unfair advantage.

---

W2 and Q2: T2V or I2V Task

W2: The proposed method is only used for text-to-video generation, and the paper does not discuss or evaluate the model's capability for image-to-video generation, which is possible to evaluate with the curated dataset.
Q2: How could the model and curated dataset be used for image-to-video generation?

A2: In principle, WISA can also be applied to the I2V task. The main difference between image-to-video (I2V) and text-to-video (T2V) tasks lies in the input: I2V models receive a clean latent code for the first frame, while the rest of the input text and model architecture remain the same. The I2V task has explicit initial frame information and state, the model primarily learns to infer subsequent possible processes, generating intermediate and final states of physical phenomena. The T2V task is more challenging and closer to the vision of a true 'world simulator', as it requires the model to imagine the entire physical process from scratch—including the initial, intermediate, and final states—based solely on an abstract description. Consequently, we focus on building WISA for the T2V task, which poses a greater challenge to the model’s physical perception capabilities.

---

W3 and Q3 Physical Consistency and Semantic Coherence

W3: Details are missing. For example, the details of computing physical consistency, semantic coherence are not included.
Q3: How to compute physical consistency, semantic coherence?

A3: We adopt the automatic evaluation approach VideoCon-Physics proposed in VideoPhy to compute the Physical Consistency (PC) and Semantic Accuracy (SA) metrics. VideoCon-Physics was trained by collecting videos generated from nine different models, which were manually annotated for adherence to real-world physical laws and strong semantic consistency. Using this data, a vision-language model (VLM) was fine-tuned to serve as a reward model. During inference, the generated video and corresponding text prompt are fed into VideoCon-Physics, which outputs scores ranging from 0 to 1 for both metrics. We compute the average scores over the entire test set to obtain the PC and SA results reported in the Tables 1 and 2 of manuscript.

---

W4 and Q4 Missing Discussion with Physics-IQ

W4: Missing discussion with important prior works. "Physics-IQ: Benchmarking physical understanding in generative video models (2025)" is another benchmark for analyzing physical plausibility of video generation models, but it is not discussed in the paper.
Q4: Please include and discuss "Physics-IQ: Benchmarking physical understanding in generative video models (2025)"

A4: We will include a discussion of this benchmark in the Related Works section.

Dear Reviewer 7j5D,

Thank you very much for your valuable and insightful comments, as well as for recognizing the novelty of our approach, the value of our carefully annotated dataset, and the quality and reproducibility of our manuscript. Below, we address and clarify each of your comments (including identified weaknesses and questions) in detail.

W1 and Q1: Evaluation Metrics

W1: The evaluation metrics used focus primarily on text-video consistency and visual quality, but they largely overlook how well the generated videos adhere to physical principles (beyond user studies). Could the authors design specific metrics to assess physical plausibility? A feasible approach might be to evaluate physical correctness in toy scenarios. For example, generate a video where a ball undergoes free fall motion, and measure how well it follows expected physical behavior.
Q1: The need for more rigorous evaluation specifically targeting physical plausibility (related to Weakness 1).

A1: Thank you for this crucial point on evaluation. We agree that robustly and quantitatively measuring physical plausibility is a major challenge for the field.

As you noted, we used the established Physical Consistency (PC) metric from VideoPhy, which is a VLM-based evaluator specifically fine-tuned to assess adherence to physical laws. Our method shows significant gains on this metric (Table 1).

We find your suggestion of using toy scenarios (e.g., a free-falling ball) for precise dynamic analysis very insightful. This is a standard approach in physics-based animation. However, applying it to general-purpose text-to-video models presents significant challenges. The generated videos often have unknown or variable factors such as camera distance/angle, frame rate, and 2D projection effects, which make it difficult to reliably extract and verify kinematic properties (e.g., acceleration g). Moreover, many physical categories we study, such as melting, combustion, or optical phenomena (interference), cannot be easily evaluated with rigid-body dynamics models.

Given these challenges, the current state-of-the-art for evaluating physical commonsense in general T2V relies on VLM-based evaluators (like VideoPhy, PhyGenBench) or large-scale human studies. While we acknowledge their limitations, our work adheres to the current best practices in the field. Developing more robust, general-purpose physics evaluation metrics is an important open research problem.

---

W2: Video Comparison

It would be beneficial to include video comparison results in the supplementary materials. This would help reviewers and area chairs better assess the effectiveness of the proposed method.

A2: Thank you for your suggestion. In the PDF abstract, we have provided a link to an anonymous repository that includes comparisons with other methods, dataset video examples, and more. We are also willing to further supplement these materials in the supplementary section.

---

Thank you for these detailed questions about our core components. Let us clarify the roles of and relationship between MoPA and the Physical Classifier.

W3.1 and Q2: MoPA and the Physics Classifier

W3.1: Between MoPA and the physics classifier, which contributes more to enhancing physics awareness? The paper lacks an ablation study on this aspect.
Q2: A more detailed ablation study comparing the contributions of MoPA and the physics classifier (related to Weakness 3).

A3.1 The two components have a synergistic and progressive relationship. The Physical Classifier's role is to provide a direct supervisory signal during training. This signal guides the model to learn abstract physical categories. The MoPA module then leverages these learned categories to activate specific 'expert' attention heads, enabling it to model the distinct features of each phenomenon. The classifier enables MoPA's specialization. As our ablation study shows (Table 2, w/o PC), removing the classifier's supervision leads to a noticeable performance drop, confirming its crucial guiding role.

---

W3.2 and Q3: Expanded Description of the Physics Classifier

W3.2: The core component—the physics classifier—is intended to guide the T2V model to understand physical information. However, the implementation details are vague, making it difficult for other researchers to reproduce the work.
Q3: A clearer and more thorough description of the physics classifier, including implementation details (related to Weakness 3).

A3.2: We apologize for any confusion caused by the unclear description of the Physical Consistency (PC) component. We provide a more detailed explanation as follows:

We initialize a learnable token, referred to as the physical token, which is concatenated with the noise tokens and text tokens as input to the MM-DiT and MoPA modules. The physical token serves as a query to aggregate physical features. Just before the output projection layer, we extract the physical token and feed it into a multi-layer perceptron (MLP) classifier that outputs logits corresponding to the 29 quantitative physical categories. This classifier is supervised with a multi-label binary cross-entropy loss.

We will revise the manuscript accordingly to clarify the description of PC and improve reproducibility.

---

W3.3 and Q4: Physics Classifier and DPO

W3.3: Have you compared the proposed physics classifier with DPO-related methods? Which approach is more effective in guiding the model to understand and follow physical principles?
Q4: A comparison or discussion of the proposed physics classifier with DPO-based methods, particularly regarding their effectiveness in guiding models to learn physical reasoning (related to Weakness 3).

A3.3: This is a great question. Our Physical Classifier and DPO are complementary, not conflicting, methodologies. Our approach operates during the Supervised Fine-Tuning (SFT) stage, using explicit labels from WISA-80K. DPO, on the other hand, is a post-training alignment technique that relies on preference data. A promising future direction would be to first train WISA with our SFT approach and then use DPO with human preference data on physical plausibility to further refine the model.

We will add a discussion in the paper to clarify this distinction and position DPO as a potential future work.

---

Dear Reviewer jiPQ,

Thank you very much for your valuable and insightful comments. We sincerely appreciate your recognition of our paper’s clear motivation, the value of our carefully annotated dataset, the suprior performance of our approach, and the strength of our visual demonstrations. Below, we address each of your comments (including identified weaknesses and questions) in detail.

---

W1 and Q4: Introduing New Physical Categories

W1: Scalability: Adding a new physical category requires (i) manually defining the label, (ii) collecting extra data, and (iii) re-training the MoPA heads. This pipeline limits WISA’s ability to scale to larger, more diverse physics datasets. Q4: Is there an easy way to extend the pretrained WISA model when new physical categories are introduced?)

A1: Thank you for raising this important point about scalability. This is a crucial aspect for any foundational framework.

We respectfully argue that WISA's modular design, particularly the Mixture-of-Physical-Experts Attention (MoPA), makes it more scalable than monolithic models. When introducing a new physical category, traditional fine-tuning (e.g., via LoRA on a base model) would require retraining the entire network to incorporate the new knowledge. In contrast, our MoPA architecture allows us to simply add and train a new expert head dedicated to the new phenomenon, while keeping the existing expert heads largely frozen. This is conceptually more efficient and targeted.

Furthermore, as you correctly noted, the cost of generating annotations for new categories is low (approx. 2000 tokens per instance) thanks to our LLM-based pipeline. This combination of targeted modular retraining and low-cost data annotation provides a practical and scalable path forward.

We will add a discussion in the revised manuscript (e.g., in the Limitations or Future Work section) to explicitly detail this scalable extension process and highlight the advantages of our modular design.

---

W2, Q1 and Q2: Failure Cases in Unseen Physical Categories

W2: How does WISA behave on unseen physical categories? Should provide some failure cases. They would be beneficial for a more comprehensive analysis. Q1: Does performance drop noticeably on unseen physical categories? Q2: Should provide representative failure cases.

A2: This is an excellent suggestion. A comprehensive analysis must include failure cases. We performed additional qualitative evaluations on unseen physical categories like corrosion and electromagnetism.

As expected, both our WISA-enhanced model and the base model struggle to generate physically plausible videos for these categories. This is primarily due to the lack of relevant concepts and visual examples in the training data (WISA-80K), highlighting the current limitations of generalization to entirely out-of-distribution physical phenomena.

To provide a more comprehensive analysis, we will add a new subsection in the appendix discussing these limitations and, critically, we will include representative visual failure cases for these unseen categories. This will offer readers a clearer understanding of our method's current boundaries.

---

W3: Sparsely Inserted MoPA

W3: MoPA is inserted only in the final layer; does this leave early representations under-constrained? If MoPA modules are sparsely inserted into earlier layers, does performance improve?

A3: Thank you for this insightful question regarding the placement of MoPA. We did in fact explore this alternative design during our research.

We experimented with a configuration where six physical modules were sparsely inserted throughout the model (every 7 layers). As shown in the table below, this design yielded only marginal gains in performance (SA: 0.623 vs. 0.621; PC: 0.452 vs. 0.450) while incurring a significant computational cost: an additional 155M parameters and a ~40s increase in inference time.

After carefully weighing this performance-efficiency trade-off, we concluded that inserting a single MoPA module at the final layer offers the best balance, achieving substantial improvements over the baseline with minimal computational overhead.

---

W4: What is the model’s classification accuracy on physical categories during evaluation?

A4: We evaluated CogVideoX on the VideoPhy prompt set by computing the average of the logits over 50 sampling steps. The resulting mean accuracy across the 29 physical categories is 93.70%.

---

W5: Missing some important references [1][2][3][4]

A5: Thank you for your suggestion. We will add a discussion of these references in the revised manuscript.

---