Response to Q2: Design for Epipolar Attention when Epipolar Lines Vanish

Please also refer to our general response titled 4. Regarding the Register Token for additional context.

We apologize for not clearly presenting the details regarding this issue earlier. When we attempted to apply epipolar attention without register tokens, we encountered NaN errors during training and were unable to successfully train a model for ablation studies.

Our novelty lies in consideration of situations without epipolar lines, which is crucial for robust performance in practical applications. Regarding this scenario, we have observed that previous works on epipolar attention often do not provide special designs or detailed descriptions for handling cases where epipolar lines are absent. This gap motivated us to develop an effective solution.

When epipolar line exists but is far away from the queried frame, the zero length kv tokens in memory-efficient attention leads to NaN during training, making it impossible to train a model. In this case, two added register tokens serve as placeholder to ensure the non-zero length of kv tokens.

Despite utilizing only two register tokens is insufficient, it ensures the model's basic functionality and only part of pixels on the current frame who have no epipolar line degrade to CameraCtrl without epipolar attention.

Response to Q1: Elaboration on the insight of the clean/noised conditions.

Please also refer to our general response titled “1. The Connection Between Our Insights on Conditions and the Proposed Method” for additional context.

Clean conditions do not necessarily reduce uncertainty more effectively than noisy conditions. Their effectiveness depends on the information provided and is often related to the amount of condition information. For example, recaptioning in Stable Diffusion 3 generates more detailed text captions (clean conditions), resulting in better generation quality. In image-to-video generation settings, noisy conditions (noisy latent features such as $z_t$ typically originate from user-provided reference images. If $z_0$ is not provided by users, noisy conditions also arise from $z_t$ generated in previous timesteps. The noisy features $z_t$ have a shape of $F \times C \times H \times W$, which is significantly larger than clean conditions (e.g., text, which contains only textual information). When the added noise does not dominate the valid information, noisy conditions can reduce uncertainty more effectively by providing extensive details, colors, layout, and other relevant information.

Previous methods have introduced epipolar attention primarily with the motivation of enhancing geometric consistency, but these approaches were not specifically designed for settings with inherent noise. In contrast, our work focuses on the unique challenges posed by the noisy feature spaces inherent in diffusion models.

Our motivation is to understand how noise affects cross-frame interactions and to design strategies that mitigate the misleading effects of noise. In clean feature spaces, issues like feature copying due to large receptive fields can be mitigated by increasing model capacity and improving feature extraction. The main challenge in such settings is incorrect attention allocation when the aggregation space is too large.

However, in noisy feature spaces, the situation is different. Due to the added noise, dissimilar features may erroneously receive high attention scores. In this scenario, even powerful models struggle to reconstruct clean pixels from random noise—especially at high noise levels—because the noise can mislead the attention mechanism, resulting in degraded performance.

Our novel insight is that simply increasing the receptive field, such as using 3D full attention, leads to a greater amount of noisy conditions. Due to the randomness inherent in the noise, accessing more noisy conditions does not necessarily reduce uncertainty and thus not necessarily lead to better performance. This understanding suggests that models should focus on aggregating information from more reliable locations to effectively reduce uncertainty caused by noise. The idea of uncertainty introduced by noisy features but not clean features leads to a new insight.

We have found that applying epipolar constraints is one of the most suitable way to prevent the model from being misled by noise. By restricting attention to features along the epipolar lines, the model can interact with more relevant and less noisy information, improving cross-frame interactions in diffusion models. We believe these new insights are significant as they may inspire researchers to develop new architectural designs that better handle noisy conditions in generative models.

Response to Performance Improvement

Please refer to the Table we provided in general response in To all reviwers: 1. The Connection Between Our Insights on Conditions and the Proposed Method” for additional context.

Our method demonstrates significant improvements over CameraCtrl, achieving a 32.96% reduction in Rotation Error, a 25.64% decrease in CamMC, and a 20.77% improvement in Translation Error, without any decrease in FVD. These results were obtained using Text and Image CFG set to 7.5, 25 steps, and camera CFG set to 1.0 (no camera cfg).

Response to Evaluation Benchmark Improvement

We have identified that current evaluation benchmarks are unreliable, and no open-source evaluation code is available. To address this, we propose several improvements:

• Performing multiple measurements (averaging over five runs) to ensure consistency.
• Conducting extensive evaluations on a large number of samples (1000 samples) for statistical significance.
• Replacing the SfM algorithm with a more robust GLOMAP (new work of COLMAP's author, small improvement) to improve accuracy.
• Computing relative poses with respect to the initial frame to standardize evaluations.
• Introducing normalization to address the issue of unknown global scene scale caused by SfM predictions (the most important improvement).

We also commit to open-sourcing all our training and evaluation code, as well as the checkpoints. Although these modifications are small and straightforward, they hold significant importance for improving reliability and reproducibility in the community.

Response to Q6: More Qualitative Results.

Please refer to provided supplementary, and open the index.html.

Response to Q4: Potential Dynamics Drecrease

Please refer to provided supplementary, and open the index.html. Dynamics in generated videos are quite good.

Only applying epipolar attention without original spatial+temporal attention will lead to dynmaics decrease. Therefore, we freeze pretrained model and add extra epipolar attention, functioning as a supplement for direct cross-frame interaction in an effective way with access to an appropriate amount of noisy conditions along the epipolar lines.

Response to Q5: Multiple CFG

Thank you for your valuable suggestion. We have already cited Dynamicrafter in our submission. The multiple CFGs include camera control and represent a minor inference technique. Following your advice, we will revise it as a minor contribution.

Response to Q3: Soft Mask vs. Hard Mask

We are grateful for the insightful suggestion to apply soft masks and find it to be an excellent idea. Although we have read relevant literature on soft masking, we opted not to incorporate it before to maintain simplicity and avoid introducing additional equations.

The running time remains consistent since all methods utilize dynamic attention masks in current optimized memory-efficient attention frameworks. They are only slightly slower than attention mechanisms without masks.

We define attention bias as $log(exp(-d * temperature ))$ and add attention bias into attention logits, where $d$ represents distance to the epipolar line or plane. The application of soft masks results in only a slight improvement, which may be attributed to the limited time available for adjusting the temperature hyperparameters. We anticipate that with further optimization, soft masks could yield better results.

I have thoroughly read the authors' response as well as the feedback from other reviewers. I appreciate the authors' efforts in addressing our concerns.

Some of the key messages that I find particularly helpful to my major concerns:

• Elaboration on the insight of the clean/noised conditions. To my understanding, by introducing the concept of clean/noised conditions, the authors aim to demonstrate that the widely adopted epipolar attention mechanism is likely one of the most suitable way to handle noised cross-frame interactions. More specifically, applying epipolar constraints across frames could include adequate noised features (large noises) while preventing deterministic information from being obscured by noise. This interpretation is supported by the provided quantitative results and currently makes sense to me.
• Regarding the register token. The authors state in their response that the register token is essential for training stability. Without register tokens, applying epipolar attention becomes infeasible, and thus no ablation.
• Regarding evaluation metrics. From my perspective, the contribution here is not the use of GLOMAP but the commitment to open-sourcing the evaluation code.

Some concerns remain unresolved, and further clarification is needed:

• I notice that the quantitative results in the updated revision differ from those in the original submission. The authors should explicitly clarify this distinction in their response. In Tab. 1, why is there a performance drop in the "DynamiCrafter+CameraCtrl" method? The response to performance improvement seems questionable.
• In the top middle of Fig. 4, what do the "Project" module and its learnable weights refer to?

We sincerely appreciate your valuable feedback.

Response to different quantitative results

To quickly finish the experiments, we change to six machines equipped with eight H100 GPUs and reproduce all the experiments. We also train from scratch 512x320 model for generating visualizations in index.html.

The primary factor contributing to performance differences was that, in the initial draft, our CameraCtrl and MotionCtrl models employed a fixed condition frame index, consistently using the first frame as the reference image. In contrast, our CamI2V model is designed to support any frame as the reference by using a random condition frame index during training.

To ensure a fair and transparent comparison between the methods, the latest version of our submission uniformly employs a random condition frame index across all models. Additionally, to mitigate the influence of camera CFG, we set the camera CFG to 1.0 (no camera CFG) while in our first submission we use camera cfg 1.875, where an overemphasis on camera controllability may result in a substantial decrease in generation quality.

We believe these changes provide a more equitable evaluation of our proposed method and clearly highlight the differences between the approaches.

Response to "Project" Module

We follow the "Projection" module of CameraCtrl, which functions as a linear projection to align the channels of the Plücker embedding output from pose encoder before adding to unet features. In CameraCtrl paper, it is described that "A learnable linear layer is adopted to further fuse two representations which are then fed into the first temporal attention layer of each temporal block" and it can be seen in the pipelilne image(fig 2 in their paper).

I thank the authors for their response and have read through the revised draft.

Regarding W1, the authors reconfirm that their primary contribution lies in the effective modeling of noisy cross-frame interactions. While it is well-motivated and somewhat makes sense, I have to maintain my concern as the current interpretation still seems intuitive and not sufficiently convincing to me. As the primary contribution, more technical insights or findings supporting the proposed concept, such as intermediate observations and informative comparisons, are definitely essential. This is not something that can solved with minor edits and clarification alone but needs restructuring and further investigations.

I'll keep my rating.

Response to More Qualitative Results

Please refer to provided supplementary, and open the index.html for additional generated videos.

Response to Key Difference between CamCo and Ours

We highly acknowledged the contribution of CamCo(arxiv in June) and have correctly cited it in our first submission. We will follow your adivces to further emphasize the difference/improvement between CamCo and our CamI2V. In addition, we will also add citation of CVD and more description about it in our paper.

Please refer to the general response: "To all reviewers: 3. Key Differences Between CamCo and CamI2V (Ours)"

Please refer to the table in general response: "To all reviewers: 1.The Connection Between Our Insights on Conditions and the Proposed Method", where we follow your suggestion to add the quantative results of epipolar attention only on reference frame (similar to CamCo, but camco implements by resampling visible points while we implement by attention mask).

Response to Performance Improvement

Our method demonstrates significant improvements over CameraCtrl, achieving a 32.96% reduction in Rotation Error, a 25.64% decrease in CamMC, and a 20.77% improvement in Translation Error, without decrease in FVD. These results were obtained using Text and Image CFG set to 7.5, 25 steps, and camera CFG set to 1.0 (no camera cfg).

Compared with CamCO-like (arxiv in June) method, we improve 17.08%, 6.61%, 9.00% on RotErr, TransErr, and CamMC without FVD decrease, respectively.

Response to Proposed New Insights about Noised Conditions

Other reviewers have highly acknowledged our insights on clean and noised conditions. We have made substantial efforts to present these new perspectives to share this understanding with the research community. We apologize for not effectively connecting these insights with our proposed method in the initial submission.

Please refer to the general response to all reviewers: 1. The Connection Between Our Insights on Conditions and the Proposed Method for a detailed explanation.

Response to Evaluation Benchmark Improvement

We have identified that current evaluation benchmarks are unreliable, and no open-source evaluation code is available. To address this, we propose several improvements:

• Performing multiple measurements (averaging over five runs) to ensure consistency.
• Conducting extensive evaluations on a large number of samples (1000 samples) for statistical significance.
• Replacing the SfM algorithm with a more robust GLOMAP (new work of COLMAP's author, small improvement) to improve accuracy.
• Computing relative poses with respect to the initial frame to standardize evaluations.
• Introducing normalization to address the issue of unknown global scene scale caused by SfM predictions (the most important improvement).
• Providing SFM with Ground-truth camera intrinsics.

The most important improvement is normalization to address the issue of unknown global scene scale. It's described in RealEstate10K's paper that "there is no way to determine global scene scale, so our reconstructed camera poses are up to an arbitrary scale per clip". Although they attempt to scale normalize the scene scale according to the 5% near depth planes, the camera pose for camera-controlled image-to-video generation is slightly different for different scenes and thus remains relative scene scale. Therefore, it's necessary to calculate normalized (relative) metrics if we can not control the global scene scale.

We also commit to open-sourcing all our training and evaluation code, as well as the checkpoints. Although these modifications are small and straightforward, they hold significant importance for improving reliability and reproducibility in the community.

Final

We sincerely thank you for your time and valuable feedback on our paper. Your insights have been instrumental in improving our work. We have carefully addressed the concerns raised and hope that our rebuttal provides a clearer understanding of our contributions. We wholeheartedly wish that your own submissions receive positive feedback and successful outcomes. We kindly ask that you take the time to thoroughly review our rebuttal.

Regardless of the final decision, we remain committed to open-sourcing all our code and checkpoints to support the community's growth and development. Thank you once again for your consideration and support.

I appreciate the authors' thorough response and additional experiments addressing my previous concerns. After reviewing the rebuttal and other reviews, I have several remaining concerns about this work's technical novelty.

My primary concern relates to the paper's core technical contribution. Epipolar-based attention has been widely adopted in previous image/video generation works ([1], [2], [3], [4]), with [4] specifically implementing epipolar attention on noisy frames. While the authors present the register token as their main contribution, this appears incremental - one could achieve similar results by applying attention to all pixels (or sub-sampled pixels) when the epipolar line falls outside the frame. The authors note that their reproduction of CamCo results in NaN without the register token, which suggests CamCo may already employ similar techniques. The major benefit of register token as I understand is that it can reduce the memory cost to be O(HWL), yet the method still uses attention masks in their implementation which theoretically is still O(HW^2).

However, I do acknowledge two valuable contributions:

This appears to be the first model applying epipolar attention across all frames simultaneously, rather than the one-to-one approach of previous works. Experiments demonstrate this improves camera accuracy. The proposed Glomap-based evaluation could largely benefit the research community if made publicly available.

Overall, while this work makes some improvements to epipolar line-based camera-controlled video diffusion, its technical insights remain limited to me. Therefore, I recommend a borderline reject.

References: [1] Tseng et al., "Consistent View Synthesis with Pose-Guided Diffusion Model" [2] Huang et al., "EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion" [3] Xu et al., "CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation" [4] Kuang et al., "Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control"

Response to Q2: More Qualitative Results

Please refer to provided supplementary, and open the index.html.

Response to Q1: The Connection Between the Proposed Noisy Condition and Epipolar Attention, and the Necessity of These New Insights

We appreciate your question regarding the relationship between our concept of noisy conditions and epipolar attention. Please also refer to our general response titled “1. The Connection Between Our Insights on Conditions and the Proposed Method” for additional context.

Previous methods have introduced epipolar attention primarily with the motivation of enhancing geometric consistency, but these approaches were not specifically designed for settings with inherent noise. In contrast, our work focuses on the unique challenges posed by the noisy feature spaces inherent in diffusion models.

Our motivation is to understand how noise affects cross-frame interactions and to design strategies that mitigate the misleading effects of noise. In clean feature spaces, issues like feature copying due to large receptive fields can be mitigated by increasing model capacity and improving feature extraction. The main challenge in such settings is incorrect attention allocation when the aggregation space is too large.

However, in noisy feature spaces, the situation is different. Due to the added noise, dissimilar features may erroneously receive high attention scores. In this scenario, even powerful models struggle to reconstruct clean pixels from random noise—especially at high noise levels—because the noise can mislead the attention mechanism, resulting in degraded performance.

Our novel insight is that simply increasing the receptive field, such as using 3D full attention, leads to a greater amount of noisy conditions. Due to the randomness inherent in the noise, accessing more noisy conditions does not necessarily reduce uncertainty and thus not necessarily lead to better performance. This understanding suggests that models should focus on aggregating information from more reliable locations to effectively reduce uncertainty caused by noise. The idea of uncertainty introduced by noisy features but not clean features leads to a new insight.

We have found that applying epipolar constraints is one of the most suitable way to prevent the model from being misled by noise. By restricting attention to features along the epipolar lines, the model can interact with more relevant and less noisy information, improving cross-frame interactions in diffusion models. We believe these new insights are significant as they may inspire researchers to develop new architectural designs that better handle noisy conditions in generative models.

Response to Q1: design for epipolar attention when epipolar lines vanish

Please also refer to our general response titled 4. Regarding the Register Token for additional context.

We apologize for not clearly presenting the details regarding this issue earlier. When we attempted to apply epipolar attention without register tokens, we encountered NaN errors during training and were unable to successfully train a model for ablation studies.

Our novelty lies in addressing situations where epipolar lines vanish, which is crucial for robust performance in practical applications. Regarding this scenario, we have observed that previous works on epipolar attention often do not provide special designs or detailed descriptions for handling cases where epipolar lines are absent. This gap motivated us to develop an effective solution.

Response to Multi Resolution Epipolar Mask

Dynamicrafter employs a UNet-based diffusion model with multiple attention resolutions, such as 32×32, 16×16, 8×8, and 4×4 for input images at a resolution of 256×256. When integrating epipolar attention as a plug-and-play module, we freeze the original spatial and temporal attention mechanisms and add extra epipolar attention at these various latent attention resolutions.

Additionally, to efficiently train models at higher resolutions (512×320 and 1024×576), we choose not to add epipolar attention at certain latent resolutions. Specifically, for the 512×320 model, epipolar attention is replaced by 3d full attention (extremely memory-efficient due to optimiation of flash-attn) at the 32×32 resolution, and 32×32 and 16×16 resolutions for 1024×576 model. This selective integration allows for minimal memory increase, enabling us to perform inference on the 1024×576 model with only 18 GB of memory, little memory increase compared to dynamicrafter baseline.

This flexible approach ensures that our method remains memory-efficient while maintaining high performance across different image resolutions.

Response to Register Token

Please refer to the general response "to all reviewers: 4.Regarding the Register Token."

When epipolar line exists but is far away from the queried frame, the zero length kv tokens in memory-efficient attention without register tokens leads to NaN during training, making it impossible to train a model. In this case, two added register tokens serve as placeholder to ensure the non-zero length of kv tokens.

Despite utilizing only two register tokens is insufficient, it ensures the model's basic functionality and only part of pixels on the current frame who have no epipolar line degrade to CameraCtrl without epipolar attention (it can also be considered as a kind of epipolar attention as in this situation, it is appropriate to aggregate no features under epipolar constraint).

Now we apply the token-level register token (only two tokens are added like cls token) but image-level register token can also be applied. For example, in image-to-video generation, the pixels of the reference image can always be seen and serve as image-level register tokens. However, this may also lead to dynamics decrease and it can be seen from the results of 3D full attention (too many noisy conditions) and epipolar only on reference frame (dynamics decrease due to always copying from reference frame, seen in videos of supplementary).

Response to More Qualitative Results

Please refer to provided supplementary, and open the index.html.

Response to Q1: The Connection Between the Proposed Noisy Condition and Epipolar Attention

Please also refer to our general response titled “1. The Connection Between Our Insights on Conditions and the Proposed Method” for additional context.

Our motivation is to understand how noise affects cross-frame interactions and to design strategies that mitigate the misleading effects of noise. In clean feature spaces, issues like feature copying due to large receptive fields can be mitigated by increasing model capacity and improving feature extraction. The main challenge in such settings is incorrect attention allocation when the aggregation space is too large.

However, in noisy feature spaces, the situation is different. Due to the added noise, dissimilar features may erroneously receive high attention scores. In this scenario, even powerful models struggle to reconstruct clean pixels from random noise—especially at high noise levels—because the noise can mislead the attention mechanism, resulting in degraded performance.

Our novel insight is that simply increasing the receptive field, such as using 3D full attention, leads to a greater amount of noisy conditions. Due to the randomness inherent in the noise, accessing more noisy conditions does not necessarily reduce uncertainty and thus not necessarily lead to better performance. This understanding suggests that models should focus on aggregating information from more reliable locations to effectively reduce uncertainty caused by noise. The idea of uncertainty introduced by noisy features but not clean features leads to a new insight.

We have found that applying epipolar constraints is one of the most suitable way to prevent the model from being misled by noise. By restricting attention to features along the epipolar lines, the model can interact with more relevant and less noisy information, improving cross-frame interactions in diffusion models. We believe these new insights are significant as they may inspire researchers to develop new architectural designs that better handle noisy conditions in generative models.

Response to Cross-View Consistency

We appreciate the suggestion to evaluate cross-view consistency, recognizing it as a valuable metric for assessing our model's performance. Currently, we follow MotionCtrl to calculate CamMC, CameraCtrl to determine Rotation Error (RotError) and Translation Error (TransError), and CamCo to transfer relative camera poses. However, these methods do not directly measure cross-view consistency. While we acknowledge the importance of cross-view consistency, we are uncertain about the most appropriate metrics to apply. We would greatly appreciate any recommendations on suitable metrics.

In our current evaluation, cross-view consistency is implicitly reflected in the qualitative results of the generated videos. For quantitative assessments such as CamMC, RotError, and TransError, we employ an SFM-based approach to identify similar key points and perform bundle adjustment to optimize camera poses. This process inherently involves a form of cross-view consistency by ensuring that key points are consistently tracked across frames. Consequently, improved camera controllability often correlates with enhanced cross-view consistency.

Our method demonstrates significant improvements over CameraCtrl, achieving a 32.96% reduction in Rotation Error, a 25.64% decrease in CamMC, and a 20.77% improvement in Translation Error, without any decrease in FVD. These results were obtained using Text and Image CFG set to 7.5, 25 steps, and camera CFG set to 1.0 (no camera cfg).

Response to Key Difference between CamCo and Ours

Please refer to the general response: "To all reviewers: 3. Key Differences Between CamCo and CamI2V (Ours)"

To quickly finish the experiments, we change to six machines equipped with eight H100 GPUs and reproduce all the experiments. We also train from scratch 512x320 model for generating visualizations in index.html. All experiments are conducted on the same baseline with same seed by changing only a few settings in config file to ensure a fair comparison. The results are tested with the same seed on the same 1k samples of test set.

The primary factor contributing to performance differences was that, in the initial draft, our CameraCtrl and MotionCtrl models employed a fixed condition frame index, consistently using the first frame as the reference image. In contrast, our CamI2V model is designed to support any frame as the reference by using a random condition frame index during training.

To ensure a fair and transparent comparison between the methods, the latest version of our submission uniformly employs a random condition frame index across all models. Additionally, to mitigate the influence of camera CFG, we set the camera CFG to 1.0 (no camera CFG) while in our first submission we use camera cfg 1.875, where an overemphasis on camera controllability may result in a substantial decrease in generation quality.

We believe these changes provide a more equitable evaluation of our proposed method and clearly highlight the differences between the approaches.