Smooth and Flexible Camera Movement Synthesis via Temporal Masked Generative Modeling

Q1: Lacks Analysis of Design Choices

Thank you for this valuable feedback. We agree that design justification is essential, and we address this both from a theoretical and empirical perspective.

1. Theoretical Design Rationale

Our design separates long-term and short-term memory handling based on their distinct functional roles:

• Long-term memory encodes macro-level, non-sequential features, such as global musical style or tempo. These features are extracted by compressing and enhancing historical information into a fixed-length representation. Since they carry holistic, non-time-aligned context, we inject them into the generation process using Adaptive Instance Normalization (AdaIN), which excels at style modulation without enforcing temporal alignment.
• Short-term memory, in contrast, maintains direct temporal correspondence with the frames being generated. It encodes local continuity and movement cues that must align tightly with the current output timestep. Therefore, we use cross-attention to inject this information, allowing the model to attend to temporally aligned features for fine-grained control.

This design reflects a key intuition: temporal alignment matters for short-term memory but not for long-term global context. Each injection strategy is carefully chosen to reflect the memory type's characteristics.

2. Empirical Design Validation (Ablation Study)

To validate these choices, we conducted targeted ablation experiments, summarized below:

Key Findings:

• CME Ablations (Cases 1–3):
  • Disabling long-term enhancement or causal masking degrades performance, confirming the importance of hierarchical memory and proper temporal modeling.
  • Excluding long-term memory (Case 3) significantly worsens all metrics, highlighting the value of historical conditioning.
• CMT Ablations (Cases 4–6):
  • Case 5 shows severe degradation when both memories are injected via AdaIN—short-term memory requires temporal alignment, which AdaIN (being global) cannot provide.
  • Case 4 (cross-attention for both) performs slightly worse than the default, suggesting sequential interference when treating long-term memory as temporally structured.
  • Case 6 (sequential injection: long-term then short-term) shows that injecting long-term memory before short-term memory yields stable results, supporting our design choice.

These findings empirically validate our architectural choices and demonstrate that our hybrid memory injection scheme is both theoretically sound and performance-critical.

These analyses and ablation results will be included in Section 4.4 of the revised manuscript.

Q2: The method appears largely built upon existing transformer components; lacks clear novelty or technical difficulty.

Thank you for this thoughtful comment. While our method builds upon established components—as is common in practical machine learning systems—we would like to emphasize that our work introduces a novel problem formulation and tackles non-trivial technical challenges that have not been addressed in prior literature.

1. Novel Task: Unified Online + Offline Camera Movements Synthesis

We propose, for the first time, a unified framework capable of handling both online and offline camera movements generation.

• Existing works focus exclusively on offline settings, where full access to future music and motion sequences is assumed.
• In contrast, our system is designed to operate causally and frame-by-frame, enabling deployment in live performance, gaming, and streaming scenarios.

This dual-capability task formulation is novel and demands a fundamentally different design philosophy, as online synthesis introduces unique latency and causality constraints that prior offline models do not address.

2. Technical Challenge: Balancing Real-Time Latency with Camera Quality

Supporting real-time generation while maintaining high-quality output introduces multiple, interdependent challenges:

• The model must maintain low per-frame latency, excluding heavy iterative approaches like diffusion or autoregressive sampling.
• It must be strictly causal, relying only on past and current inputs—without access to future frames.
• Despite this limitation, it must still produce visually coherent, musically aligned, and semantically relevant camera trajectories.

Addressing these challenges required several novel contributions:

• An innovative training and inference strategy that does not require repeated iterations or resampling, in order to meet the demands of online setting.
• A causally structured memory module (CME) that hierarchically encodes and fuses both short-term and long-term information.
• A hybrid memory injection mechanism, where:
  • Long-term memory (global, non-sequential) is injected using AdaIN to modulate overall style and dynamics.
  • Short-term memory (local, temporally aligned) is injected via cross-attention to retain fine-grained motion continuity.

3. Novel Memory Design and Integration

Our CME module is not a standard transformer encoder. It introduces several non-trivial innovations:

• A two-stream memory structure, where short-term and long-term memory are treated as distinct entities.
• A hierarchical conditioning strategy, where long-term context is used to enhance short-term dynamics.
• A causal-aware short-term encoder that avoids future information leakage—essential for real-time systems, but often ignored in prior work.

This design allows us to capture both global context and local temporal alignment, a key requirement for temporally sensitive tasks like camera control.

Hybrid Memory Injection (AdaIN + Cross-Attention)

Although both AdaIN and cross-attention are established mechanisms, our method is the first to use them in a structured, hybrid way for memory integration:

• AdaIN is ideal for injecting style-level, non-temporal memory (long-term).
• Cross-attention is better suited for fine-grained, temporally aligned memory (short-term).

This asymmetric injection strategy is carefully designed and empirically validated through comprehensive ablation studies (see Q1), which show significant performance drops when either mechanism is misapplied.

Summary

In summary, our paper makes the first attempt to unify online and offline camera synthesis within a single framework. Achieving this required:

• Introducing a new task formulation with practical applications.
• Solving non-trivial architectural and algorithmic problems.
• Producing a deployable system that supports both causal inference and high-quality camera generation.

We acknowledge that our system uses established modules, but the design choices, memory architecture, and training strategies are all specifically engineered for this new and challenging problem. These efforts have led to a practical, extensible, and high-performance camera generation system—the first of its kind for real-time use.

We appreciate the insightful comments and positive support with constructive feedback. We hope our responses address the reviewer's concerns.

Summary:

- 1. We measured the average inference time per frame on three commonly used GPU devices.
- 2. We extended our experiments to two additional datasets beyond DanceCamera3D, covering different domains and task formulations.
- 3. We redesign six ablation variants that systematically remove or alter key components from CME and CMT. 

Q1: Lack of Inference Efficiency Analysis

We conducted a runtime benchmark comparing our method with DanceCamera3D under the same hardware and software settings in the online generation setting. Specifically, we measured the average inference time per frame on three commonly used GPU devices:

Key Observations:

• Our method is about 40× faster than DanceCamera3D across all tested devices.
• Even on older GPUs like the V100, our model achieves ~42 FPS, easily satisfying real-time requirements.

This significant efficiency gain stems from our lightweight transformer design, use of discrete camera tokens, and avoidance of iterative diffusion steps.

Q2: More applications to other scenes

DataDoP dataset is reasonable for our task, but the dataset is not fully public, so we found two other datasets to complete the experiment. The DataDoP work is well-executed and represents a promising direction for future research. After the dataset is released, we will include additional experimental analysis, cite this work, and provide a detailed discussion of its contributions.

To further address the concern, we extended our experiments to two additional datasets beyond DanceCamera3D, covering different domains and task formulations:

a. E.T. Dataset[1]– Text-to-Camera Generation with Varied Trajectories: captured in real-world settings, representing authentic cinematic motion.

We evaluated generation quality using:

• Camera trajectory quality metrics: Frechet CLaTr Distance (FD_CLaTr), Precision, Recall, Density, Coverage
• Text-to-camera coherence metrics: CLaTr Score, Classifier Precision / Recall / F1

b. CCD Dataset[2] – Script-to-Camera Trajectory Generation: simulating everyday environments using game engines.

We used standard metrics:

• FID: Fréchet Inception Distance
• R Precision: retrieval-based alignment
• Div: Diversity
• MM: Multi-modality

Our model demonstrates strong performance across both, highlighting its robustness and generalization across real, synthetic, and diverse domains. We hope these results address your concerns regarding real-world applicability.

[1] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

[2] Cinematographic camera diffusion model.

Q3: Missing Module-Level Ablations (e.g., CME, AdaIN)

Thank you for this helpful suggestion. We have conducted a comprehensive set of module-level ablation studies to evaluate the effectiveness of the Consecutive Memory Encoder (CME) and the Conditional Masked Transformer (CMT) design.

We designed six ablation variants that systematically remove or alter key components from CME and CMT. The results are summarized below:

Key Findings:

• CME Variants (Cases 1–3): Each design change within CME results in noticeable performance degradation, confirming that:

  • Long-term memory enhancement is beneficial for refining short-term features.
  • Causal masking ensures proper temporal modeling for short-term memory.
  • Long-term memory contributes essential historical context for prediction.

• CMT Variants (Cases 4–6):

  • Case 5 shows a significant performance drop when both memories are injected using AdaIN, highlighting that short-term memory requires precise temporal alignment, which AdaIN (as a global modulation method) cannot provide.
  • Case 4 (cross-attention for both) performs slightly worse than our default setup, suggesting that injecting long-term memory via cross-attention introduces unnecessary sequential interference.
  • Case 6 shows that injecting long-term memory before short-term memory yields stable results, supporting our design choice.

These results provide empirical justification for our architecture:

• AdaIN is better suited for injecting non-temporal, global style (long-term memory).
• Cross-attention is more effective for temporally aligned, fine-grained control (short-term memory).
• Our hybrid, structured memory modeling strategy (CME) contributes clearly to generation quality.

Q4: No Comparison of Qualitative Results

In fact, we have included qualitative comparisons in the supplementary material (Section B.3). These comparisons include both:

• Camera movement visualizations (2D projected trajectories, every 40 frames)
• Rendered video results showing full camera-dance-music compositions.

Specifically:

• Videos titled Online_video.mp4 and Offline_video.mp4 compare our method with DanceCamera3D under both online and offline settings.
• The right-hand side shows our method’s output, while the left-hand side presents the baseline results.

These visualizations clearly demonstrate that our method produces smoother, more coherent, and more musically aligned camera trajectories, both in real-time and offline settings.

Q5: No Significant Improvement in Offline Metrics / Online Lower Performance

Thank you for your thoughtful feedback. We would like to address this concern from two perspectives: the online task setting and the offline evaluation results.

1. TemMEGA Is the First Model Supporting Real-Time (Online) Generation

To the best of our knowledge, TemMEGA is the first model that enables real-time, online camera movement synthesis. In contrast:

• Prior methods like DanceCamAnimator require access to future frames or the entire sequence, making them unsuitable for live or streaming scenarios.
• Our approach generates each camera frame using only past and current information, satisfying the causal constraint of real-time applications.

This makes a direct numerical comparison with offline models inherently unfair, since offline models have access to richer information (i.e., full sequence context).

2. Online Performance Is Reasonable Given the Causal Constraint

While it's true that our online results are quantitatively weaker than our offline results, this is expected and reasonable: Online models operate under limited temporal context, without access to future dance or music frames.

Thus, the value of our online model lies not only in quality, but also in latency, deployability, and responsiveness.

3. Offline Performance Demonstrates Flexibility and Generalization

Although our model is designed primarily for online generation, it can be easily adapted for offline settings by modifying MASK token. Our offline experiments demonstrate this flexibility.

Importantly, even in the offline setting, TemMEGA achieves comparable or better performance than prior work:

• FID (Generation Quality) improves by 6% compared to DanceCamera3D.
• DMR (Dancer Missing Rate) improves by 9%.
• LCD (Limbs Capture Difference) improves by 7%.

These results validate that TemMEGA is not only suitable for real-time deployment but also generalizes well to offline generation, showing its potential as a unified and extensible framework.

Q1: More empirical validation and application.

A: Thank you for raising this important concern about the scope and generalizability of our empirical validation. We fully agree that broader experimental coverage is essential to demonstrate the robustness and applicability of the proposed method. We offer the following clarifications and new results:

1. DanceCamera3D Dataset Scale and Structure

While the DanceCamera3D dataset contains 108 complete choreographed sequences, each is relatively long and rich in content. In both training and evaluation, we process these sequences as numerous shorter clips, each representing an independent instance of the online prediction task. This clip-level breakdown results in a significantly larger number of training and test samples—offering sufficient diversity in dancer motion, rhythm patterns, and camera behavior.

Thus, while the number of unique sequences appears small, the effective sample size is much larger, and the model is trained and evaluated under meaningful data diversity.

2. New Experiments on External Datasets

Our task is fundamentally distinct from video-based camera motion estimation methods such as PACE. Rather than analyzing existing footage, our focus lies in generating camera motions for creative applications—specifically, using inputs like music and dance or motion and text to synthesize camera trajectories that support video creation. In contrast, PACE performs post-hoc trajectory estimation by decoupling camera motion from pre-recorded videos, which falls outside the generative scope of our pipeline.

That said, we appreciate your thoughtful perspective. To explore our model’s capacity for estimation, we tested it in a PACE-like setting (as PACE dataset is not directly available): feeding RICH[1] dataset videos into our pipeline and comparing the predicted camera trajectories against those estimated by PACE. Despite not being designed for this task, our model performs competitively, benefiting from CME’s effective temporal encoding and CMT’s strong use of conditional cues.

Additionally, we evaluate our method on datasets beyond synthetic environments:

a. E.T. Dataset[2]– Text-to-Camera Generation with Varied Trajectories: captured in real-world settings, representing authentic cinematic motion.

We evaluated generation quality using:

• Camera trajectory quality metrics: Frechet CLaTr Distance (FD_CLaTr), Precision, Recall, Density, Coverage
• Text-to-camera coherence metrics: CLaTr Score, Classifier Precision / Recall / F1

b. CCD Dataset[3] – Script-to-Camera Trajectory Generation: simulating everyday environments using game engines.

We used standard metrics:

• FID: Fréchet Inception Distance
• R Precision: retrieval-based alignment
• Div: Diversity
• MM: Multi-modality

Our model demonstrates strong performance across both, highlighting its robustness and generalization across real, synthetic, and diverse domains. We hope these results address your concerns regarding real-world applicability.

[1] Capturing and Inferring Dense Full-Body Human-Scene Contact

[2] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

[3] Cinematographic camera diffusion model.

Q2: Lack of Correlation with Human Perception in Metrics

Thank you for pointing this out. We agree that standard quantitative metrics—such as FID, DMR, and Dist—may not fully reflect how humans perceive the quality of generated camera motions.

1. User Preference Study (Section 4.3)

In fact, we have conducted a user preference study, as described in Section 4.3 of the paper. We recruited 21 participants from diverse backgrounds, including dancers, animators, and general users. In pairwise comparisons between TemMEGA and baseline methods:

• 83.4% of participants preferred TemMEGA in the online setting
• 73.6% preferred TemMEGA in the offline setting

This suggests that our method is consistently favored by human observers across use cases.

2. Human Perception Rating Study

To further validate perceptual alignment, we conducted an additional subjective quality assessment with 10 participants, including dancers and animators. Each participant rated the generated videos (rendered from predicted camera paths) across three perceptual dimensions:

• Naturalness: How realistic and smooth the camera motion feels.

• Fluency: How continuous and stable the camera motion transitions are.

• Beat Matching: How well the camera motion aligns with the musical rhythm and dancer.

Each dimension was rated on a scale from 1 to 10. The averaged results are shown below:

These scores confirm that users perceived TemMEGA’s outputs as more natural, smoother, and better synchronized with music and dancer across both online and offline conditions.

These two user-centered evaluations strongly suggest that TemMEGA provides improved perceptual quality from a human perspective, even if some standard metrics show marginal gains. We will add these new results in Section 4.3.

Q3: Lack of Deployment Considerations

Thank you for highlighting this important point. We agree that deployment feasibility, especially in real-time settings, is critical for evaluating the practicality of our method.

To address this, we conducted a runtime benchmark comparing our method with DanceCamera3D under the same hardware and software settings in the online generation setting. Specifically, we measured the average inference time per frame on three commonly used GPU devices:

Key Observations:

• Our method is about 40× faster than DanceCamera3D across all tested devices.
• Even on older GPUs like the V100, our model achieves ~42 FPS, easily satisfying real-time requirements.

This significant efficiency gain stems from our masked training strategy to avoid iterative diffusion steps and our lightweight transformer design.

We will add these results to Section 4.3 in the revised version and provide a clear discussion on runtime and deployment implications in the revised version.

Thank you to the author for the detailed response. All of my concerns have been fully addressed, and the revisions are thorough and well-executed. I strongly recommend accepting this paper.