MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation

To Reviewer Yz76:

Q1: Suggestions on providing the complexity and scalability analysis.
A1: Thank you for the insightful question. Beyond generation latency, we agree that analyzing model complexity and scalability is essential, and we make following clarifications:

1. In practical deployment scenarios, runtime latency is typically the most critical metric, which is why we reported it in Line 279 (Section 4.2). MEGADance achieves 0.19s per second of generated dance, enabling real-time applications.
2. As you pointed out, complexity analysis is also important. Thus, we include a comparison of complexity for generating a 1024-frame (34.13s) dance sequence. Note that MEGDance activates only the relevant experts rather than all experts when computing "Parameters". All "Run Time" evaluations were conducted on an RTX 3090 GPU with an Intel Xeon Gold 5218 CPU. As summarized below, MEGDance has slightly higher complexity than FineNet but substantially better generation quality (refer to Tables 1-3 for details). Compared to Bailando++ and Lodge, it has lower latency and fewer parameters while maintaining superior performance. These results demonstrate MEGDance's favorable trade-off between efficiency and quality. We will include this analysis in the revision. | Model | Parameters↓ | Run Time ↓ | |-------|-------------|------------| | Bailando++ | 152M | 5.46 s | | FineNet | 94M | 3.97 s | | Lodge | 235M | 4.57 s | | MEGDance | 120M | 4.31 s |
3. Regarding scalability with dataset size, MEGADance exhibits favorable scaling properties. Its two-stage structure benefits from increased training data: the FSQ-based quantization achieves better codebook coverage, while the genre-aware MoE design enables each expert to specialize more effectively. Moreover, the sparse routing in our MoE ensures that only one expert is activated per input, keeping computational cost nearly constant regardless of the total number of genres. We have evaluated performance of MEGADance on two large-scale datasets (FineDance and AIST++), and observed stable and generalizable performance across varied genre distributions, as shown in Tables 1 and 2.

Q2: Suggestion on sharing the code.
A2: Thank you for the helpful suggestion. We will release the MEGDance code to benefit the community. As NeurIPS policy prohibits external links in the rebuttal, the code will be made publicly available in the final revision.

Q3: Concerns about the generalization ability of MEGADance to unseen genres
A3: Thank you for the insightful question. As discussed in Section 3.3.1 (Lines 176-192), MEGADance adopts a Mixture-of-Experts (MoE) design where Specialized Experts are selected based on discrete genre labels. This modular architecture inherently supports genre scalability. Specifically: (1) The General Expert and existing Specialized Experts can be retained; (2) A new Specialized Expert can be added and trained solely on a subset containing the new genre; (3) Only the genre-to-expert mapping in the routing policy needs to be updated—no structural changes are required.
To validate this capability, we conducted an additional experiment: we removed Dunhuang and Breaking from the original 16-genre training set, and then reintroduced each genre using the extension strategy stated above. As shown below, the performance gap compared to full retraining is negligible, demonstrating MEGDance's ability to handle genre extension effectively without requiring full retraining. We will include this result in the revision.

Q4: Question on MEGADance's ability to generate dances that blend multiple genres.
A4: Thank you for the insightful question. Here is our answer:

1. As you suggested, all experts in MEGDance share a unified codebook, so the latent tokens (i.e., FSQ indices) produced by different Specialized Experts reside in the same space. This allows direct addition, interpolation, and fusion of latent representations, making mixed-genre dance generation a viable and promising direction.
2. Furthermore, MEGDance’s autoregressive generation scheme ensures strong motion continuity, even when switching genres mid-sequence (e.g., from Hangtang to Breaking). As NeurIPS policy prohibits external links in the rebuttal, we will include corresponding visualizations in the revision.
3. To evaluate this capability—an essential aspect of genre controllability—we conducted an additional user study using the same music and participant setup as shown in Section 4.2. Specifically, the first half of each sequence was conditioned on genre1 and the second half on genre 2, where genre pairs were randomly sampled. Participants rated the results on a 5-point scale for Style Quality (SQ: alignment with target genre) and Motion Quality (MQ: smoothness and naturalness). As summarized below, MEGDance achieves high MQ and clearly outperforms other methods in SQ for mixed-genre generation. We will include this in the revision. | Model | SQ | MQ | |-------|----|----| | FineNet | 2.81 | 4.12 | | Lodge | 3.14 | 4.27 | | MEGADance | 4.58 | 4.25 |

Q5: Concerns about the absence of variance reporting in experiments.
A5: Thank you for the insightful question. Here is our answer: (1) In music-to-dance generation, prior works [1,2] typically report mean performance without variance metrics, as evaluations are based on deterministic inference under fixed settings. We followed this convention to ensure comparability. (2) However, we agree that reporting variance is valuable. We now additionally provide standard deviations of our main evaluation metrics below. We will include it in the revision.

[1] Li, Ronghui, et al. "Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[2] Li, Ronghui, et al. "Finedance: A fine-grained choreography dataset for 3d full body dance generation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Q6: Suggestions the clarity and readability of Fig. 2.
A6: Thank you for the helpful suggestion. We will revise the caption of Fig. 2 to improve clarity and readability, as follows:
Figure 2: Overview of MEGADance. Stage 1 (HFDQ) quantizes dance into upper/lower-body latent codes using FSQ with kinematic and dynamic reconstruction constraints. Stage 2 (GADG) maps music to codes via an $L$-layer Mixture-of-Experts architecture (blue: specialized, pink: universal), where each expert is equipped with a Mamba-Transformer hybrid backbone.

To Reviewer 1pZx

Q1: Concerns on the evaluations on AIST++.
A1: Thank you for the helpful suggestion. We clarify as follows:

1. Table 2 contains a typo and it actually represents $DIV_k$ and $DIV_g$.
2. FineDance, as the largest dataset with more diverse genres, already provides strong evidence of MEGADance's superiority through its competitive $FID_s$ and $DIV_s$ results. And most baseline methods on AIST++ do not incorporate genre conditioning (except Lodge), making direct comparisons on $FID_s$ and $DIV_s$ potentially unfair.
3. Following your suggestion, we additionally report AIST++ results on $FID_s$ and $DIV_s$. As summarized below, MEGADance substantially outperforms all baselines. We will include this in the revision.

Q2: Suggestions on providing the computational cost comparison or ablation.
A2: Thank you for the helpful suggestion. We make clarification as follows:

1. In practical applications, users usually prioritize inference latency (reported in Line 279, Section 4.2). MEGADance achieves 0.19 seconds per second of generated dance, enabling real-time responsiveness.
2. We agree that providing additional computational cost analysis, such as Parameters and Run Time comparisons, helps clarify the efficiency benefits of our design. We now include a comparison for generating a 1024-frame (34.13s) dance sequence. Importantly, due to sparse expert activation, MEGADance only activates one Specialized Expert per input, which significantly reduces the effective computation during inference. All “Run Time” evaluations were conducted on an RTX 3090 GPU with an Intel Xeon Gold 5218 CPU. As summarized below, MEGDance has slightly higher complexity than FineNet but substantially better generation quality (refer to Tables 1-3). Compared to Bailando++ and Lodge, it achieves lower latency and fewer parameters while maintaining superior performance. These results reflect MEGDance's favorable trade-off between efficiency and quality. We will include this analysis in the revision.

3. Since MEGADance already achieves real-time performance, our ablation study primarily focuses on generation quality and controllability rather than parameters or runtime, as these aspects are more critical for evaluating the effectiveness of our design—particularly in genre-aware choreography generation. However, we agree that computational efficiency is an important perspective and will discuss it in the revision.

Q3: Suggestions on the statement of Eq. 3.
A3: Thank you for the helpful suggestion. In Eq. 3:

• S′ and S″ denote first- and second-order derivatives of S
• J′ and J″ denote first- and second-order derivatives of J
  We will clarify these in the revision.

Q4: Question about the exposure bias in autoregressive manner.
A4: Thank you for the insightful question. Here is our response:
1.In practice, exposure bias does not always have a significant negative impact in motion generation tasks, especially when strong temporal models (e.g., Transformer or Mamba) and well-structured conditions are employed—models trained with cross-entropy loss can still produce high-quality sequences.
2. Following your suggestion, we implemented Scheduled Sampling (SS) to mitigate exposure bias. As shown in the table below, some metrics even slightly degrade with SS. We will include this discussion in the revision.

To Reviewer oRpN:

Q1: Concerns on the novelty of MEGADance.
A1: Thank you for the insightful comment. While autoregressive Transformer-based token generation is indeed a common paradigm, MEGADance introduces several key architectural distinctions beyond simple module application:

1. Structural Decoupling of Semantics. Unlike monolithic sequence modeling in prior work, we explicitly decouple choreographic semantics into: Genre-invariant rhythmic structure (modeled by Universal Expert) and Genre-specific stylistic priors (modeled by Specialized Experts). This reflects real-world choreographic composition and improves both genre control and generalization. Ablation (Table 4a): Removing either of these two structures significantly degrades genre fidelity or structural coherence.
2. Task-Specific Modular Routing via MoE. MEGADance uses a genre-conditioned hard routing policy where each sample activates a single genre-specific expert, which differs fundamentally from Conventional sparse MoE (e.g., for language modeling) where activation is learned. Our approach encodes task-specific inductive bias and prevents expert collapse.
3. Hybrid Modeling of Cross/Intra-modal Dynamics. We replace homogeneous Transformers with a Mamba–Transformer hybrid: Mamba captures fine-grained intra-modal continuity (especially motion), and Transformer handles long-range cross-modal alignment (music and dance). This addresses Transformers' lack of inductive bias for temporal locality. Ablation (Table 4a): Hybrid modeling design improves generation quality and synchronization.
   We will further clarify these points in the revision.

Q2: Suggestions on the complexity analysis.
A2: Thank you for the helpful suggestion. We make following clarification regarding this issue:

1. Runtime Latency Focus . In practical applications, user priority is run-time latency (reported in Line 279, Section 4.2). This remains our primary efficiency metric.
2. Complexity Analysis Expansion. As suggested, we now include comparison for generating a 1024-frame (34.13s) sequence. MEGADance activates only relevant experts (not all) during computation. All benchmarks run on RTX 3090 GPU + Intel Xeon Gold 5218 CPU. As summarized below, MEGDance has slightly higher complexity than FineNet but substantially better generation quality (refer to Tables 1-3). Compared to Bailando++ and Lodge, it has lower latency and fewer parameters while maintaining superior performance. These results reflect MEGDance's favorable trade-off between efficiency and quality. We will include this analysis in the revision.

Dear reviewer,

Please read the rebuttal and start discussion with authors.

AC

To Reviewer xz7a:

Q1: Questions about the experimental result of Bailando++.
A1: Thank you for the insightful question. Here is our answer:

1. The reported results for Bailando++ are obtained using the publicly available official code, which we retrained and evaluated following the instructions. Despite our efforts to replicate the original experimental setup, we were unable to reproduce the exact numbers reported in the paper. Therefore, we report our evaluation results for fair and consistent comparison across all baselines under the same environment. We will explicitly annotate in the revised table that these numbers are from our reproduction.
2. Regarding your question about genre awareness: it is true that Bailando++ does not incorporate genre conditioning, which may partially explain its lower performance compared to genre-aware methods like MEGADance. However, this difference in model design does not account for the discrepancy between our reproduced results and those reported in the original paper. We will clarify it in the revision.

Q2: Typos in the paper.
A2: Thank you for pointing them out. The following is our response:

1. We acknowledge the typo in Table 2—column headers should be $DIV_k$ and $DIV_g$.
2. The $DIV_g$ value of FACT in Table 2 should indeed be bolded.
3. We will standardize citation spacing throughout the paper.
4. We would like to point out that Figure 4 does not contain subfigures (a) and (b).

These typos will be corrected in the revision.

Q3: Concerns on whether specialized and universal experts truly capture distinct roles as intended.
A3: Thank you for the insightful question. Here is our answer:

1. $FID_k$ and $FID_g$ primarily reflect universal dance quality, while $FID_s$ captures genre-specific fidelity. As shown in Table 4, removing the Universal Expert (UE) notably degrades $FID_k$ and $FID_g$, whereas removing the Specialized Expert (SE) leads to a sharp drop in $FID_s$—indicating their distinct contributions.
2. We further conducted a t-SNE analysis on the final-layer features of SE and UE. Visualizing genre-specific color-coded clusters, SE features form well-separated clusters while UE features do not—empirically supporting their intended functional distinction. As figures are not allowed during the rebuttal phase, we only provide this textual description.

We will include both figure and explanation in the revision.