Mitigating Mode Collapse in Sequential Disentanglement via an Architecture Bias

Dear Reviewer WJJF,

Thank you for your thorough review and the critical points you have raised. We value your insights and provide the following clarifications and responses to your concerns:

• We would like to address your observation regarding the latent space of static and dynamic features. In our approach, both feature types are indeed encoded within the same latent space. This shared latent space is fundamental to our model’s design, allowing for the direct comparison and manipulation of static and dynamic features. The process we describe, where static features are considered in the computation of dynamic features, is intended to explicitly separate the influence of time-invariant factors from time-variant ones. The operation to “subtract” static from dynamic features is a conceptual simplification. In practice, this is achieved through a designed inductive bias within the model architecture, ensuring that the static features do not “leak” into the dynamic representations. We fully understand that subtracting static content from dynamic factors may seem naive. By subtracting the static representation s, which is less prone to dynamic variation, we facilitate the LSTM to concentrate solely on dynamic differences, thus streamlining its function. To empirically validate this approach, we included a series of experiments (table 4 and table 8) in the paper. These experiments specifically demonstrate that removing the subtraction operation or the loss term that penalizes the static component’s presence in dynamic factors leads to a significant drop in performance. These results substantiate the necessity and effectiveness of our subtraction method, confirming that it plays a critical role in enhancing the model’s ability to capture and separate time-varying elements accurately. Without this step, the LSTM module’s capacity to discern complete dynamic changes is notably diminished, as reflected in the lower results.

• We appreciate your query regarding the visualization of static and dynamic embeddings in Figure 2. Your point touches on a critical aspect of our methodology—the representation of static and dynamic features within the same dimensional space. In our approach, both static and dynamic features are encoded into latent vectors of the same dimensionality. This allows us to compare and contrast these features within the same embedding space, which is crucial for our analysis of disentanglement. The static features, derived from a single frame, capture the invariant aspects of the data, while the dynamic features, obtained from a sequence, encapsulate the time-varying attributes. For visualization purposes in Figure 2, we average the dynamic features across the sequence to reduce them to a single representative vector, comparable in dimension to the static features. We then employ t-SNE, to project these high-dimensional vectors into a two-dimensional space for ease of visualization and interpretation. This method allows us to observe how well the static and dynamic features are separated or entangled. The reason for plotting static and dynamic features together is to demonstrate their relationship and the effectiveness of our model in disentangling them. It is to visually confirm that the static features do not influence the dynamic representation and vice versa, which aligns with our goal of achieving clear disentanglement between invariant and variant factors. We will ensure that the revised manuscript includes a more detailed explanation of the dimensionality and handling of static and dynamic features, as well as the rationale behind their joint visualization.

• Representation-Level Evaluation and Comparison with Class-Content Disentanglement Methods: We acknowledge the potential insights such a comparison could offer. However, our work is centered on sequential disentanglement, which inherently diverges from static disentanglement in its fundamental assumptions, modeling approach, and the applicable evaluation benchmarks. Sequential disentanglement is tailored to unravel time-related changes and maintain a consistent representation of static features over time, while class-content disentanglement typically addresses static representations where temporal dynamics do not play a role. The methodologies and tools we have developed are thus specifically designed for the challenges and nuances of sequential data. Given this significant conceptual and methodological difference, a direct comparison with class-content disentanglement methods may not adequately reflect the unique contributions and the efficacy of our approach.

• We recognize that the term ‘mode collapse’ is typically associated with unconditional generative models. We used this term to draw an analogy to the concept of representation degeneration; however, we see how this could be confusing. We will consider alternative terminology that more accurately describes the challenges in disentanglement in sequential settings.

Dear Reviewer gfeQ,

Thank you for your thoughtful review and for highlighting the clarity of the manuscript as well as the rationale behind our approach to disentangle static and dynamic factors.

• We leverage the fact that in sequential data, a single frame in a series has no time-dependent information. Therefore, a single frame will not encode dynamic features but only static features. Ablation experiments were conducted to assess the impact of utilizing different frames, and the results unequivocally demonstrated that employing the initial frame yields the most favorable outcome. We acknowledge the scenario you described, where the first data point might be more entangled, could be problematic. To mitigate this, we propose an additional check of different frames for the static factors, which we included in our paper. Our results (table 4 and table 8) show that even in cases where the first frame may not be ideal, our method still successfully disentangles the factors, as evidenced by the strong performance across different checkpoints in a sequence. This suggests that the model does not merely rely on the first frame but rather learns a robust representation of time-invariant features that are consistent across the entire sequence.

• Regarding the subtraction method outlined in Equation 8, your concern is valid. if $s$ erroneously contains dynamic information, this could indeed lead to a loss of dynamic content. To mitigate this risk, our model is designed to ensure that $s$ is as pure a representation of static features as possible. We acknowledge your concern regarding the subtraction methodology. To validate its efficacy, we conducted experiments (table 4), detailed in our paper, that compare the performance with and without the subtraction of the static component (ablation study). These experiments clearly show that excluding the subtraction step significantly reduces the model’s performance, highlighting the method’s importance in achieving effective disentanglement of static and dynamic features.

• We understand your concern that subtracting the static part from the dynamic might sound too naive. We believe that the simplicity we bring is what makes our method unique and the empirical experiments (table 4) we performed indeed prove that this subtraction is significant to the results of the model on the benchmarks.

• Small time-varying vectors are captured by the dynamic LSTM module, which is trained to focus on changes between frames after the static component has been subtracted. This is designed to be sensitive to even minimal dynamic variations.

• About your question about the length of the time series affecting the model’s performance, we have indeed tested our model on datasets with varying sequence lengths and found that our approach consistently achieves state-of-the-art performance across all of them. At the same time, since the perception of the dynamic parts in our model depends on LSTM, for particularly long series the model may fail due to the use of the LSTM network.

We hope these explanations address your concerns, and we appreciate the opportunity to clarify these aspects of our work. We believe our method contributes meaningfully to the field of disentangled representation learning, and we are open to further questions or suggestions you might have.

Dear Reviewer MtBA,

Thank you for your detailed review and for acknowledging the motivation and structure of our paper. We appreciate your feedback on the challenge of disentangling time-invariant and time-dependent factors in time series data, particularly in high-dimensional datasets like videos.

• We leverage the fact that in sequential data, a single frame in a series has no time-dependent information. therefore, a single frame will not encode dynamic features but only static features. Ablation experiments were conducted to assess the impact of utilizing different frames, and the results unequivocally demonstrated that employing the initial frame yields the most favorable outcome. We acknowledge the scenario you described, where the first data point might be more entangled, could be problematic. To mitigate this, we propose an additional check of different frames for the static factors, which we included in our paper. Our results (table 4 and table 8) show that even in cases where the first frame may not be ideal, our method still successfully disentangles the factors, as evidenced by the strong performance across different checkpoints in a sequence. This suggests that the model does not merely rely on the first frame but rather learns a robust representation of time-invariant features that are consistent across the entire sequence.

• Our empirical experiments reveal that employing the 'average' encoding method leads to the incorporation of undesirable dynamic features, exacerbating the mode collapse problem. The current model architecture, involving arithmetic subtraction operations, inadvertently discards essential 'dynamic' information, resulting in unintended information leakage. In contrast, when coding static information using a single frame within our architecture, we ensure the exclusion of dynamic information. This approach offers a more effective solution, mitigating the risk of information leakage and addressing the limitations associated with the 'average' encoding technique.

• Our approach involves independently establishing a representation for the initial frame, irrespective of its level of entanglement with subsequent frames. Regardless of the degree of entanglement, we consistently learn the static aspect, with the dynamic aspect consistently determined to be zero for the chosen frame. Consequently, it can be inferred that the remaining frames are encoded with respect to the selected anchor frame. This method streamlines the learning process by allowing the network to concentrate on coding the dynamic component, as the static part has already been encoded. Moreover, our model exhibits superior results compared to other models, as demonstrated by our findings.

We believe that these revisions will substantially improve the manuscript and provide a clearer understanding of our model’s contributions. We thank you once again for your valuable feedback.

Dear Reviewer nhge,

Thank you for your thoughtful review and the insights you have provided. We are grateful for the positive remarks about the motivation and clarity of our model, as well as the extensive testing conducted. We also appreciate your constructive feedback, which we address below:

• We acknowledge that the clarity of Figure 4 can be improved. We intended to illustrate the swapping of expressions while keeping the identity constant. To clarify, we will revise the figure caption to explicitly state what elements are swapped and ensure that the visual representation is more apparent.

• Thank you for pointing out the performance improvements shown in Table 1. In the paper we detail under 'Failure case analysis on MUG' why this dataset is difficult because some of the facial expressions, such as surprise and fear, look very similar. It is hard to tell them apart, even for people. Thus, little improvements are important. This means our model is really good at spotting the tiny differences in these expressions. We will take care to explain this better in our article, and see why these small steps forward are important.

• Figure 6 demonstrates the model’s ability to autonomously detect and categorize patterns within the static seasonal context of the Air Quality dataset, notably distinguishing between days with different rainfall levels. While seasonality provides a consistent background throughout the year, our model is not limited by this uniformity. It adeptly clusters days with similar weather patterns, such as grouping rainy days, which is significant as it was achieved without explicit instruction to do so. This self-guided clustering, evident in the figure, showcases the model’s nuanced comprehension of static data components. We will improve figure 6 description in our paper.

• We acknowledge its importance and in our current work, we have primarily relied on theoretical foundations and qualitative analysis to demonstrate our model's capability in mitigating mode collapse. While further empirical comparisons could be insightful, we believe the existing evidence sufficiently supports our claims. We will consider such comparisons in future research.

• We would like to point out an advantage of our model that is relevant to this discussion: our model's flexibility in defining the dimensions of static and dynamic components. Contrasting with previous methods that allocate different dimensions for static and dynamic features, our model designates identical dimensionality to both. This consistent dimensionality is what permits us to apply t-SNE for projecting into a two-dimensional space, enabling the clear comparative visualizations showcased in our work. We would like to point out an advantage of our model that is relevant to this discussion: our model's flexibility in defining the dimensions of static and dynamic components. Other methods developed before us give different dimensions to the static component and the dynamic component. Unlike them, we gave the static factor and the dynamic factor the same dimension, which allows us to use tsnt to project the latent space into two dimensions and present it as seen in our work.

We believe that these revisions will substantially improve the manuscript and provide a clearer understanding of our model’s contributions. We thank you once again for your valuable feedback.