Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

We appreciate Reviewer a9TX for the valuable feedback and acknowledge our technical contributions and the effectiveness of the proposed method. We address the concerns raised by the reviewer as follows.

W1:Computational Overhead

Answer: Thanks for your comments. It is true that the multi-directional supervision module requires three forward passes, resulting in a slightly higher training time costs and computational requirements. We provide an additional quantitative experiment on NExT-QA dataset to analyze the time cost in the Table below. It can be observed that compared to the backward propagation, the forward pass incurs much lower computational cost, making it well-suited for multiple executions to enhance model performance.

W2:Calculation of Contrastive loss

Answer: We appreciate your comments. The contrastive loss is mainly discussed in the last paragraph of Sec. 3.3. As shown in Eq. (4), for a reweighted prompt vector computed based on the question token matrix and task-aware prompt matrix, its positive sample is the current task type embedding, while its negative samples include both the previous and current task type embeddings. For the first task, the contrastive loss value equals zero because there are no actual negative samples. By defining the contrastive loss in this way, 1) the loss computation is compatible with the first task, 2) it supports dynamically evolving task streams without the need to pre-define the task sequence.

We are sorry for not making it clear in our current version. We will add related discussion to Sec. 3.3.

Q1:How is task type embedding initialized

Answer: Thanks for your comments. Since the task type embeddings themselves are learnable parameters (line 243), we directly use random initialization to initialize them, similar to other parameters in neural networks. We added the implementation details about how the task type embeddings are initialized in Sec. 3.3 in our future revision.

We appreciate Reviewer 9h35 for the positive review and constructive comments. We provide our responses as follows.

W1: Sensitivity or Failure Modes of Contrastive Learning

Answer: Thanks for your valuable question. Since the task embeddings are fully learnable, the inner mechanism of contrastive loss (i.e., closing the positive pairs and separating the negative pairs) can regularize each task embedding to be close to the corresponding task-related query while pushing it away from unrelated task queries, thus promoting better discrimination between different task embeddings. Fig.6 in the supplemental material can prove such separation. This mechanism can mitigate potential failure modes of this loss, such as collapsed or entangled task embeddings. Moreover, Eq. (4) is an additional regularization loss in Bisecle and will not directly control the final prediction of the model. In this case, even if the contrastive loss degenerates into failure modes, the worst-case scenario is that the contrastive mechanism becomes ineffective, without causing a significant negative impact on the final model performance (e.g., worse than the original model).

W2: Requirements of Prompt Injection Layers

Answer: We appreciate the reviewer for the comment. As discussed in our sensitivity experiments (Line 341-349), there is a trade-off between injected layers and final performance. As shown in Table 4 and Answer to Q1, 12&16 injected layers can also result in competitive performance that exceeds all baselines. Meanwhile, even with 32 injected layers, the number of overall learnable parameters is still far less than the LLM backbone (see Table 3), which demonstrates the parameter efficiency of Bisecle from a marco perspective. We hope our explanation can address your concern.

Q1: More Points to Table 4

Answer: We thank the reviewer for the valuable comment. As a response, we add more points (12, 20, and 28 layers) to Table 4, and the added results are shown as follows. As we can see in the results, a general trend is that more injected layers can lead to higher accuracy and lower forgetting rates. When the layer number is larger than or equal to 12, the performance becomes competitive, which demonstrates the parameter efficiency of Bisecle.

Q2: Latency Comparsion

Answer: We appreciate the reviewer for the suggestion. According to your advice, we compared the training time of Bisecle and two variants (including the original model), which is shown in the table below. From the results, we can see that the multi-directional supervision mechanism can lead to longer training time, while contrastive prompt learning only brings minor computational cost. It is reasonable because multi-directional supervision requires end-to-end model training on extra data, leading to increased computational overhead. Compared to the original model, the additional training time is acceptable, while the performance gain is substantial, demonstrating the efficiency of Bisecle.

Q3: Application to MoE LLMs

Answer: Thanks for your inspiring question. While Bisecle can be a plug-and-play module for normal LLMs, it may require several special designs to be adapted to MoE LLMs. The difficulties can lie in how to insert adapter modules into sparsely activated experts without breaking the gating mechanism, and how to ensure efficiency and stability under expert parallelism. Fortunately, several novel approaches, such as MoRA and DynAdapter, have shed light on the application of adapter-like techniques in sparse and expert-based architectures. In this case, we believe that Bisecle can also work with MoE LLMs with minor modification, which can be one of our future directions.

Q4: Experiments on more LLM Backbones

Answer: Thanks for your comments. To examine the generalizability of Bisecle, we further conducted extra experiments on more open-source backbone LLMs, including QWEN-7B and Gemma-7B, with the experimental results shown below. According to the results, we can see that Bisecle consistently leads to the optimal results compared to the baseline model (i.e., LLaMA-Adapter). This observation illustrates the flexibility and generalizability of our proposed method.

We are grateful to Reviewer tR7C for providing insightful feedback. The detailed responses are provided below.

W1: Concern of Neurobiological Problem Reframing

Answer: Thank you for the thoughtful feedback. We agree that many prior works are inspired by neuroscience, and our goal is not to position the neurobiological analogy itself as a novel contribution. Rather, our contribution lies in the structured reframing of continual video-language learning through two key hippocampal mechanisms, i.e., rapid binding and pattern separation, which directly motivate the design of our multi-directional supervision and contrastive prompt learning modules. We take the first step to reframe the MLLM-based continual learning from the perspective of how human's hippocampus works. While we do not claim a precise biological simulation, these modules embody the functional principles of association formation and representation disentanglement observed in the hippocampus. Similar to how neural networks abstractly draw from biological neurons (despite the significant difference between neural networks and real biological neuron systems), our approach uses these neurobiological insights as motivational analogies to guide architectural choices and learning dynamics.

We will add a discussion to further clarify how our neurobiologically-inspired method makes its unique contribution in the current stage of multimodal LLMs and also include an explanation on the contribution compared with other neuroscience-inspired works listed by the reviewer.

W2: Difference between Ours and ColPro

Answer: We appreciate this insightful suggestion. The difference between Bisecle (our method) and ColPro is three-fold: 1) Bisecle incorporates a multi-directional supervision module to handle catastrophic forgetting via additional supervision signals. In contrast, ColPro does not have any design to augment the supervision signals. 2) ColPro introduces a series of prompt learning techniques in their architecture design, but none of them is designed to handle the update conflict issue. Differently, we introduce a contrastive prompt learning mechanism to address the update conflict problem, using different motivations and designs compared to ColPro. 3) From the motivation perspective, the key designs in our method are inspired by neurobiological mechanisms, with carefully discussed motivations, biological mimicking, and tailored architectural designs. In contrast, ColPro is mainly based on other heuristic motivations, such as question constraint, knowledge acquisition, and visual temporal awareness.

We will include this analysis and comparison in our future revision to better demonstrate the difference between our method and the key baseline.

W3: Reason for Improvement by $L_Q$

Answer: We appreciate this valuable insight. We would like to clarify that the $L_Q$ loss is not limited to predicting the question type (e.g., "why" or "how"). Instead, it involves reconstructing the full question content, including the subject, predicate, and object, and these elements that are often implicitly or explicitly grounded in both the video and the answer. This requires the model to learn structured correspondences across modalities and to recover event semantics, such as causality or temporal dependencies, rather than relying solely on surface-level task cues. Therefore, the observed performance gain reflects enhanced multimodal reasoning, particularly in aligning causal factors across video and textual modalities.

W4-1: Single-Run Experiments

Answer: Thanks for your question. We now include error bars (standard deviation over 3 runs with different random seeds) and report the accuracy and forgetting rate in the following table. It can be observed that Bisecle has a relatively low standard deviation in both accuracy and forgetting rate, illustrating its stability and robustness across different runs and low sensitivity to initialization.

We did not report error bars on most experiments for the following two reasons: 1) The reported test accuracy and forgetting rate are already averaged over all previous tasks, which partially reflects the model's stability; 2) Generating error bars would require significant additional computation, which was not feasible due to limited computational resources. Considering its significance, we will report more statistical error bars in our future revision.

W4-2: Performance Improvement on DramaQA

Answer: We appreciate the review for pointing out the problem. DramaQA is a relatively mature and saturated dataset, where many recent methods already reach a high performance ceiling. In such scenarios, even a small improvement can be meaningful. Also, the forgetting rate of our method is significantly lower than the baselines on this dataset. Moreover, Bisecle demonstrates gains over strong baselines like ColPro on all datasets. This supports the generalizability and robustness of our method across different video-language reasoning tasks.

We appreciate Reviewer EdJu for the perception of our contributions and thank the reviewer for the insightful feedback. The detailed responses are provided below.

W1. Complexity of Mechanisms

Answer: We appreciate your point regarding the overhead of Bisecle, especially contrastive prompt learning. We would like to verify its computational efficiency from two aspects. 1) Complexity analysis. Given the contrastive module, the time complexity for each sample is $O(NDT + DT^2)$ (detailed analysis will be provided in our future version), where $N$ is the token number, $D$ is the latent dimension, and $T$ is the task number. Since $T$ is usually far less than $N$ and $D$, while the cost is linear in the sequence length, the overall computation cost of the contrastive module is relatively low and scalable in practice. 2) As shown in the table in the answer to Q1, the contrastive learning module only brings 2.9% (5 min of 171 min) extra time, which confirms its practical efficiency and minimal cost.

Q1. Error Bars and Statistical Significance

Answer: Thanks for your question. We now include error bars (standard deviation over 3 runs with different random seeds) and report the accuracy and forgetting rate in the following table. It can be observed that Bisecle has a relatively low standard deviation in both accuracy and forgetting rate, illustrating its stability and robustness across different runs and low sensitivity to initialization.

We did not report error bars on most experiments for the following two reasons: 1) The reported test accuracy and forgetting rate are already averaged over all previous tasks, which partially reflects the model's stability; 2) Generating error bars would require significant additional computation, which was not feasible due to limited computational resources. Considering its significance, we will report more statistical error bars in our future revision.

Q2: Clarification of Mechanisms

Answer: Thank you for the insightful question. In terms of model size, the results in Table 3 show that Bisecle is compatible with LLMs ranging from 1B to 13B, introducing only a small number of additional parameters and computational cost. In terms of task complexity, it is usually correlated with the length of videos and the length of texts, which are both approximately proportional to the number of tokens. As the complexity of all modules in Bisecle is typically either independent of or linear with respect to this number, the additional computational cost introduced by task complexity is aligned with that of the transformer-based LLM itself. To sum up, Bisecle enjoys comparable scalability to the LLM backbone with respect to both model sizes and task complexities.

Q3: Computational Efficiency

Answer: We appreciate the reviewer for the suggestion. According to your advice, we compared the training time cost of Bisecle and two variants (including the original model), which is shown in the table below. From the results, we can see that the multi-directional supervision mechanism can lead to longer training time, while contrastive prompt learning only brings minor computational cost. It is reasonable because multi-directional supervision requires end-to-end model training on extra data, leading to increased computational overhead. Compared to the original model, the additional training time is acceptable (<2x), while the performance gain is substantial (33.8%), demonstrating the efficiency of Bisecle. Due to the limitations of time and space, we did not report the running time for more baselines and datasets, and these will be provided in our future revision.

C1: Consistent Use of Bold and Italics

Answer: Thank you for pointing this out. We will correct and standardize all bold/italic usages in the final version.