MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition

1. Ablation studies focusing on the layer design

Thanks for your advice!

As discussed in Section 3.5, unlike Jamba, which focuses on fine-grained architectural design, our architecture builds upon CLIP and cannot undergo drastic changes. Instead, we focus on how to maximize the advantages of Mamba without disrupting the original pre-trained weights.

Below, we supplement an ablation on architectural patterns.

• [TM]12: Alternating sequence of Transformer and SSM layers, each repeated 12 times (our current design).
• [T]12[M]12: 12 consecutive Transformer layers followed by 12 SSM layers, with the SSMs functioning as the decoder rather than being inserted as an adapter in the middle of the encoder.
• [T]6[TM]6: No modulation for the first half of the Transformer layers.
• [TTM]6: One modulation layer is inserted between every two transformer layers.

From the table we find that:

• Using MoMa directly as the decoder performs the worst. Under this pattern, the encoder is entirely non-trainable, preventing it from adapting to understand video inputs. As a result, all temporal information must be learned within the lightweight SSM layer, which makes it difficult to capture temporal dependencies effectively.
• Adapting only the latter half of the backbone is sub-optimal, Since learn to recognizing temporal information in the early stages of video encoding is essential for effective processing.
• The [TTMM]6 pattern performs worse than [TM]12, suggesting that a more balanced integration of both components in the adapter structure is more beneficial.

We will add this table into our final version.

2. The benchmark might be outdated

Here we supplement a new experiment by equipping MoMa with MLLMs and answer complicated video understanding tasks.

2.1 Equipping MoMa into MLLMs

Experiment setting

We first train MoMa with CLIP-L backbone on Kinetics-700 for video understanding. Then we apply a VideoLLaMA2-style VLM architecture and train the only projection part on video instruction tuning dataset. We use Qwen2.5-1.5B LLM decoder and LLaVA-Video-178k for training. Finally, we evaluate on a subset ([action_antonym, action_localization,action_sequence]) of MVBench.

• action_antonym: distinguish the correct action from two inversely ordered actions.
• action_localization: determine the time period when a certain action occurs.
• action_sequence: retrieve the events occurring before or after a specific action.

Notably, our training configuration involves significantly fewer resources compared to contemporary VLMs. We use only 0.5M video data for pre-training and 178k video instruction tuning data for fine-tuning, only 5% of the 12M+ video-text data used in VideoLLaMA. This resource-constrained setup inevitably limits the model's generalizability across comprehensive video understanding tasks. Therefore, we choose some action-related subtasks from MVBench since these are most similar with our training resource.

We achieve comparable performance with LLaVA and Video LLaMA while operating with merely 5% of their training data and 21% of their parameter budget (1.5B vs. 7B parameters), demonstrating our efficiency while showing our scaling potential across a broader range of wider application.

3. Include image size in Table 1 & 2

Thanks for the suggestion. For a fair comparison, all input resolution in our Table 1 and Table 2 are 224, unless otherwise specified. (For example, the MViTv2-L (312$\uparrow$) in the sixth row (L284) of Table 1 indicates an input size of 312, which is larger than the default size.) We will clarify this in the final version.

1. Whether CLIP is the best model

Note that our MoMa is an adapter method and is thus backbone-agnostic. We choose CLIP as our backbone following previous methods DiST and AIM for fair comparison.

Here we replace the CLIP backbone with ViT-MAE from facebook, considering that MAE-style pre-training may be more focused on image local feature. The performance drops since the MAE is pre-trained on ImageNet-1K, which is not as large as CLIP. Besides, discussing which image pre-trained model better suits video understanding is out of the scope of this paper.

2. Why different merging method matters so much. Explain Tab. 5 more.

We’ve provided a new ablation study on our SeqMod operation. For more details, please refer to our response to R1(5nFH) #2: Further analysis on SeqMod operation. We hope this clarifies and provides a more comprehensive understanding of SeqMod.

3. Temporal grounding & LVBench

Thanks for the suggestion!

We've add a new experiment by equipping MoMa with MLLMs and answer three complicated video understanding tasks from MVBench:

• action_antonym: distinguish the correct action from two inversely ordered actions.
• action_localization: determine the time period when a certain action occurs.
• action_sequence: retrieve the events occurring before or after a specific action.

We achieve comparable performance on these tasks compared with LLaVA and Video LLaMA while operating with merely 5% of their training data and 21% of their parameter budget (1.5B vs. 7B parameters), demonstrating our efficiency while showing our scaling potential across a broader range of wider application. Please see our response to R4(17be) #2.1 Equipping MoMa into MLLMs for implementation detail.

4. Related works

We here supplement a detailed literature review.

The concept of learning scale and bias for feature adaptation was first introduced in FiLM [1] (Feature-wise Layer Modulation) in 2017. At the same time, [2] implemented this adaptation before the network normalization layer, which led to the term ‘adaptive instance normalization’ (AdaIN). A 2018 survey [3] summarized these concepts as ‘feature-wise transformations,’ which have since been widely applied across a variety of tasks such as image recognition and generative modeling. Thus, AdaN refers to a family of methods that share this underlying principle, including, but not limited to, FiLM, AdaIN, and DiT.

We will clarify the definition of AdaN in the final version for better understanding. Thank you for the suggestion, and please let us know if there are any additional references or concepts we should include.

[1] FiLM: Visual Reasoning with a General Conditioning Layer

[2] Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

[3] https://distill.pub/2018/feature-wise-transformations

1. Divide part is not justified, using higher resolution inputs.

Thanks for advice! Our original $224 * 224$ comparison aimed to align with other baselines, but we’ve now conducted additional ablation studies at $640*480$ resolution (SD video standard) to address your concern.

Experiment details

• Input video resolution $640 * 480 * 16$
• CLIP ViT-B/16 backbone
  • After image embedding layer, the feature shapes $40 * 30 * 16$ (patches).
• Window size is defined on patch level.

(* Experiment conducted on SSv2. Speed tested on A100.)

From the table we observe that the divide operation brings both speed up and performance gain.

• Speed: Divide the window from full to $8*8$ brings significant increase in speed (175% fps) since attention operation dominates the computation. However, further dividing brings marginal improvement when FFN becomes the bottleneck.
• Performance: Conducting full attention on the whole image performs worse than window dividing. That's may because CLIP originally trained on $16*16$ attention sequence. Understanding long sequence is out of its training scope.

2. Ablation of the Mamba state size

I'm not sure whether you're asking for Mamba's state size or the model’s hidden dimension.

For state size, we’ve set to 16 as default. Increasing the state size does not enhance the model’s capacity. Even large-scale models like falcon-mamba-7b ([1]) and Jamba (52B MoE) ([2]) use a state size of 16.

[1] huggingface:falcon-mamba-7b [2] huggingface:Jamba-v0.1

Regarding the model’s hidden dimension, we’ve set to 1024, which aligns with the hidden size used in CLIP. Further increasing the hidden dimension would result in a denser Mamba component, making training more challenging, as it would primarily increase parameters related to MLPs.

We also conducted a comparison to assess the impact of increasing Mamba’s capacity by expanding the hidden dimension. With nearly double the number of parameters, the improvement was marginal.

Additionally, we’ve provided a new ablation on SeqMod operation. Please refer to our response to R1(5nFH) #2: Further analysis on SeqMod operation.

3. Adding a VLM task

Yes. See our response to R4(17be) #2.1 Equipping MoMa into MLLMs for detail.

4. Experimental Designs

4.1 Fine-tuning results on K400 and SSv2

MoMa is an adapter method that conducts parameter-efficient fine-tuning (PEFT). Which means, it always fine-tunes a pre-trained image encoder (eg., CLIP) with a small number of trainable parameters. Therefore, all the experiments in Tab.1-3 are finetuning experiments:

• Tab.1: Fine-tune CLIP on K400
• Tab.2: Fine-tune CLIP on SSv2
• Tab.3: Fine-tune Tab.1 checkpoint on COIN and Breakfast

4.2 Long-video understanding experiment

Sorry for term mis-use. Our MoMa should be clarified as non-e2e method since it conduct PEFT. We will correct this.

For implementation details, both standard and long-video training follows the details in Sec.4. Batch size varies according to the input frame. When input 32 frames, the batch size is 8/gpu.

4.3 Fix misleading table

Thanks for kind reminder!

We used dividing lines to distinguish between non-PEFT and PEFT methods. All PEFT methods are based on CLIP and can be fairly compared with ours. For non-PEFT methods, there are both large-scale data based methods (eg., ActionCLIP) and those based solely on ImageNet (eg., ViViT).

To avoid misleading, we'll update Tab.1-3 to include pre-training datasets, and we'll take CLIP pre-training dataset into account.

4.4 12-Hour training time

Although the CLIP parameters do not require training, they still participate in gradient propagation since the adapter layers are interspersed between the CLIP transformer layers. In fact, for video understanding, 12 hours of training time is already quite fast. For comparison, AIM takes over 16 hours for training with the same number of parameters.

5. Literature review

Thank you for the suggestion. Indeed, there are several excellent linear-complexity sequence models besides Mamba, such as S4, TRecViT and RWKV. We will make sure to include them in the related work section.

6. Typos and unclear notions

We apologies for any confusion caused by the typos and unclear notions. Here are some clarifications:

• The $16*16$ windows are expressed in terms of patches.
• The frame number 16 refers to extracting 16 frames from a video clip for downstream tasks (randomly sampling for training, average sample for inference).
• $y$ or $y_1,y_2$: We modify the SSM module to output two sequences $(y_1, y_2)$ during SeqMod operation. For the baseline operations, the SSM module only outputs a single sequence $(y)$.

1. Marginal improvement on K400 and SSv2 datasets

We focus on striking a balance between performance and efficiency instead of solely pursuing the highest performance. Besides achieving state-of-the-art performance, we managed to cut nearly 30% calculation FLOPs off with minimum training parameters compared with other PEFT methods. Furthermore, our speed advantage becomes increasingly pronounced with more frames while sustains its performance superiority, as demonstrated in Figure 4.

We also conduct another experiment to demonstrate our model's potential by equipping MoMa with MLLMs and answering complicated video understanding tasks. Please see our response to R4(17be) #2.1 Equipping MoMa into MLLMs for detail.

2. Further analysis on SeqMod operation.

We apologize for the omission of a comparison with the raw AdaN method and some explanation in our initial submission. Here we supplement more detailed comparative experiments.

The Raw-AdaN method learns scalar parameters $\alpha_y$ and $\beta_y$ for adaptation. The table shows that the Raw-AdaN already outperforms other methods like Add, Max, etc. Inspired by its promising results, we further design SeqMod in our paper. We here give some detailed discuss.

2.1. Why Raw-AdaN superior Add/Max/Concat, etc.

This is because the feature spaces of Transformer and Mamba do not match or are not perfectly aligned, as observed in [1]. Direct Add/Max/Concat operations interfere with the feature of the original Transformer model.

[1]: ReMamber: Referring Image Segmentation with Mamba Twister

• Add/Max: Adding the output sequence of SSM $y$ to the original features $x$ can lead to information “confusion”, especially when the two features are misaligned or have inconsistent distributions. These two operations can cause information overlap, leading to the loss of key details, especially in multi-scale spatial and temporal features.

• Concat: While it preserves more information by adding learnable linear layer, it does not guarantee that the modification of the sequence is orthogonal.

Different from above, Raw-AdaN uses a global scalar modulation. The scalar acts as a “soft gating” mechanism, where the changes to the sequence’s features are orthogonal to the original CLIP feature outputs. This helps avoid altering the feature distribution of the original CLIP outputs when learning global spatiotemporal representations, reducing interference with pre-trained knowledge.

2.2 Why SeqMod significantly outperform Raw-AdaN

Fine-grained spatiotemporal awareness for video tasks.

Spatiotemporal modeling is the core to video understanding tasks. Though powerful, Raw-AdaN’s global scalar adjustment (with the same scaling and bias parameters shared across all positions) cannot distinguish dynamic changes in different regions, which is essential for video tasks.

By using fine-grained modulation (retaining independent spatiotemporal modulation parameters at each position), we can capture more complex spatiotemporal dynamics, thus preventing information loss.

Experimental results

The table above (lines 5 and 6) shows that extending Raw-AdaN’s global scalar modulation to SeqMod’s fine-grained spatiotemporal modulation significantly improves model performance (both $y_1$ and $y_2$ are vectors generated by SSM).

To sum up:

1. AdaN-like methods bring orthogonal modulation to the original CLIP, making it more suitable for adaptation tasks and avoid altering the feature distribution too much.
2. SeqMod further extends the AdaN-like methods to fine-grained spatiotemporal modulation, which is more suitable for video tasks.

We will add the ablation and the according discussion in our final version.