State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding

W1 & Q1 Response: What is the specific technical novelty of the proposed approach beyond incremental improvements, and how does it effectively learn discriminative spatio-temporal information? What is the actual mechanism of "spreading gathered local spatial information temporally"?

Thank you for your valuable review. We highlight the following aspects to illustrate the novelty and operational mechanism of our approach:

(1) Firstly, unlike existing fine-tuning methods, our approach is specifically designed for Mamba architecture models.

• Through the low-rank convolution of the IFG module, we learn discriminative spatial features and superimpose the aggregated local spatial information onto tokens. The IFG module effectively introduces two-dimensional spatial inductive bias into sequential Mamba models while requiring only ~3%(2.41M/74.00M) of tunable parameters compared to full fine-tuning, as shown in Table 2, enabling effective spatial feature aggregation and learning.

• Leveraging the inherent characteristic of Mamba models to sequentially compress tokens, our IFS generates inter-frame prompts through global attention using sampled tokens from each frame, achieving temporal propagation of local spatial information in this process. As discussed in Sec 6.1 of the appendix, the IFS module significantly reduces the path length for information transmission, facilitating the flow of temporal information.

(2) Secondly, the proposed IFG and IFS work in a complementary manner to learn discriminative spatio-temporal information.

• As mentioned above, the intra-frame prompts generated by IFG are superimposed onto the tokens, enabling intra-frame tokens to acquire discriminative local spatial information.

• IFS conducts low-rank attention computation on sampled tokens, allowing the aggregated spatial information from each frame to interact and generate inter-frame prompts.

• The inter-frame prompts are inserted at the boundaries between frames and fed into the next Mamba layer, enabling the Mamba layer to simultaneously model the local spatial information contained in intra-frame tokens and the global temporal information contained in inter-frame prompts when processing tokens.

(3) We clarify the spreading mechanism of gathered local spatial information through the IFG and IFS modules as follows:

• Firstly, the IFG module aggregates spatial information from each frame into intra-frame prompts and superimposes the intra-frame prompts onto tokens. Secondly, the IFS module performs global attention computation through sampled tokens, enabling global interaction of the aggregated spatial information to generate inter-frame prompts. Thirdly, the inter-frame prompt corresponding to each frame thus contains spatial information from other frames, thereby propagating the aggregated local spatial information in a temporal manner.

• As shown by the visualization study in Figure 9 in the appendix, the intra-frame prompts generated by the IFG module successfully aggregate key local spatial information. The intra-frame prompts are superimposed onto tokens, further facilitating the IFS in propagating local spatial information.

• As demonstrated by the visualization study in Figure 10 in the appendix, after the propagation of discriminative information in a temporal manner, our method successfully focuses on key temporal transitions in the video, which further verifies our effectiveness.

• As shown by the ablation experiments in Table 3, when the IFG module is removed, the accuracy decreases by an average of 4.25%, and when the IFS module is removed, the accuracy decreases by an average of 3.67%. This indicates that the IFG and IFS modules play important roles in the aggregation and propagation of discriminative spatiotemporal information.

(4) As shown in Table 1 and Table 2, our method achieves average improvements of 1.21% and 2.76% compared to existing state-of-the-art fine-tuning methods, realizing effective learning of discriminative spatiotemporal features, which further verifies our effectiveness.

Table 1. The comparison results on VideoMamba-S (Parameters 25.42M)

Table 2. The comparison results on VideoMamba-M (Parameters 74.00M)

Table 3. Ablation of different prompting modules

W2 & Q3 Response: Can the authors clearly explain how the proposed spatial variant works and how it enhances the representation of discriminative action features?

Thanks for your valuable question. We elucidate the working mechanism of spatial variance and its approach to enhancing discriminative action feature representation through the following three points:

(1) As shown in Figure 2 in the main text, the inter-frame prompts generated by IFS correspond one-to-one with the video frames from which the sampled tokens originate, and they are inserted at the boundaries between video frames, enabling the next Mamba layer to simultaneously model the global temporal information brought by inter-frame prompts when processing adjacent frames.

(2) However, different video frames have varying degrees of association with global information. We aim to further learn a feature called "spatial variant" through the spatial features aggregated by IFG, where spatial variant gates the weight of each inter-frame prompt, thereby controlling the degree of influence each video frame has on global information and achieving more refined temporal information propagation.

(3) As shown in the ablation experiment results in Table 4, when the spatial variant mechanism is removed, the classification accuracy drops by an average of 2.33%, indicating that the spatial variant mechanism is beneficial for enhancing the modeling capability of our method.

In summary, spatial variant can achieve more refined temporal information propagation and enhance the representation of discriminative action features by effectively controlling the influence of local information on global information.

Table 4. Ablation of spatial variant

W3 Response: How the parameters are calculated? Figure 5 shows 1.30M parameters, but the numbers in Table 1 and Table 2 are different.

The parameter variation in our method stems from changes in the number of IFS modules, which is demonstrated by the ablation experiment in Figure 7 in the main text.

The 1.30M parameters shown in Figure 5 come from a configuration using only one IFS module, intended to demonstrate that our method surpasses the performance of existing methods while using the minimum number of parameters. This is explained in both the caption of Figure 5 and lines 273-274 of the main text. Our method achieves better performance under the same parameter scale, which further verifies our effectiveness.

Q2 Response: How exactly do the Intra-Frame Gathering and Inter-Frame Spreading modules spread gathered local spatial information in a temporal manner?

Thanks for your valuable question. We have clarified the spreading mechanism of gathered local spatial information in the W1 & Q1 section above.

Firstly, the IFG module aggregates spatial information from each frame. Secondly, the IFS module enables global interaction of the aggregated spatial information. Thirdly, the inter-frame prompt corresponding to each frame thus contains spatial information from other frames, propagating the aggregated information in a temporal manner.

As shown in Table 4, when the IFG or IFS module is removed, the accuracy decreases by an average of 4.25% or 3.67%. This indicates the important roles of IFG and IFS modules in the aggregation and propagation of spatiotemporal information.

W1 & Q1 Response: Evaluation on more tasks beyond video classification

Thank you for your valuable suggestion. We have conducted more experiments on video understanding tasks beyond video classification.

Following the parameter-efficient video learning community, we conducted fair comparisons with existing fine-tuning methods under the same video classification paradigm using four mainstream datasets in the main text. Additionally, we also compared with existing methods on the THUMOS dataset under the temporal action localization paradigm and on the 50salads and GTEA datasets under the temporal action segmentation paradigm.

(1) For the temporal action localization paradigm, as shown in Table 1, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by 1.83. Our proposed SSP demonstrates stronger temporal action localization capability through efficient aggregation and propagation of discriminative spatiotemporal features, which further showcases the effectiveness of SSP in challenging spatial-temporal reasoning tasks.

(2) For the temporal action segmentation paradigm, as shown in Table 2, on the 50salads dataset, SSP achieves an accuracy that exceeds the existing best fine-tuning method by 2.90%, with the F1@50 score obtaining an improvement of 2.27. As shown in Table 3, on the GTEA dataset, SSP achieves an accuracy that surpasses existing state-of-the-art methods by 1.47%, with the F1@50 score also obtaining an improvement of 1.73. This demonstrates that our method achieves efficient extraction of discriminative spatiotemporal information from Mamba hidden states through effective aggregation of local information and efficient propagation of temporal information, making improvements in spatio-temporal reasoning on tasks beyond classification as well, which further verifies the effectiveness of our SSP method.

In summary, all the above experimental results demonstrate our effectiveness and generalization ability on various video tasks.

Table 1. Comparison on the THUMOS datasets under the temporal action localization paradigm

Table 2. Comparison on the 50salads datasets under the temporal action segmentation paradigm

Table 3. Comparison on the GTEA datasets under the temporal action segmentation paradigm

W2 Response: The potential loss of fine-grained spatial or temporal details

Compared to existing fine-tuning methods, our SSP method achieves better preservation of fine-grained spatial and temporal details.

Therefore, we performed further experiments on temporal action localization and temporal action segmentation tasks to validate the effectiveness of our approach in extracting fine-grained video spatiotemporal features.

(1) Regarding the temporal action localization paradigm, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by 1.83, as shown in Table 1, demonstrating better fine-grained spatiotemporal understanding capability during fine-tuning. This is because our IFG and IFS modules both perform low-rank projection during convolution and attention computation, enabling the modules to effectively aggregate and propagate discriminative spatiotemporal information while preserving discriminative fine-grained spatial or temporal details.

(2) Regarding the temporal action segmentation paradigm, our SSP surpasses existing fine-tuning methods with accuracy gains of 2.90% and 1.47% demonstrated in Table 2 and Table 3, respectively. It illustrates that our approach better preserves fine-grained spatial and temporal details through the 2D local convolution of the IFG module and the inter-frame information propagation of the IFS module, which further proves the effectiveness of our fine-tuning method on the Mamba architecture.

In summary, these experimental results above indicate that our SSP method does not suffer from the potential loss of fine-grained spatial or temporal details. Instead, our approach successfully preserves and leverages fine-grained spatial and temporal details, validating the effectiveness of our proposed fine-tuning method.

Q2 Response: How do you mitigate the risk of losing fine-grained spatial or temporal details due to the heavy aggregation performed by IFG and IFS?

In order to effectively aggregate and propagate fine-grained discriminative spatiotemporal information, our IFG and IFS modules both perform low-rank projection during convolution and attention computation.

(1) Regarding the temporal action localization paradigm, the average mAP score of our method exceeds the SOTA fine-tuning methods by 1.83, as shown in Table 1, demonstrating better fine-grained spatiotemporal understanding capability. Regarding the temporal action segmentation paradigm, our SSP surpasses SOTA methods with accuracy gains of 2.90% and 1.47% demonstrated in Table 2 and Table 3, respectively. This is because our low rank IFG and IFS modules effectively aggregate and propagate discriminative information while preserving discriminative fine-grained spatial or temporal details, which further proves the effectiveness of our fine-tuning method.

(2) We conducted ablation experiments to demonstrate the effectiveness of our approach. As shown in Table 4 and Table 5, on the HMDB51 and Breakfast datasets, when the internal dimension of IFG is increased to 576, the accuracy decreases by 0.65% and 4.69% compared to the optimal configuration, respectfully. When the internal dimension of IFS is increased to 512, the accuracy decreases by 2.35% and 4.69%, respectfully. Therefore, we achieve a balance between heavy aggregation and fine-grained modeling by selecting appropriate intrinsic low-rank dimensions for IFG and IFS to ensure effective aggregation of spatiotemporal details.

(3) As shown by the visualization study in Figure 9 in the appendix, even after heavy aggregation, our intra-frame prompts can still preserve fine-grained spatial details compared to other fine-tuning methods. As demonstrated by the visualization study in Figure 10 in the appendix, our intra-frame prompts accurately focus on key temporal transitions in the video, preserving fine-grained temporal information, which further verifies our effectiveness.

Table 4. Ablation on the Internal Dimensions of IFG

Table 5. Ablation on the Internal Dimensions of IFS

W3 & Q3 Response: About the title

Thank you for your suggestion. We have already supplemented experiments on temporal action localization and temporal action segmentation in previous responses and validated the effectiveness of our method in fine-grained spatiotemporal feature extraction.

As shown in Table 1, the average mAP score of our method exceeds existing state-of-the-art fine-tuning methods by 1.83 on the temporal action localization task. As shown in Table 2 and Table 3, our method achieves accuracy gains of 2.90% and 1.47% on the temporal action segmentation task, respectively. These results indicate that our SSP method effectively aggregates and propagates discriminative spatiotemporal information, proving our effectiveness and generalization ability on various video understanding tasks.

W1 Response: The Placement Depth of IFG/IFS

Thank you for your valuable suggestion. We conducted ablation experiments on the placement depth of IFG/IFS on the HMDB51 and Breakfast datasets, with results shown in the Table 1, which implies the following conclusions:

(1) Applying the modules to deeper layers does not yield performance improvements. When our module is applied after the 8th layer, our method experiences slight accuracy decreases of 0.19% and 0.01% on HMDB51 and Breakfast datasets, respectively. When applied to deeper layers, the accuracy gradually declines. When our module is applied after the 24th layer, the accuracy significantly drops by 5.21% on the Breakfast dataset and by 3.00% on the HMDB51 dataset. It is because that after sequential compression by multiple Mamba layers, some discriminative information in the token sequence becomes more difficult for IFG/IFS to extract. While our method achieves optimal performance through appropriate hyperparameter selection, obtaining the highest accuracy of 76.66% and 93.23% on the HMDB51 and Breakfast datasets, respectively

(2) Applying the modules to deeper layers does not change the parameter efficiency, as our IFG/IFS weights are shared across layers. As shown in Table 1, when SSP modules are applied at different depths, the trainable parameters are fixed at 2.41M. Our method achieves effective aggregation and propagation of discriminative spatiotemporal information using only a small number of trainable parameters, surpassing existing state-of-the-art fine-tuning methods by an average of 2.76% as demonstrated in Table 2 in the main text of our paper, with the number of trainable parameters remaining constant regardless of placement depth, which further verifies our robustness.

Table 1. Ablation Experiments on the Placement Depth of IFG/IFS

W2 Response: Comparison with STOP

Thanks for your valuable suggestion. We have reported results under the same matching-based setup with STOP.

In the original STOP paper, the authors performed evaluations in two paradigms: classification and matching.

(1) As for the classification task, our method achieves accuracy improvements of 4.66%, 1.73%, and 32.32% over STOP on the HMDB51, UCF101, and SS-V2 datasets respectively, as shown in Table 1 of the original STOP paper. Our method efficiently aggregates and propagates key information through fine-tuning designs tailored for the Mamba architecture, demonstrating superior performance on classification tasks, which further verifies the effectiveness of our SSP method in parameter-efficient fine-tuning on the Mamba architecture.

(2) As for matching task, we have conducted more experiments on text-video matching tasks using MSVD dataset. As shown in Table 2, our method achieves an average recall rate that exceeds STOP by 9.06 in text-to-video retrieval and by 12.33 in video-to-text retrieval. As a prompt tuning method that similarly focuses on the Visual Encoder, SSP demonstrates stronger text-video matching capability through efficient aggregation and propagation of discriminative spatiotemporal features, which further showcases the effectiveness of SSP in parameter-efficient fine-tuning on the Mamba architecture.

Table 2. Comparison results with STOP under the text-video matching paradigm

W3 Response: About Figure 2

Thank you for your comments. As you said, our proposed modules are applied after the first layer and every subsequent Mamba layer. Therefore, we have supplemented Figure 2 with the insertion of our modules after the 1st layer to further clarify our method, which will be reflected in the camera-ready version. Thank you again for your valuable suggestions.

Q1 Response: Where the additional parameters come from?

It should be clarified that the variation in the number of trainable parameters in our SSP stems from changes in the number of stacked IFS modules, rather than changes in model size.

Firstly, when applied to the smaller VideoMamba-S model, we only need to use 1 IFS module to surpass other fine-tuning methods, with a parameter count of 0.98M. Stacking more modules does not yield performance improvements.

Secondly, when applied to the larger VideoMamba-M model, as shown in the ablation experiment in Figure 7, the best performance is achieved with 3 stacked IFS modules, corresponding to a parameter count of 2.41M. Following the original STOP paper, we did not stack the attention layers or convolutional layers of the STOP method, thus the parameters remained stable at 1.49M.

In summary, regardless of whether under the setting of fewer trainable parameters (0.98M) or more trainable parameters (2.41M), our method achieves average improvements of 1.21% and 2.76% compared to existing state-of-the-art fine-tuning methods in Table 1 and Table 2 in the main text, respectively, through the extraction of discriminative features from the hidden states of the Mamba model. Under the same parameter efficiency, this demonstrates the effectiveness of our approach.

Q1 Response: STOP shows significantly lower performances compared to other methods

Thank you for your valuable suggestion. As discussed in Section 2.3 of this paper, the significant performance gap relies on two reasons:

(1) Vision prompting models with Transformer architecture leverage global attention mechanisms that allow tokens at arbitrary positions in the sequence to interact, whereas Mamba-based models process tokens sequentially, leading to adjacent tokens tending to contain more overlapping information. STOP inputs all tokens into the inter-frame prompting module for computation, introducing excessive redundant information when modeling contextual relationships, a phenomenon that is more pronounced in video data. In contrast, other fine-tuning methods do not simultaneously model all tokens, making them relatively more capable of extracting key discriminative information from tokens that have been highly compressed by Mamba compared to STOP.

(2) As shown in Table 1, when based on Transformer models, this method exhibits similarly low performance on the large-scale dataset SSv2 as when based on Mamba models, indicating that in the efficient fine-tuning domain with limited trainable parameter space, greater computational overhead does not necessarily lead to better performance. Reasonable inductive biases in method design, such as the complementary information aggregation and propagation mechanisms in SSP, can yield superior results.

Compared with STOP, our method utilizes the IFS module to generate inter-frame prompts through token sampling while aggregating local spatial features via IFG, and precisely regulates the influence of local features on global temporal features through entropy weight and spatial variance, thereby learning discriminative spatiotemporal information during fine-tuning. Therefore, as shown in Table 1, our method achieves a 30.28% improvement on the large-scale dataset SSV2, which verifies that our approach more effectively adapts to the Mamba architecture during fine-tuning.

Table 1. Comparison with STOP

Q2 Response: Comparisons on more benchmarks

Thanks for your valuable suggestion. We have conducted additional experiments as follows:

(1) For the Kinetics-400 benchmark, since our VideoMamba is pre-trained on Kinetics-400, where high zero-shot performance of 83.3% has already been achieved, and existing fine-tuning work typically does not test on pre-training datasets [1][2], we did not conduct further fine-tuning experiments on Kinetics-400 to avoid redundancy.

(2) For the LVU benchmark, we conduct experiments on the LVU benchmark, as shown in Table 2 below, SSP achieves an average performance improvement of 3.64% over existing state-of-the-art methods on LVU through efficient aggregation and propagation of discriminative spatiotemporal information, which further confirms the effectiveness of our approach.

Table 2. Comparisons on LVU

[1]: Pan, Junting, et al. "St-adapter: Parameter-efficient image-to-video transfer learning." Advances in Neural Information Processing Systems 35 (2022): 26462-26477.

[2]: Liu, Zichen, et al. "STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.