Storyboard-guided Alignment for Fine-grained Video Action Recognition

Thank you for your comments and for recognizing that the problem formulation of this work is interesting. Below, we address your concerns point by point.

Q1: Novelty.

A1. Please refer to L8–14 of the abstract and L31-64 of the introduction section. 'We propose a multi-granularity framework, SFAR. SFAR generates fine-grained descriptions of common atomic actions for each global semantic using a large language model. Unlike existing works that refine global semantics with auxiliary video frames, SFAR introduces a filtering metric to ensure correspondence between the descriptions and the global semantics, eliminating the need for direct video involvement and thereby enabling more nuanced recognition of subtle actions.'

Q2: Why Is It Named “global”?

A2. As detailed in Eq. (8) of the main paper, the augmented global text embedding $\hat{\mathbf{T}}\_{\text{c}}$ is constructed via a residual connection based on the original global text $\mathbf{T}\_{\text{c}}$. Compared to the fine-grained video embedding module, the coarse video embedding emphasizes holistic video semantics. Therefore, we use the term “global” to reflect the module’s target.

Q3: Formulation of Coarse-Grained Importance and Fine-Grained Importance.

A3. For coarse-grained importance, we compute scores $a\_{\text{l}}^\text{coarse}$ (Eq. (9)) using the augmented global text embedding $\hat{\mathbf{T}}\_{\text{c}}$ (Eqs. (6)–(8)) and video frame embedding $\mathbf{v}\_{\text{l}}^\text{cls}$, as described in L184–189.

For fine-grained importance score $a\_{\text{l}}^\text{fine}$, as defined in L211–212 and Eq. (11), we measure the maximum average similarity between $\mathbf{v}\_{\text{l}}^\text{cls}$ and sub-texts $\mathbf{S}\_{\text{c}, \text{n}}$, followed by SoftMax normalization.

Q4: Organisation Issue.

A4. Our intention with the detailed figure captions was to ensure that readers could understand the core methodology and concepts even when viewing the figures independently. We will improve conciseness as suggested.

Thank you for your valuable comments and for recognizing that our ablation experiment design is diverse. Below, we address your concerns point by point.

W1: Content Optimization.

A1. (1) We intentionally designed captions to be comprehensive and self-contained, ensuring symbolic formulas and context are immediately accessible without requiring readers to cross-reference text sections; (2) Thanks for the catch and fixed; (3) We will try to leave more room for figures.

W2: Motivations of Text Prompt Perplexity Score.

A2. The Text Prompt Perplexity (TPP) score (Eq. (3) in the main paper) is designed with two key factors: similarity with global text and diversity among sub-texts.

$

&\text{TPP}\_\text{c} = \exp \bigg( - \frac{1}{N} \sum\_{\text{n}=1}^{N} \log \big( \alpha (\sigma\_{\text{c},\text{n}} ) \beta(\delta\_{\text{c},\text{n}}) \big) \bigg) \ .
\tag{3}

$

First, we employ $\sigma\_{\text{c},\text{n}}$ to quantify the similarity between a global text and its corresponding sub-texts, a higher value of $\sigma\_{\text{c},\text{n}}$ indicates greater similarity. Next, $\delta\_{\text{c},\text{n}}$ is used to evaluate the similarity among sub-texts themselves, where a higher value of $\delta\_{\text{c},\text{n}}$ indicates greater diversity across the sub-texts. The linear scaling functions $\alpha(\cdot)$ and $\beta(\cdot)$ ensure that generated sub-texts with higher similarity and diversity scores receive higher TPP scores.

W3: Video Alignment.

A3. Although the sub-texts generated by SFAR adopt a verb-noun structure, they are not bound to static interpretations of a single frame. As described in Eq. (11), SFAR computes similarities between each sub-text and individual frames within the CLIP embedding space, enabling the identification of frame-level visual evidence corresponding to atomic actions. These interactions are then aggregated through temporal pooling (Eq. (12)), which captures inter-frame dependencies and facilitates the flow of contextual information over time. Additionally, SFAR incorporates a 6-layer Transformer module within its visual encoder to explicitly model the temporal progression of dynamic content. This further enriches the representation of motion and continuity across the video sequence.

Through these mechanisms, SFAR effectively ensures the adaptability of the generated text to the dynamic video information.

Thank for the authors' efforts in rebuttal. This has resolved my concerns.

1. I personally do not agree with the author's view on the writing mode, as it has had a negative impact on my reading. This is in line with the opinion of Reviewer ys6V. I hope that this will be reflected in the revised version.
2. The author resolved my confusion about the generation principle of Sub-text, which is highly commendable.
3. The authors' theoretical illustration of video-text cross-modal alignment does not convince me that, even for a single video frame, CLIP still has difficulty focusing on verb-image correlations in action primitives because its feature space is noun-dominated.

In general, this paper is valuable and promotes the research in this field. Considering the positive opinions of other reviewers, I choose to raise my rating to Borderline accept.

Thank you for your insightful comments and for acknowledging that our paper is generally well-written and easy to follow. Below, we address your concerns point by point.

W1: LLM-Generated Sub-texts Without Visual Context.

A1. We did explore the use of visual context previously and found two major limitations for this solution.

1. Generating sub-texts for each individual video sample entails significant computational and time costs. Alternatively, generating sub-texts at the class level introduces the challenge of selecting a representative video, which can affect consistency and coverage across diverse instances.

2. For ambiguous, non-uniform, or contextually intricate actions, incorporating visual context yields sub-texts dominated by noisy or spurious information. This undermines frame-level alignment and degrades model performance.

To illustrate, take 'Situp' as an example, we randomly selected three sample videos and uniformly sampled eight frames from each. These were provided as input to GPT-4o to generate visually informed sub-texts:

These sub-texts introduce scene-specific elements (e.g., attire, environment, presence of others) that do not directly contribute to the representation of atomic actions and can confound alignment.

In contrast, our proposed sub-text generation method is guided by the Text Prompt Perplexity (TPP) score, which promotes relevance by emphasizing informative atomic actions. To further minimize ambiguity and redundancy, we employ an attention-based alignment mechanism (Eqs. (6–8) and (11)) that assigns lower weights to less informative content, yielding sub-texts that are both precise and semantically aligned with frame-level dynamics.

W2: Dual Embeddings.

A2. We demonstrate the complementary nature of the coarse and fine-grained embeddings through corresponding heatmaps. As visualized in Fig. 2, the attention maps for coarse, fine, and their combination show distinct focus regions across three examples: 'Making a sandwich', 'Pull up', and 'Shearing sheep'. For instance, in 'Shearing sheep', the coarse branch (Fig. 2c) primarily attends to the head of the sheep and the area where shearing occurs on the right, while the fine branch (Fig. 2d) highlights the body of the sheep, the shearing tool, and the wool falling to the ground. The combined attention (Fig. 2e) effectively integrates both perspectives.

These visualizations confirm that coarse and fine embeddings capture different aspects of the video, thus mitigating redundancy and supporting the effectiveness of the dual-embedding design. Quantitative results are shown in Tab. 2e.

W3: Ablations of Alternative Prompt Templates.

A3. Please kindly refer to Fig. 7b of the main paper, where we present the relationship between TPP scores and the corresponding action recognition accuracy of sub-texts generated using alternative prompt templates. Representative examples of these templates are provided in Tab. 5 of the supplementary material. These results clearly demonstrate that prompts yielding higher TPP scores also lead to superior recognition performance, validating the effectiveness of our TPP-based selection strategy. We will highlight these comparisons more explicitly in the revised version for added clarity.

We appreciate your insightful comments and your recognition of our novel multi-granularity framework. We address your concerns point by point below.

Q1: Fixed Number of Sub-Actions (N).

A1. We previously observed that the accuracy of different action categories changes with respect to the number of sub-actions (N). However, we set a fixed N, rather than using a class-aware N. The reason is given below.

Taking N = 2 and N = 5 as examples, the accuracy differences across representative actions are given below.

The results indicate that complex, multi-stage action categories (e.g., 'Dribbling basketball' and 'Clay pottery making') benefit from large N, whereas repetitive or simple motions such as 'Clapping' and 'Slicing onion' get better accuracies with small N.

We had investigated overall performance gains by using class-aware N, and only found marginal improvement over fixed N=5, 0.5% and 0.2% for Top-1 and Top-5, respectively.

Given the computational cost of involving a class-aware value of N for each action, we choose to use fixed N for efficiency.

Q2: Sequential Diversity in Sub-Action Decomposition.

A2. While our method adopts a fixed sub-action sequence for each coarse action, it does not rely on this ordering for accurate recognition. In fact, our model is designed to remain robust to different valid temporal arrangements of sub-actions through the use of temporally agnostic attention mechanisms.

Specifically, the fine-grained importance score $a\_{\text{l}}^\text{fine}$ is computed independently for each frame $\mathbf{v}\_{\text{l}}^\text{cls}$ by measuring its maximum average similarity with all sub-texts $\mathbf{S}\_{\text{c}, \text{n}}$. This design allows the model to capture fine-grained relevance without relying on a fixed temporal sequence of atomic actions. Similarly, the coarse-grained importance score $a\_{\text{l}}^\text{coarse}$ leverages the augmented global text embedding $\hat{\mathbf{T}}\_{\text{c}}$, which is constructed to be invariant to sub-action order. Together, these components ensure that both local and global relevance signals are captured without enforcing a fixed temporal sequence.

To validate this, we randomly shuffled the order of the sub-texts and observed that the top-5 accuracy remained unchanged, while the top-1 accuracy decreased marginally by only 0.01.

Q3: Absence of Temporal Modeling.

A3. To address the absence of explicit temporal modeling, we construct a baseline (Temporal) using a 6-layer Transformer encoder to capture sequence dependencies among video frames (Line 300, main paper). To effectively model critical temporal dynamics, especially for actions reliant on sub-action ordering, we compute fine-grained and coarse-grained importance scores $a\_{\text{l}}^\text{fine}$ and $a\_{\text{l}}^\text{coarse}$, respectively, which are adaptively aligned with sub-action semantics. These scores modulate the corresponding frame embeddings $\mathbf{v}\_{\text{l}}^\text{cls}$ without disrupting the video’s inherent temporal structure. The resulting weighted embeddings are then aggregated and fused via two feedforward layers (Eq. 13), enabling the model to integrate multi-granularity information and produce a temporally adaptive video-level representation.

Thanks for the author's respones, it resovled my almost concerns, I will keep my positive rating. For the fixed number N of sub-actions ablations, it is better to be considered in the final version, as it is an important discussion for the reproducbility and motivation for the design.