Smooth Regularization for Efficient Video Recognition

Thank you, Reviewer hQEB, for your careful review and constructive feedback. We appreciate the opportunity to clarify the points you raised. If the paper is accepted, we will make the appropriate changes in the camera-ready version. Below, we provide a brief overview to contextualize our contribution, followed by point-by-point responses to your comments.

The scope of this work is to show that temporal regularization, which steers optimization toward solutions that are (piecewise) smooth in time, is highly beneficial for building efficient video-recognition models. Although the method is general and can be applied across a wide range of architectures, its advantage is especially pronounced at the low-compute end of the spectrum. We will emphasize this in the introduction and, if the reviewer concurs, update the paper’s title to reflect this focus.

The results we present strongly substantiate our claims. Specifically, we advance the state of the art on Kinetics-600 by 2.8–3.4 pp Top-1 for all models in the ≤1 TFLOP range (Table 1). While some reviewers expressed concerns about comparisons with the most recent models, we compared against every published model within this budget to the best of our knowledge. If there are models that we missed, we would greatly appreciate it if you could point us to them.

Efficient video recognition in this compute range is dominated by CNN and hybrid architectures. We therefore applied our GRW smoothing to the MoViNet family, released in 2021, because it remains the strongest lightweight backbone on Kinetics-600. For example, our MoViNet-A3-GRW achieves 84.8% Top-1 at only 56.4 GFLOPs. The first model that surpasses this accuracy, MViTv2-B-32×3, requires 1,030 GFLOPs—more than 18× greater computation.

Finally, we draw the reviewers’ attention to Video-Mamba (2024), the most recent sub-1 TFLOP model. It trails our MoViNet-A3-GRW by 3.8 pp Top-1 while consuming nearly twice the computational budget (see Table 3 and the typo correction in Supplement §1).

Point-by-point responses

1. “The verification is mainly conducted on lightweight models. Is the smooth regularization technique also effective when verified on non-lightweight networks?”

Our regularization technique may apply to non-lightweight video models; however, these typically require more than 2 DGX machines to train, which is the infrastructure that we have, and indeed much more to run several experiments. This is out of scope for many university labs.

2. “The latest base used currently is from 2021. It is suggested to add ablation verification of GRW on the latest base.”

The 2021 MoViNet family holds the current state of the art on Kinetics for models with a compute budget ≤1 TFLOP. We compare against all models in that range (Table 1) and improve the SOTA by 2.8–3.4 pp Top-1. No newer lightweight architecture is truly competitive in this regime. To the best of our knowledge, the most recently published lightweight model which is somewhat competitive to the SOTA is Video-Mamba (2024) (Table 3, ref. [15]), which reaches 76.9% Top-1 at 108 GFLOPs on Kinetics-400. By contrast, vanilla MoViNet-A3 already attains 78.2% Top-1 at 56.9 GFLOPs, and our MoViNet-A3-GRW pushes this to 80.7% Top-1 at 56.4 GFLOPs (see Table 3 and the typo correction in Supplement §1). Thus, GRW provides a substantial gain even over the strongest current baseline.

3. “The current verification results are from fully-supervised scenarios. Can consistent effects be achieved in semi-supervised and few-shot scenarios?”

The scope of this work is supervised learning. Investigating the applicability of GRW smoothing to semi-supervised learning is an exciting direction we are currently exploring in the lab.

4. “Is this smoothing technique applicable to the network structure of transformers?”

The current work demonstrates the effectiveness of GRW smoothing on CNN architectures. CNN and hybrid architectures dominate accuracy in the ≤1 TFLOP regime, and we outperform all of them by 2.8–3.4 percentage points.

To the best of our knowledge, the only transformer models in this budget are MViT-B-16×4 (82.1 Top-1 at 353 GFLOPs) and MViT-B-32×3 (84.3 Top-1 at 850 GFLOPs). In comparison, MoViNet-A3-GRW achieves 84.8 Top-1 at just 56.4 GFLOPs—higher accuracy with roughly 6× and 15× less compute, respectively.

Extending GRW to transformer backbones is underway but was outside the scope of this submission.

We hope these clarifications fully address your concerns. If they do, we would be grateful if you could consider updating your score to reflect the new information. Thank you again for your constructive feedback.

Dear Reviewer hQEB,

Thank you again for your thoughtful and constructive feedback. In our point-by-point response we have:

1. Addressed each of your questions, clarifying our focus on the ≤ 1 TFLOP regime and showing that GRW smoothing yields +2.8 – 3.4 pp Top-1 gains over all published baselines within this budget.

2. Confirmed coverage of every lightweight model, including the recent Video-Mamba (2024); our MoViNet-A3-GRW achieves 84.8 % Top-1, +3.8 pp higher while requiring roughly 50 % of the compute.

We hope these clarifications resolve your concerns, and we would be happy to elaborate further if anything remains unclear. Otherwise, we would greatly appreciate it if you could kindly reconsider your score in light of this information.

We thank Reviewer 8e7P for the careful reading of our submission and the thoughtful feedback. Below we address each point in turn.

Point-by-Point Responses

1. “The method is only evaluated on action classification with Kinetics datasets. It remains unclear how well it generalizes to other video understanding tasks (e.g., action detection, video captioning) or datasets with higher temporal discontinuities.”

Our primary claim is that taking a trained model and applying GRW smoothing over short temporal windows, thereby making the representation piecewise smooth, yields more efficient models. To isolate that effect, we begin with publicly released weights and fine-tune with GRW. Extending to other tasks hinges on the availability of lightweight backbones with released checkpoints. Unfortunately, such weights are scarce for lightweight models. For example, the MoViNet family reports results on Charades, but releases checkpoints only for Kinetics. When weights are unavailable, reproducing a strong baseline requires extensive hyper-parameter tuning (weight decay, label smoothing, dropout, stochastic depth, batch size, LR schedule, layer-wise LRs, data augmentation, AutoAugment, etc.), which exceeds our compute budget (two DGX machines). For these reasons, we focused on Kinetics-400/600, where strong baselines exist, allowing a clean evaluation of GRW smoothing. We will nevertheless attempt to add an additional classification dataset for the camera-ready version.

2. “The paper lacks error bars or variance analysis across multiple training seeds. This could help solidify the claims, especially given the relatively small number of epochs and resource-constrained training setup.”

Variance on Kinetics-400/600 is low. We trained MoViNet-A0-GRW four times with different seeds, obtaining Top-1 accuracies of 77.42, 77.36, 77.30, 77.30 (mean 77.35 ± 0.05). We will report these values with error bars in the camera-ready version.

Reviewer Questions:

”I am interested in seeing the performance on downstream tasks. However, even if the results are not particularly strong, I will not lower my score because of that.”

Please see our response to the first comment above regarding downstream tasks and compute constraints.

Thank you again for your constructive feedback.

We thank Reviewer 8e7P for the careful review and constructive feedback. We appreciate the opportunity to clarify the points you raised. If the paper is accepted, we will incorporate the appropriate changes in the camera-ready version.

Point-by-Point Responses

1. “Computational Cost of Permutations. The loss function in Equation (5) involves a sum over all permutations of frames within a sub-clip, which grows exponentially with the sub-clip length T. While the authors mention sampling k permutations for large T, the computational overhead of this sampling and the contrastive loss itself could still be substantial, especially during training. More details on the practical values of T and k used in experiments and their impact on training time would be beneficial.”

GRW-smoothing assumes piecewise temporal coherence in video data. On Kinetics, we used temporal windows of 1 second. Depending on the FPS of each model, this corresponds to the following values of T: 5, 5, 5, 5, and 12 for MobileNetV3-GRW, A0-GRW, A1-GRW, A2-GRW, and A3-GRW, respectively.

In our implementation, we compute all permutations when T ≤ 7, and uniformly sample k = 1000 permutations when T > 7. Since the number of permutations for T ≤ 7 is (T − 1)! < 1000, this keeps the computational complexity effectively constant.

We include the following ablation study on different values of T:

We emphasize that the training cost is small. GRW-smoothing is applied to the frame embeddings $Z = (z_i)$, where each frame $x_i$ is significantly reduced in size. For example, in A2-GRW, the input frame $x_i$ has dimension 224×224×3, while $z_i$ has dimension 640—over 230× smaller. Thus, even with k = 1000, the cost of GRW-smoothing is negligible compared to the early layers of the model.

Moreover, the wall-clock time of a single training epoch for A2-GRW is only 2% higher than A2 without smoothing, making the difference practically negligible.

GRW-smoothing is not applied during inference, so there is no runtime overhead.

We thank the reviewer for this important comment and, if permitted, will clarify this in the final version.

2. “Frame Ordering" Complexity: The introduction of a "frame ordering" contrastive loss to prevent degenerate solutions adds complexity. While necessary, a simpler or more efficient mechanism to achieve this goal could be explored in future work.”

We agree that exploring simpler alternatives is a promising direction. As noted above, the current implementation incurs minimal cost relative to the model’s overall computation.

We also highlight that GRW-smoothing has desirable mathematical properties: it keeps the embeddings bounded within a ball of a certain radius, as proven in the supplementary material.

3. “The figures and tables of this work can be improved. Some figures are blurry and some tables take up too much space.”

We appreciate this suggestion and will gladly revise the figures and tables to improve clarity and layout in the camera-ready version.

Reviewer Questions

1. “Computational Overhead of GRW Loss. Could the authors provide more specific details on the computational overhead (e.g., FLOPs or actual runtime increase during training) introduced by the GRW loss term, especially considering the permutation sampling?”

Please see our response to Point 1.

2. “Sensitivity to Hyperparameters: How sensitive are the results to the choice of hyperparameters $\lambda$ and $\alpha$? The paper states robustness to perturbing $\lambda$, but a more detailed analysis of the hyperparameter space for both parameters would be valuable.”

We include the following ablation studies on different values of $\lambda$ and $\alpha$:

We hope these clarifications fully address your concerns. If they do, we would be grateful if you could consider updating your score to reflect the new information. Thank you again for your constructive feedback.

Line 3 of the ablation table for parameter T should read:

MoViNet-A0-S-GRW 75.4 3

This row represents the minimal applicable value of T.

Dear Reviewer 8e7P,

Thank you once again for your detailed and constructive feedback. In our point-by-point response we have:

1. Quantified the training overhead you flagged: the extra cost is only ≈ 2% wall-clock time per epoch, with no impact during inference.

2. Specified the practical settings of T and permutation sampling k (all permutations for T ≤ 7; k = 1000 otherwise) and supplied an ablation table for different window lengths T.

3. Added ablation studies for the hyper-parameters α and λ.

4. Committed to clearer figures and tables, which will be incorporated in the camera-ready version.

Please see our rebuttal response for full details.

We hope these clarifications fully address your concerns. If any questions remain, we would be happy to elaborate further at your convenience. Otherwise, we would be grateful if you could kindly reconsider your score in light of this additional information.

Thank you for your careful assessment and constructive feedback. We appreciate the opportunity to clarify the issues you raised. Should the paper be accepted, we will incorporate the revisions below into the camera-ready version. Below, we first contextualize our contribution and then respond point-by-point to your comments. Scope and main claims Our goal is to show that temporal regularization that biases optimization toward piecewise-smooth solutions is particularly effective for efficient video-recognition models. Although the technique is architecture-agnostic, its benefit is most pronounced in the low-compute regime. We will emphasize this in the introduction and, with your approval, update the title accordingly. Our experiments substantiate the claim. On Kinetics-600 we advance the state of the art by 2.8–3.4 pp Top-1 for all models within ≤ 1 TFLOP (Table 1). We compared against every published model in this budget to the best of our knowledge; we would be grateful for pointers to any we might have missed. CNN and hybrid backbones dominate this compute range. We therefore applied GRW smoothing to MoViNet, which remains the strongest lightweight backbone on Kinetics-600. For instance, our MoViNet-A3-GRW reaches 84.8 % Top-1 at 56.4 GFLOPs, while the first model that surpasses this accuracy (MViTv2-B-32 × 3) requires 1,030 GFLOPs, over 18× more computation. We also note Video-Mamba (2024), the latest sub-1 TFLOP model: it lags behind MoViNet-A3-GRW by 3.8 pp Top-1 while using nearly twice the compute (Table 3; see Supplement §1 for the corrected typo).

Point-by-Point Responses

1. “The related works section is quite lacking and not comprehensive. It does not talk about other temporal regularization techniques at all which have been explored in this domain quite extensively. Moreover, the realted works section also doesnt explore the lightweight networks literature which is the setting for the proposed paper.”

We will expand the section to survey prior temporal-regularization methods and the lightweight video-model literature.

2. “From the writing point of view, the mathematical symbols introduced lack the definitions (normally given below as soon as the symbol/variable is introduced). E.g. line 96 introduces X but does not define M or N and later C in the equation 1. There other instances in the rest of the paper as well.”

We agree that explicit definitions improve readability. We will define M, N, C, and any other symbols at first mention.

3. “The introduction lacks the buildup of the motivation and the problem at hand. This can be made better.”

We will strengthen the motivation and problem statement in the introduction.

4. “A big issue with the approach is it's assumption of smooth transitions between the frames in the videos. This may be true for some applications (e.g. surveillance/security footage etc), but not for most video content, which has sudden transitions. For example, I would be very interesed in knowing the results on the SSV2 dataset and other temporally challenging datasets. Even in normal videos encoded with prevalent video codecs, it may be reasonable to assume the smoothness in the "p-frames" but there are also I-frames where the difference is simply too huge to be encoded as a p/b frame. This proposed technique won't work good with fast moving objects, scenes or other videos of this nature.”

Our smoothness prior is imposed within short windows (~1 s), not across an entire video, analogous to assuming continuity only between I-frames. In other words, we assume that the representation can be made piecewise smooth (see § 3.1).

SSV2: Google has not released MoViNet weights or training recipes for SSV2. Reproducing SOTA on SSV2 from scratch requires tuning many hyper-parameters (weight decay, label smoothing, dropout, stochastic depth, batch size, LR schedule, layer-wise LRs, several augmentation parameters, AutoAugment, etc.), which exceeds the capacity of the two DGX machines available to us. We therefore focused on Kinetics, where strong baselines exist, allowing us to isolate the effect of GRW smoothing.

5. “The evaluations are quite small and need to be expanded to other video datasets such as SSV2 and can also be shown to work with localization and detections datasets as well to show the effectiveness of the proposed technique.”

Please see the SSV2 explanation above. Extending GRW to localization/detection is promising future work but beyond the scope of our supervised-classification study.

6. *”The additional complexity introduced by frame ordering component in the contrastive loss is not quantified.”

The overhead is negligible. GRW operates on the frame embeddings $Z = (z_i)$, where each frame $x_i$ is significantly reduced in size. For example, in MoViNet-A2-GRW, each input frame $x_i$​ is 224 × 224 × 3, whereas $z_i$ is 640-D—> 230 × smaller. A full training epoch for A2-GRW is only 2 % slower than baseline A2. We will report this explicitly.

7. *"The comparison methods/models are quite old..."

We compare all models with ≤ 1 TFLOP on Kinetics-600 (Table 1) and ≤ 100 GFLOPs on Kinetics-400 (Table 3 and the typo correction in Supplement §1). Video-Mamba (2024) is included and still trails our best model by 3.8 pp while using nearly 2 × the compute.

8. “Ablation on lambda (eq 7) and alpha (eq 6) is not provided.”

We thank the reviewer for this important comment. We include the following ablation studies on different values of $\lambda$ and $\alpha$:

†We include here our most current results, which improve upon the submitted version and advance the SOTA by 5.4 pp.

We would gladly make the above additions to the paper.

Reviewer Questions:

1. ”How can the proposed technique be adapted to account for the sudden temporal changes?”

Please see response on smoothness assumption in 4. above (window-level, § 3.1).

2. ”Why are the evaluations on SSV2 missing?”

Please see SSV2 explanation in 4. above

3. ”Why is the method not compared with latest state of the art methods?”

Latest sub-1 TFLOP model (Video-Mamba 2024) is included and outperformed; Please see our response to 7. above.

Thank you again for your constructive feedback.