PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement

We appreciate the reviewer’s recognition of the novelty and effectiveness of our frame-wise quantization strategy and the progressive multi-teacher distillation framework. We also appreciate the positive feedback regarding the clarity of our paper and the strong experimental results demonstrated across STVSR, VSR, and VFI tasks.

Q1: Computational Trade-offs across Different Quantization Levels

A1: We quantitatively analyze the trade-offs between bit-width, visual quality, model compression, and inference speed. As shown in Table 1, 4-bit quantization strikes a good balance, achieving ~3× compression, ~1.5× speedup, and retaining over 96% of the full-precision PSNR. In contrast, 2-bit models offer the highest efficiency, but at the cost of a significant quality drop (~2.8dB PSNR loss).

Table 1: Trade-offs Between Bit-width, Efficiency, and Quality (Vid4)

Q2: Reducing Complexity via Frame Grouping for Bound Search

A2: We conduct experiments by modifying the granularity of quantization bound search from per-frame to per-group, where one group contains multiple consecutive frames. Using our two-stage framework, we tested different group sizes (1, 2, 3, 4). In preliminary experiments, we observed that grouping frames and sharing quantization bounds within short temporal windows (e.g., every 2–3 frames) can reduce the computational overhead of BTBI with negligible impact on accuracy. However, for fast motion or complex scenes, where inter-frame differences are more significant, per-frame quantization still offers better robustness. The results (on STVSR task, 2-bit setting) are summarized in Table 2:

Table 2: Impact of Frame Grouping on BTBI Efficiency and Quantization Performance

Q3: Effect of Adding Intermediate-Bit Teachers in PMTD

A3: We extended our PMTD framework by introducing a 6-bit teacher, trained via distillation from FP32 and 8-bit models. We then evaluated 4-bit and 2-bit students with and without the 6-bit teacher. As shown in Table 3, adding a 6-bit teacher generally improves performance, suggesting smoother supervision and enhanced knowledge transfer.

However, the gains are not always substantial, and come at the cost of increased memory and training overhead. This demonstrates that our PMTD framework enables effective trade-offs between accuracy and resource consumption.

Table 3: Impact of Intermediate-Bit Teachers on Student Performance and Training Overhead

Q4: Effectiveness of Percentile-Based Initialization in Suppressing Outliers and Optimal Range Selection

A4: In BMFQ, we initialize the clipping bounds using fixed percentile ranges ([0.1–10] for lower bounds, [90–99.9] for upper bounds) instead of min-max. This helps suppress outliers by explicitly excluding the extreme tails of activation distributions, which are often noisy and unrepresentative. This design is motivated by empirical observation: after BTBI and PMTD optimization, most learned clipping points lie within this percentile range. We further validated this choice through an ablation study (Table 4), where our default setting consistently outperforms both wider and narrower ranges.

In summary, percentile-based initialization provides a robust starting point that avoids distortion from outliers while preserving meaningful dynamic range.

Table 4: Impact of Percentile Initialization Ranges on Quantization Performance (4-bit)

We appreciate the reviewer’s recognition of our novel quantization strategies and the effectiveness of our empirical validation across multiple video enhancement tasks. In response to the reviewer’s questions, we provide the following clarifications:

Q1: What's the point and meaning of doing quantization? Is BMFQ a PTQ or QAT method?

A1: Quantization reduces model size and inference cost by converting high-precision computations into low-precision ones, enabling efficient deployment without significant accuracy loss. Our approach is Post-Training Quantization (PTQ). Specifically, we apply BMFQ and PMTD using unlabeled data, without updating the model weights. The purpose is to learn quantization parameters for both weights and activations. Consistent with other PTQ methods, our approach does not run during inference, ensuring no additional overhead at deployment time.

To clarify:

• QAT updates both weights and quantization parameters during training.
• PTQ keeps weights fixed and only learns clipping bounds.

QAT provides higher accuracy by simulating quantization during training, but it requires retraining and more resources. PTQ is simpler and faster to apply, making it suitable for efficient deployment without modifying the original training pipeline — that’s why we choose PTQ.

Q2: How frame-specific bounds are generalized to unseen test frames?

A2: We'd like to clarify that our approach follows the standard practice of all PTQ methods, where quantization parameters (e.g., clipping bounds) are learned once offline from a calibration set and then frozen for inference, This is a widely adopted paradigm in PTQ literature [1, 2, 3]

Regarding generalization to unseen frames, we note that activation ranges are generally stable across samples, thanks in large part to the use of normalization layers (e.g., LayerNorm) in Transformer-based video enhancement models. While the distribution shape of activations may vary depending on content or motion, the value range—which is the key factor for quantization—remains relatively consistent across different samples.

Since these ranges are primarily determined by the model architecture and its pretrained parameters—rather than by the raw input pixels or the specific temporal position of a frame—they tend to remain consistent across different sequences. This enables the learned clipping bounds to generalize well to unseen data.

Q3: The motivation of BMFQ is not well introduced

A3: We would like to clarify that this has been discussed in detail in Lines 152–168 of the main paper. The motivation for proposing Backtracking-based Multi-Frame Quantization (BMFQ) stems from two key challenges in quantizing multi-frame video enhancement models:

1. Per-frame activation variation:
   As shown in Figure 2-a of our paper, we collect per-frame activation statistics and observe significant disparities in activation distributions across frames. Traditional quantization methods — using shared clipping bounds across frames — fail to capture this variation, leading to suboptimal quantization and degraded reconstruction quality. BMFQ addresses this by introducing frame-specific bounds, allowing for better adaptation to heterogeneous activation statistics.
2. Robust and efficient bound initialization:
   Our backtracking strategy refines the initialization by recursively searching within a constrained percentile range. This approach avoids exhaustive scanning and ensures high-quality quantization with negligible training overhead.

Additionally, as demonstrated in our ablation study, directly applying PMTD from a naive min-max initialization leads to suboptimal results. In contrast, using BMFQ provides a reliable starting point for PMTD with only ~1 minute of initialization time, significantly improving training efficiency and final model quality.

Q4: Computational Analysis of the BTBI Module and other baselines

A4: We evaluate the cost time, number of search iterations, and final performance (PSNR) on multiple STVSR benchmarks under both 2-bit and 4-bit settings. As shown in the Table 1, our BTBI method achieves significantly lower runtime compared to other methods such as DOBI(2dquant) and DBDC, while delivering the best overall performance. This efficiency stems from two key design choices:A tightly bounded search space based on percentiles and An early-pruning strategy that avoids unnecessary evaluations. Although BTBI involves slightly more iterations (due to per-frame bound search), this is a one-time offline process. During inference, the model uses the precomputed quantization bounds just like other PTQ methods, introducing no additional runtime or memory overhead. These results validate the Superiority of our approach: BTBI enables finer-grained per-frame quantization with minimal cost.

Table 1: Performance and Computational Cost Comparison of BTBI on STVSR

Q5: Please clarify the workflow.

A5: The workflow of PMQ-VE follows a simple and modular two-stage pipeline:

1. Input: A pretrained full-precision (FP32) model.
2. Stage 1 — Coarse (BMFQ): Perform per-frame clipping bound search using our Backtracking-based Multi-Frame Quantization (BMFQ), which computes robust activation ranges for each frame.
3. Stage 2 — Fine (PMTD): Apply Progressive Multi-Teacher Distillation (PMTD) to transfer knowledge from full-precision and intermediate-bit (e.g., 8/4-bit) teacher models to the low-bit student model (e.g., 4/2-bit).
4. Output: A quantized model with integer weights and fixed activation bounds.
5. Deployment and Inference: The model performs efficient integer matrix operations using precomputed quantization parameters, without any additional search or adaptation

Q6: How are the intermediate-bit teachers trained?

A6: Teachers are progressively trained:

• 8-bit model: Trained using BMFQ + distillation from FP32 supervision.
• 4-bit model: Trained using BMFQ + distillation from both FP32 and 8-bit supervision.
• 2-bit model: Trained using BMFQ + distillation from FP32, 8-bit, and 4-bit supervision. This hierarchical supervision bridges the capacity gap between FP32 and low-bit models, and is validated through ablation studies (Table 4, Table 6) in our submission.

We noticed that several of your comments reflect a misunderstanding of some fundamental concepts underlying our work. Please feel free to reach out if anything remains unclear.

[1] Chen Z, Qin H, Guo Y, et al. Binarized diffusion model for image super-resolution[J]. Advances in Neural Information Processing Systems, 2024, 37: 30651-30669.

[2] Yuan Z, Xue C, Chen Y, et al. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 191-207.

[3] Tu Z, Hu J, Chen H, et al. Toward accurate post-training quantization for image super resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5856-5865.

Thank you for your explanations and experiments. My questions about the computational complexity and intermediate-bit teachers have been addressed. However, I still have many critical questions about BMFQ.

1. There are two contradictory statements in the rebuttal: “Our approach is Post-Training Quantization (PTQ). … The purpose is to learn quantization parameters for both weights and activations.” and “PTQ keeps weights fixed and only learns clipping bounds.” These two contradictory statements make me confused. Does your approach actually learn the quantized weights?
2. Since this is a PTQ method according to your rebuttal, why does the Problem Formulation sub-section in your main text introduce fake quantization and say “adopt STE during training”? (As far as I know, fake quantization is a QAT method.) This may mislead readers to regard your work as a QAT approach.
3. Based on my understanding of your manuscript, the term “multi-frame quantization” implies that you are searching for the optimal lb and ub for each frame in the calibration video and saving the parameters. If that is the case, there are three key issues that need to be clarified:

3.1 What if the number of frames in the calibration video differs from the number of frames in the test video? For example, if the calibration video has only 1000 frames, you can only obtain 1000 groups of lb and ub parameters. However, if the test video has 2000 frames, how would you apply 1000 groups of parameters to 2000 frames? 3.2 Let’s still take the 1000-frame calibration video as an example. After quantization, you need to store 1000 groups of parameters. This storage overhead may already be far greater than storing a full-precision (FP32) model. The larger the calibration set, the more parameters you need to store. Have you conducted a quantitative analysis of this issue? 3.3 The temporal characteristics of the calibration video and the test video may be completely different. For example, the video used for calibration gradually brightens from extremely dark, while the video used for testing gradually darkens from extremely bright. However, the parameters obtained by overfitting on the calibration video are only applicable to videos with the “gradually brightening from dark” temporal characteristic, and may be completely unsuitable for new test videos.

Please provide explanations for these key issues.

We appreciate the reviewer’s recognition of our work, particularly the novelty of our frame-wise backtracking quantization strategy, the effectiveness and simplicity of the progressive multi-teacher distillation framework, and the strong empirical results across VFI, VSR, and STVSR tasks. We are also grateful for the positive feedback regarding the clarity and generalizability of our approach.

Q1: Computational Complexity and Average recursion Iterations of the BTBI Module and other baseline methods

A1: Following your advice, we compare the average number of iterations and the running time of our BTBI method with other baseline methods. As shown in Table 1, while our per-frame backtracking strategy introduces more iterations due to per-frame bound search, the actual runtime remains significantly lower than existing search-based approaches(e.g., DOBI from 2DQuant, DBDC). This efficiency is mainly attributed to our percentile-constrained search space and early-pruning design, which effectively narrows down the candidate bounds and avoids unnecessary evaluations. Notably, this overhead is only introduced once during the offline quantization phase. During inference, BTBI shares the same runtime cost as other post-training quantization methods, as it simply uses the precomputed quantization parameters without any additional computation, regardless of video length or resolution.

Table 1: Performance and Computational Cost Comparison of BTBI on STVSR

Q2: Task-Specificity of PMQ-VE

A2: We agree with the reviewer that our method is general and can be applied to other tasks beyond video enhancement. In this paper, we focus on three representative video enhancement tasks—VFI, VSR, and STVSR—because they cover the core challenges in video enhancement:

• VFI captures temporal dependencies,
• VSR focuses on spatial reconstruction,
• STVSR requires joint spatio-temporal modeling.

These tasks are not only fundamental but also diverse in their modeling requirements, making them ideal benchmarks to validate the generality and robustness of our quantization framework. Since most other video tasks also rely on spatial and/or temporal cues, our method can be naturally extended to them.

Q3: Whether Spatial Resolution (Patch vs. Frame level) Affects the Quantization Process

A3: Spatial dimensions do not significantly impact the quantization process in our framework. Although we use patch-based augmentation(e.g., cropping) during training, quantization is consistently performed at the frame level. At inference, the learned per-frame clipping bounds are applied to full-resolution frames, and this works well because normalization layers (e.g., LayerNorm) stabilize activation distributions across spatial scales. As a result, whether the input is a full frame or a patch, activations are mapped to a consistent range, ensuring quantization stability.

We do not adopt patch-level quantization because it would require modifying the inference pipeline, adding unnecessary complexity and inconsistency with training.

Q4: Leveraging Temporal Redundancy to Reduce Recursive Search Overhead in BTBI

A4: Thank you for the insightful suggestion. We agree that consecutive video frames share considerable temporal redundancy, and this can be leveraged to reduce the computational cost of the per-frame recursive bound search in BTBI. To address this, we implemented a temporal warm-start strategy, where the BTBI search for frame i is initialized using the optimal bounds obtained from the previous frame i−1. Below is a simplified illustration of the idea:

# Temporal warm-start BTBI
for i in range(T):
    if i == 0:
        lb_i, ub_i = BTBI_search(X[0])  # standard search
    else:
        lb_i, ub_i = BTBI_search(X[i], init=(lb_{i−1}, ub_{i−1}))  # warm start

We compare the original BTBI and the warm-start variant in the table2. As shown, the warm-start version substantially reduces both the average iteration count and runtime cost, while maintaining nearly identical performance in terms of PSNR. This warm-start strategy brings consistent improvements in both runtime and iteration number, with negligible impact on final restoration quality. It demonstrates that leveraging temporal continuity is an effective way to reduce BTBI overhead in real video applications.

We sincerely thank the reviewer for this excellent suggestion, which directly helped us improve the efficiency of the BTBI module. We will include this strategy, its implementation, and the empirical results in the final version of the paper.

Table 2: Effect of Warm-Start Strategy on BTBI (2-bit and 4-bit Quantization)

We greatly appreciate the reviewer’s thoughtful and constructive feedback. All raised concerns—including BTBI’s computational overhead, task generality, spatial resolution effects, and the suggestion to leverage temporal redundancy—will be further clarified in the final version to strengthen the technical clarity and quanlity of our work.

We sincerely appreciate the reviewers' recognition of the novelty, practicality, and effectiveness of our proposed PMQ-VE framework for quantization in multi-frame video enhancement, as well as its strong performance across various benchmarks and the clarity of our writing.

Q1: Lack of quantitative analysis and comparison of computational cost for BTBI and baseline methods

A1: Following your advice, we evaluate the cost time, number of search iterations, and final performance (PSNR) on multiple STVSR benchmarks under both 2-bit and 4-bit settings. As shown in Table 1, our BTBI method achieves significantly lower running time compared to other search-based methods (e.g., DOBI from 2DQuant, DBDC), while achieveing the best overall performance. This efficiency stems from two key design: (1) a tightly bounded search space based on percentiles, and (2) an early-pruning mechanism that eliminates redundant evaluations. Although BTBI performs more iterations due to per-frame bound search, this is a one-time offline process. During inference, the model uses precomputed bounds like all other PTQ methods, incurring no additional running time or memory overhead.

Table 1: Performance and Computational Cost Comparison of BTBI on STVSR (5-run average ± std)

Q2: Lack of statistical reporting across multiple runs; concerns about stability and reproducibility

A2: As shown in Tables 2, 3, and 4, we conduct five independent runs of our method on all three tasks (STVSR, VFI, and VSR), each with a different random seed and calibration set. The resulting PSNR values and their standard deviations (typically < ±0.1 dB) demonstrate the stability, robustness, and reproducibility of our method under low-bit quantization.

Table 2: STVSR (5-run average ± std)

Table 3: VFI (5-run average ± std)

Table 4: VSR (5-run average ± std)

Q3: Sensitivity of newly introduced hyperparameters and their impact on robustness

A3: We conducted a sensitivity analysis on the key hyperparameters of PMQ-VE, including:

• The distillation loss weight λ
• The number of BTBI search steps N (which controls ΔL/ΔU)
• The early-stopping threshold ε
• The percentile search ranges for lb and ub We varied each parameter individually while keeping the others fixed, and evaluated the resulting performance under 4-bit quantization. As shown in Table 5, across all variations, the PSNR remains within ±0.1 dB of the default setting, demonstrating that our method is robust to reasonable changes in hyperparameter configurations. These minimal fluctuations confirm that our coarse-to-fine framework is robust to reasonable changes in hyperparameter configurations.

Table 5: Hyperparameter Sensitivity Analysis Under 4-bit Quantization (5-run average ± std)

We greatly appreciate the reviewer’s valuable suggestions. All raised concerns—including computational analysis, statistical reporting, and hyperparameter sensitivity—will be addressed in the final version to enhance the completeness and clarity of our submission.