On Quantizing Neural Representation for Variable-Rate Video Coding

Q3: It would be better to provide more intuitive explanations of the proposed approach, e.g. diagram figures and pseudo algorithm.

A3: Thank you for this thoughtful suggestion. We have included a pseudo-algorithm detailing the NeuroQuant pipeline in the updated Appendix D. Additionally, we plan to open-source our code upon publication of the manuscript to further aid reproducibility and understanding.

Q4: I wonder why the improvement is such impressive, compared to the existing quantization approaches. Is the channel-wise quantization conducted on both weight and activation?

A4: Thank you for raising this important question. Below, we try to analyse their quantization approaches for inferior performance in INR-VCs.

Weight Activation: The QAT technique in FFNeRV applies a weight activation function:

$

\text{Forward:}  \ \ \ \hat{w} = sign(w) \cdot \frac{\lfloor (2^b-1) \cdot \tanh(|w|) \rfloor}{2^b-1}.

$

Here, weights are constrained with $(-1,1)$ using $tanh(|w|)$. While this simplifies quantization, it also limits the network’s representational capacity, leading to degraded performance. In contrast, other methods avoid weight activation, preserving the better representational capacity of the network.

Straight-Through Estimator (STE): Current QAT techniques rely on STE for backward due to in-differentiable round function, e.g., in FFNeRV:

$

\text{Backward:} \ \ \ \frac{\partial \mathcal{L}}{\partial \hat{w}} \approx \frac{\partial \mathcal{L}}{\partial w}.

$

However, STE leads to biasd gradients, hindering the accurate optimization. HiNeRV attempts to mitigate this issue by introducing a random binary mask:

$

    \text{Forward:}  \ \ \ & \tilde{w} = w \cdot mask + \hat{w} \cdot (1-mask), \\ \ \ \ \ 
    \text{Backward:} \ \ \ & \frac{\partial \mathcal{L}}{\partial \tilde{w}} \approx \frac{\partial \mathcal{L}}{\partial w} \cdot mask, 

$

While this approach reduces the impact of quantized weights during training, it does not fully address STE’s inherent limitations. AdaRound, BRECQ and NeuroQuant employ differentiable round function combined with an annealing strategy, which allows the rounding operation to converge to the true rounding function over iterations. This avoids biased gradients and improves optimization accuracy. Similar STE-induced performance degradation has been observed in AdaRound and HiNeRV, with HiNeRV reporting a $0.02 - 0.48$ PSNR drop due to STE.

Layer/Block Calibration: As shown in Fig. 3(c), INR-VCs exhibit significant cross-layer dependencies across the entire network. NeuroQuant employs network-wise calibration, capturing these dependencies more effectively than the layer-wise calibration in AdaRound or the block-wise calibration in BRECQ, QDrop, and RDOPTQ.

Mixed Precision: NeuroQuant uses mixed precision for variable-rate coding, achieving superior bit allocation compared to uniform precision methods. Unlike HAWQ-v2 [1], which relies solely on diagonal Hessian information, NeuroQuant incorporates weight perturbation directionality and off-diagonal Hessian information for better sensitivity analysis (see left subfigure of Fig. 5).

Fair Comparison: For fairness, all methods were implemented with channel-wise quantization. Original layer-wise quantization in methods like AdaRound and BRECQ would degrade performance significantly (e.g., $>1$ dB). From a transmission perspective, bitrate depends only on the weights, not activations. Therefore, activation quantization was not applied in NeuroQuant or any of the compared methods.

Q5: It seems like the key point of this approach is to adopt a mix-precision one-block BRECQ on INR video coding models. Despite the story about the unique properties of non-generalized INR models, can we further develop the method to a more general MP PTQ with calibration?"

A5: Thank you for this thoughtful question. Before addressing potential extensions, we have to emphasize, as discussed in Q2, that treating NeuroQuant as one-block BRECQ is not accurate just as we cannot treat BRECQ/QDrop as one-layer AdaRound. Below, we outline some potential pathways for extension:

Enhanced Sensitivity Analysis: NeuroQuant’s mixed-precision criteria ($\Delta w^T H^{(w)} \Delta w$, Sec. 3.1) incorporates global Hessian information ($H^{(w)}$) and accounts for weight perturbation anisotropy $\Delta w$. While current generalized networks exhibit cross-layer dependencies primarily within blocks (e.g., Residual Blocks), considering global dependencies can provide more accurate second-order loss estimations. Besides, NeuroQuant simplifies Hessian calculating by introducing the Hessian-Vector product. This makes our MP adaptable to various networks. For deeper networks with exponentially growing bitwidth configurations, fast search strategies like integer programming [2] or genetic algorithms [3] could be employed.

Broadening Calibration Granularity: NeuroQuant employs network-wide calibration for INR-VCs, while generalized networks prefer block-wise calibration. To evaluate this, we applied network-wise calibration to generalized codecs using the open-source RDOPTQ. Results, measured in BD-Rate, are shown below:

These results suggest that plain network calibration is not optimal for generalized networks. A potential strategy is to introduce a modular granularity approach, allowing transition between layer-, block-, and network-wise calibration, depending on the specific architecture and use case.

We appreciate your novel perspective and will further explore these possibilities in future work.

Q6: Is this inter-layer independence a good property for evaluated INR models, or an ill pose? I would appreciate if the authors provide some insight about this property.

A6: Thank you for this thought-provoking question. The inter-layer dependence in INRs is a double-edged sword, with both benefits and challenges:

Benefits: INRs are inherently non-generalized and tailored to represent specific video. This often leads to strong inter-layer dependencies, where the contribution of each layer to the overall representation is tightly coupled. Such strong coupling can reduce redundancy across the network, leading to better representation performance. In contrast, excessive independence could indicate poor utilization of the network's capacity.

Challenges: Strong dependencies complicate quantization, as perturbations in one layer can propagate across the network. This is where proposed network-wise calibration becomes critical. Additionally, dependencies can limit scalability and robustness, as architectural modifications can disrupt the internal balance of the network.

Experiment: For INRs, the dependence degree increases with the training iterations growing. To measure dependence, we quantized the fourth layer using vanilla MinMax quantization without any calibration in three video sequence:

More results can be found in Appendix E. The results indicate that stronger dependence means larger quantization error but provides better representation performance. For future work, exploring a middle ground—where moderate independence is encouraged to balance representation fidelity and quantization robustness—might be an interesting direction.

We appreciate your novel perspective, which has sparked meaningful discussions. We believe these discussions not only enhance our manuscript but also benefit practitioners in understanding the unique characteristics of non-generalized INRs.

Thank you again for your time, effort, and valuable feedback in reviewing our manuscript.

Ref:

1. Dong, Zhen, et al. "Hawq-v2: Hessian aware trace-weighted quantization of neural networks."

2. Hubara, Itay, et al. "Improving post training neural quantization: Layer-wise calibration and integer programming."

3. Guo, Zichao, et al. "Single path one-shot neural architecture search with uniform sampling."

Thank you for your insights and positive feedback on this paper. Below is the detailed response to each question and we hope we have addressed your concerns.

Q1: The authors should clarify their evaluation details, especially the calculation of bpp.

A1: We apologize for the lack of clarity in our original description and provide the following clarifications:

Storage of $s$: All quantization methods have to store the quantization step parameter $s$ because $s$ maps floating point weight $w$ to the quantized integer weight $\hat{w}$:

$

    \hat{w} = round (\frac{w}{s}), \ \ \ s^{l,k} = \frac{\max (w^{l,k}) - \min (w^{l,k})}{2^{b^l} - 1},

$

with $w^{l,k}$ is the weights in the $k$-th channel of the $l$-th layer. During decoding, the original weight $w$ cannot be reconstructed from $\hat{w}$ without corresponding $s$. Even for fixed-precision methods, where the bitwidth $b^l$ is consistent across all layers, the weight distributions vary significantly among layers and channels (as shown in Fig. 3(a, b)). Therefore, $s$ must be stored for all methods, regardless of whether they use fixed or mixed precision.

Size of $s$: In our implementation, we use channel-wise granularity for all compared methods, meaning all weights in a channel share the same $s$. Consequently, the number of stored $s$ depends solely on the number of channels. For example, in HNeRV (3M), the total number of weights and $s$ is 3M and 2.6K (less than 0.1% of total weights), respectively.

Bpp Calculation: The bpp calculation includes both the quantized network parameters $\hat{w}$ and the quantization parameters $s$:

$

    bpp = bpp_w + bpp_s = \frac{\sum E(\hat{w})}{H\times W \times T} + \frac{\sum s \cdot b_s}{H\times W \times T},

$

where $E$ denotes lossless entropy coding. For example, on a 1080p video sequence with INT4 $\hat{w}$ and FP16 $s$, the bpp for HNeRV-3M is approximately: $bpp_w \approx 0.01, bpp_s \approx 0.00004$. As shown, the contribution of $s$ to the overall bpp is negligible, but it is still included in all calculations for fairness.

We have detailed evaluation procedures in the updated manuscript.

Q2: I request the authors further clarify their contribution against the existing approaches.

A2: We appreciate the opportunity to clarify the distinctions between NeuroQuant and existing methods like BRECQ. NeuroQuant introduces several novel contributions tailored specifically to the unique challenges of non-generalized INR-VCs:

Block vs. Network Independence: We need to clarify that the "block" is concrete structure of a network. In INR-VCs, the "block" denotes a up-sampling block containing multiple layers. An INR-VC networks contains multiple up-sampling blocks.

AdaRound first employs layer-wise calibration, while BRECQ observes that cross-layer dependencies primarily exist within blocks (e.g., residual bottleneck blocks) in generalized networks and performs block-wise calibration. This approach, however, does not hold for non-generalized INR-VCs. Through theoretical and experimental analysis, we demonstrate dependencies span the entire network (as illustrated in Fig. 3(c)), thereby leading to network-wise calibration. Such theoretical and experimental exploration for changing granularity from block to network is our key contribution. Neglecting network space structure and equating a network to a block will oversimplify the problems.

Difference behind MSE-form Objective: NeuroQuant derives its MSE-form objective (Eq. 15) from the final MSE loss function used in video coding. In contrast, BRECQ’s MSE-form objective is derived from block-level Fisher approximations of the loss function (Eq. 10 in BRECQ) for block-wise calibration. The motivations and derivations differ significantly.

Channel-wise Quantization: While AdaRound and BRECQ primarily employ layer-wise quantization, NeuroQuant introduces channel-wise quantization to better capture the unique weight distributions in INR-VCs. For fairness, we implement channel-wise quantization for all compared methods in our experiments. Without this adjustment, NeuroQuant would exhibit even larger performance gains.

Mixed Precision: Another significant contribution of NeuroQuant is defining variable-rate coding as a mixed-precision bit allocation problem. We incorporate weight perturbation directionality and off-diagonal Hessian information to improve sensitivity assessment. Furthermore, we introduce the Hessian-Vector product to simplify computations, avoiding explicit Hessian calculations.

Better Performance: NeuroQuant achieves substantial performance gains compared to BRECQ, validating the proposed mixed-precision and network-wise calibration approaches.

These distinctions collectively form the foundation of NeuroQuant’s contributions. While we acknowledge BRECQ’s pioneering work, NeuroQuant extends the scope and applicability to address the unique challenges of INR-VCs.

Thanks for the authors' detailed response. They address my concerns, post supplementary results, and provide more insights to demonstrate their claims in this paper. After thoroughly reading all the reviews and author responses, I believe this paper is solid to push the study of INR-based video coding. I raise my rating to acceptance.

We sincerely thank the reviewer for their constructive comments and are delighted that they appreciate the methodology and state-of-the-art performance of NeuroQuant. Below, we address the raised concerns in detail:

Q1 & Q3: Could you explain why the proposed PTQ method has better RD performance compared to FFNeRV/HiNeRV?"

A1 & A3: Thank you for raising this important point. There are some potential factors for inferior performance of QATs.

Weight Activation: The QAT method in FFNeRV applies a weight activation function:

$

\text{Forward:} \ \ \ \hat{w} = sign(w) \cdot \frac{\lfloor (2^b-1) \cdot \tanh(|w|) \rfloor}{2^b-1},

$

where weights are constrained within $(-1, 1)$ using $tanh(|w|)$. While this simplifies the quantization process, it also limits the network's capacity, potentially degrading performance. In contrast, other methods avoid weight activation, preserving the better representational capacity of the network.

Straight-Through Estimator (STE): Current QAT techniques rely on STE to approximate gradients for the non-differentiable round function:

$

\text{Backward:} \ \ \ \frac{\partial \mathcal{L}}{\partial \hat{w}} \approx \frac{\partial \mathcal{L}}{\partial w}.

$

However, STE introduces biased gradients, hindering accurate optimization. HiNeRV observed similar performance degradation and replaced STE with:

$

    \text{Forward:}  \ \ \ & \tilde{w} = w \cdot mask + \hat{w} \cdot (1-mask), \\ \ \ \  \ \ 
    \text{Backward:} \ \ \ & \frac{\partial \mathcal{L}}{\partial \tilde{w}} \approx \frac{\partial \mathcal{L}}{\partial w} \cdot mask, 

$

where a random binary mask limits biased gradient propagation. While this mitigates STE’s issues, it does not address the fundamental problem.

NeuroQuant, in contrast, uses a differentiable round function and an annealing strategy, allowing the round function to converge to the true round operation over iterations. Such differentiable round operations have been widely used in PTQs, which avoid biased gradients and improves optimization accuracy. This improves gradient accuracy and reduces performance degradation, as also observed in AdaRound and HiNeRV (e.g., STE caused a $0.02 - 0.48$ dB PSNR drop in HiNeRV).

Mixed Precision and Network-wise Calibration: NeuroQuant leverages mixed precision to allocate bitwidths optimally across layers, improving efficiency and quality. The 2-bit results in Table 2 demonstrate NeuroQuant’s superior performance. The left subfigure of Fig. 5 directly compare our method with uniform quantization, and HAWQ-v2 [1], a leading mixed-precision method. Additionally, NeuroQuant’s network-wise calibration further enhances RD performance by accounting for inter-layer dependencies.

Q2 & Q4: Could you provide a detailed explanation of the Encoding Complexity section? Does it refer to the encoding complexity of a single bitrate or multiple bitrate?

A2 & A4: We apologize for any confusion caused by the original description.

For a given target bitrate, NeuroQuant performs a lightweight QPs calibration process based on pretrained weights, which requires only a few iterations. In contrast, naive NeRV, HNeRV, and HiNeRV adjust bitrates by modifying the number of weights, requiring separate models to be trained from scratch for each target bitrate, which is significantly more time-consuming.

Encoding Time of Table 2: The encoding complexity reported in Table 2 refers to the time required to support a new bitrate. Since NeuroQuant shares a pretrained model across bitrates, its pretraining time was originally excluded. However, we acknowledge that this could be misleading and have updated Table 2 to include pretraining time for clarity.

We hope these information is helpful and enhances your confidence in assessing our manuscript. Thank you again for your constructive comments and support.

Ref:

1. Dong, Zhen, et al. "Hawq-v2: Hessian aware trace-weighted quantization of neural networks." Advances in neural information processing systems 33 (2020): 18518-18529.

Q7: How much time is required to obtain additional rate points with models like NeRV, HNeRV, FFNeRV, and HiNeRV? Although these models require a pretraining phase, the pretrained model can be fine-tuned to produce multiple rate points by adjusting quantization levels or using entropy coding with regularization [1]. Fine-tuning time is substantially shorter than full training (e.g., 30 epochs for QAT versus 300 + 60 epochs for HiNeRV).

A7: The time required to obtain a new bitrate from pretrained baselines (e.g., NeRV, HNeRV, and HiNeRV) is reported in the table provided in Q1, while reconstructed performance comparisons are shown in Q3. For comparison, we conducted 300 epochs of pretraining followed by 30 epochs finetuning for [1] as you suggested. Although both NeuroQuant and entropy-based methods [1,2,3] aim to achieve variable rates, NeuroQuant clearly outperforms these methods in several key aspects:

Better Reconstructed Quality: For similar training iterations, NeuroQuant achieves better reconstructed quality, with an improvement of more than 0.32 dB over [1].

Lower Encoding Complexity: For one training iteration, [1] involves additional overhead due to entropy estimation, resulting in approximately $\times 1.4$ encoding time compared to NeuroQuant.

Accurate Bitrate Control: NeuroQuant achieves precise bitrate control by adjusting quantization parameters, as bitrate is directly proportional to the number of parameters and their bitwidth. In contrast, entropy-based methods [1,2,3] can not directly estimate compressed bitrate from the Lagrangian multiplier $\lambda$. Instead, it requires multiple encoding runs to fit a mapping from $\lambda$ to bitrate. This mapping is sequence-dependent, reducing its universality and reliability.

Additionally, NeuroQuant is fundamentally a quantization method and is compatible with entropy regularization. As mentioned in Sec. 3.3, future work will explore the integration of NeuroQuant with entropy-based methods to further Rate-Distortion trade-off.

Q8: What is the computational cost (in terms of MACs and wall time) for the proposed calibration process compared to QAT?

A8: Calibration and QAT are training processes for which exact MACs are difficult to measure. However, we highlight the runtime compared with current methods:

Compared with QAT in FFNeRV: Due to the weight activation function $tanh(|w|)$, FFNeRV requires training from scratch. Without this, post-training weights suffer significant degradation when passed through $tanh$.

Compared with QAT in HiNeRV: HiNeRV's QAT process has a runtime similar to NeuroQuant’s calibration process. However, naive HiNeRV requires training separate models from scratch for each potential bitrate. In this context, NeuroQuant provides a practical solution by leveraging a pretrained model, enabling adaptation to variable bitrates while ensuring efficient encoding.

Comparison with entropy-based Gomes et al. [1]: As shown in Q3 and Q7, while [1] supports variable rates, it falls short in reconstructed quality, encoding complexity, and bitrate control accuracy compared to NeuroQuant.

Overall, NeuroQuant demonstrates significant advantages in efficiency and practical applicability while maintaining high reconstruction quality.

We have updated the manuscript to include additional details regarding encoding time, entropy-based variable bitrate methods, and extended ablation experiments. Thank you again for your valuable feedback and efforts in helping us improve the manuscript.

Ref:

1. Gomes, Carlos, Roberto Azevedo, and Christopher Schroers. "Video compression with entropy-constrained neural representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

2. Zhang, Yunfan, et al. "Implicit neural video compression." arXiv preprint arXiv:2112.11312 (2021).

3. Kwan, Ho Man, et al. "Immersive Video Compression using Implicit Neural Representations." arXiv preprint arXiv:2402.01596 (2024).

We sincerely thank the reviewer for their insightful feedback. Below, we clarify and address the concerns raised, and hope you can find them helpful:

Q1: The authors should consider reporting the pretraining and fine-tuning times separately for both the baseline models and NeuroQuant.

A1: Thank you for this suggestion. In pretraining, all baseline models (e.g., NeRV, HNeRV, and HiNeRV) were trained for 300 epochs. In our implementation, all post-training quantization (PTQ) and fine-tuning techniques use the same pretrained model. Below, we report the pretraining and calibration times for NeuroQuant on an Nvidia RTX 3090 GPU using 32-bit floating-point (FP32) training precision:

Q2: Similarly, the claim of an 8x encoding speedup is also misleading, as it excludes the pretraining time required for INR-VC encoding (even though NeuroQuant avoids full retraining for each rate point).

A2: Thank you for pointing this out. We have updated the manuscript to include pretraining time in Table 2.

Q3: Variable/learnable quantization levels for INR-VC have been explored in related works [1,2,3], so the paper’s claim is inaccurate (e.g., line 43). These methods should be discussed and compared in the paper.

A3: Thank you for highlighting this. The related works [1, 2, 3] mentioned are entropy-based methods that use weight entropy regularization to constrain bitrate. Due to the limited rebuttal period, reproducing all three works was infeasible, especially since none of them are open-source. However, we try our best to reproduce [1] (Gomes et al.) as a representative entropy-based method. Our reproduced results are basically consistent with original [1] (NeRV-EM in original paper).

Using the NeRV architecture as the baseline, we pretrained all models for 300 epochs. The comparison results are shown below:

The results demonstrate that NeuroQuant consistently outperforms Gomes et al. [1] across all sequences and bitrates. Discussions of these related works have been added to the Appendix E.2 of the revised manuscript and will be integrated into the main paper in the final version after peer review.

Q4: For a fairer comparison, the comparisons to x264/x265 should avoid using the 'zerolatency' setting, as the INR-VCs in the paper inherently have non-zero latency.

A4: We apologize for the oversight in our execution of x264/x265 coding. The results in Figure 4 and the corresponding command details in Appendix E.1 have been updated without the "zerolatency" setting.

Q5: For ablation study, more sequences should be use for obtaining a representative result.

A5: Thank you for the suggestion. We have expanded our ablation study to include the Beauty and ReadySetGo sequences in addition to the original Jockey sequence. The updated results can be found in Appendix E.3. We appreciate your effort to enhance the clarity and comprehensiveness of the article.

Q6: Are there any differences in configuration, such as the use of FP16 precision in HiNeRV experiments?

A6: To ensure fair and consistent comparisons across all baselines and benchmarks, we conducted all experiments in the same environment using FP32 training precision and PyTorch 1.10. The encoding runtime and memory usage reported in the paper reflect this uniform setup.

For comparison, we further evaluated HiNeRV-3M using FP16 precision, as employed in the original HiNeRV paper. FP16 training was implemented using autocast and GradScaler from torch.cuda.amp. The results are summarized below:

We found that FP16 precision significantly reduces encoding time for both naive HiNeRV and NeuroQuant, while maintaining performance comparable to FP32 (e.g., within $\pm 0.1$ dB). In the future, we plan to migrate all baselines and benchmarks to PyTorch 2.0 for consistency, as done in the original HiNeRV implementation.

Thank you for the authors' response. The response and revised manuscript have addressed my concerns and questions. I will raise the rating accordingly.

We would like to express our sincere gratitude for your thoughtful feedback and for taking the time to review our work. We greatly appreciate your recognition of our proposed method and the significance of our contributions. We are pleased to note that your understanding of our paper aligns well with the insights provided by other reviewers. Below, we address your question regarding bitrate adjustment:

Q1: How does NeuroQuant differ from traditional quantization methods in terms of bitrate adjustment?

A1: Current video codecs can be broadly categorized into two types: (1) Generalized codecs: These methods used a generalized codec to extract features to various videos, including both handcrafted approaches (e.g., H.264 [1], H.265 [2]) and deep learning-based techniques (e.g., DCVC-DC [3]). (2) Non-generalized INR-VCs: These represent a specific video using a unique network, where video coding is transformed into encoding and transmitting network weights.

Compared with Quantization in Generalized Codecs: Traditional quantization methods [3-5] often focus on quantizing feature maps (also named latents) to control bitrate, an approach is fundamentally different with weight quantization in INR-VCs. In contrast, NeuroQuant is the first to achieve bitrate control by adjusting the quantization parameters (QPs) of pretrained weights in non-generalized representations. Specifically, NeuroQuant introduces: (1) Fine-grained bitrate control by redefining quantization as a mixed-precision problem. (2) QPs optimization through network-wise calibration tailored for INR-VCs.

Compared with Weights Quantization: NeuroQuant addresses two critical challenges when applying weight quantization to non-generalized INR-VCs: (1) Inter-layer Dependencies: Traditional methods assume inter-layer independence, which does not hold for INR-VCs. NeuroQuant incorporates weight perturbation directionality and off-diagonal Hessian information to address this. (2) Calibration Granularity: Existing layer/block calibration methods are ineffective for INR-VCs. NeuroQuant introduces network-wise calibration to optimize QPs.

We apologize for the earlier omission of related background details. A discussion on bitrate adjustment has now been included in the Appendix A of the revised manuscript and will be incorporated into the final main paper after peer review.

Your perspective is invaluable in deepening the understanding of our work. Thank you again for your positive feedback. We hope this information is helpful and enhances your confidence in assessing our contribution.

Ref:

1. Wiegand, Thomas, et al. "Overview of the H. 264/AVC video coding standard." IEEE Transactions on circuits and systems for video technology 13.7 (2003): 560-576.

2. Sullivan, Gary J., et al. "Overview of the high efficiency video coding (HEVC) standard." IEEE Transactions on circuits and systems for video technology 22.12 (2012): 1649-1668.

3. Li, Jiahao, Bin Li, and Yan Lu. "Neural video compression with diverse contexts." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

4. Zhao, Tiesong, Zhou Wang, and Chang Wen Chen. "Adaptive quantization parameter cascading in HEVC hierarchical coding." IEEE Transactions on Image Processing 25.7 (2016): 2997-3009.

5. Wang, Hanli, and Sam Kwong. "Rate-distortion optimization of rate control for H. 264 with adaptive initial quantization parameter determination." IEEE Transactions on Circuits and Systems for Video Technology 18.1 (2008): 140-144.

Q4: For Nagel et.al (2020) which formulation was used to compare?

A4: The network calibration objective of NeuroQuant is described by the function $\mathcal{F}(x, w)$ in equation (16) of our manuscript, where $\mathcal{F}$ represents the networks's output for a given input $x$ and weights $w$.

In contrast, AdaRound conducts layer-wise calibration, and $\mathcal{F}$ simplifies to the layer function (not activation function). Specifically, $\mathcal{F}(x, w)$ is equivalent to $f_a(wx)$ in equation (25) of AdaRound, where $f_a$ is the activation function. Therefore, in our implementation, AdaRound use equation (25) for layer calibration.

Note that not all layers in various INR-VCs are followed by an activation function. For such cases, we employ equation (21) of AdaRound.

Q5: In equation (16), how the $s$ is determined?

A5: In our approach, $s$ represents the quantization steps. Once the mixed precision scheme is determined, each layer is assigned a fixed bitwidth $b^l$. Then the initial value of $s$ is computed channel-wisely using

$

    s^{l,k} = \frac{\max (w^{l,k}) - \min (x^{l,k})}{2^{b^l} - 1},

$

where $w^{l,k}$ is the weights in the $k$-th channel of the $l$-th layer. During the calibration process, $s$ is further optimized to minimize task loss.

Q6: the network-wise calibration might also be applicable to the generalized neural network, did the authors have done any experiments on the generalized neural codec?

A6: In generalized neural codecs, inter-layer dependencies are weaker due to their general-purpose representation. Furthermore, in the context of post-training quantization (PTQ), generalized codecs are more prone to overfitting when using small datasets for network-wise calibration, leading to higher testing errors.

To explore the applicability of network-wise calibration to generalized codecs, we implemented the open-sourced RDO-PTQ, modifying its default layer/block-calibration approach to use network-wise calibration. We evaluated the generalized codec on three datasets using BD-Rate to jointly measure bitrate and PSNR, where the anchor is FP32 model:

As shown, network-wise calibration performs worse than block calibration in generalized codecs. This suggests that plain network-wise calibration is not directly suitable for generalized codecs. A potential strategy is to introduce a modular granularity approach, allowing the quantization to transition between layer-, block-, and network-wise calibration, depending on the specific models and use case.

Q7: For the quantization aware training approach, the weights initialization with post-training quantization will improve the convergence and reconstruction quality of the QAT. It would be nice to test this feature on some INR method which uses QAT.

A7: Thank you for this valuable suggestion. We tested the impact of different PTQs for weight initialization in QAT from HiNeRV, using the NeRV (3.1M). The results for 4-bit quantization are presented below:

Unfortunately, initializing weights with PTQ did not improve QAT for NeRV. This may be due to the unique characteristics of non-generalized INRs. Due to time constraints, we could not test additional sequences, QAT methods, or INR architectures. However, we recognize this as an important area of investigation and will explore other unique properties of non-generalized INRs further in future work.

Due to the limited rebuttal period, we regret that we are unable to provide additional experimental results at this time. Following the peer review process, we will integrate these findings into the final version of the main paper. We believe that the novel views presented by you not only enhance the clarity of our manuscript but also draw attention from the research community to the unique characteristics and potential of non-generalized INRs.

Thank you again for your time, effort, and valuable feedback in reviewing our manuscript.

Ref:

1. S. Wiedemann et al., "DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks," in IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 4, pp. 700-714, May 2020.

2. Dong, Zhen, et al. "Hawq-v2: Hessian aware trace-weighted quantization of neural networks." Advances in neural information processing systems 33 (2020): 18518-18529.

We appreciate the reviewer’s thoughtful review and constructive feedback. Below is the detailed response to each question, hope you can find them helpful.

Q1: The authors should compare their method with NNC [1].

A1: We appreciate the reviewer bringing up NNC [1]. We conducted a comparison between NeuroQuant and DeepCABAC [1], using the NeRV with 3.1M parameters across three video sequences. For DeepCABAC, different bitrates were achieved by adjusting the quantization step. The results are summarized in the following:

As shown, NeuroQuant consistently outperforms DeepCABAC across different sequences and bitrates. Additionally, the lossless entropy coding used in NeuroQuant currently is less advanced compared to CABAC in [1]. In future work, we aim to incorporate CABAC into NeuroQuant to further enhance performance.

Q2: How the figure 3 is generated.

A2: We apologize for the lack of clarity in our explanation of Figure 3. Below, we provide more detailes:

Network and Data: All subfigures in Figure 3 are based on the HNeRV architecture with 1M parameters, and the Beauty sequence is used for fitting.

Subfigure (a) Layer statistics: We analyze the first convolution layer (blue) and the third convolution layer (orange). The histograms depict the relative number of parameters within narrow intervals, while the solid lines represent fitted probability distributions.

Subfigure (b) Channel's statistics: We compute the maximum and minimum values across 84 channels in the fifth layer. This highlights the variability of weights among channels.

Subfigure (c) Hessian statistics: The Hessian matrix $\mathbf{H}$ is derived as second-order information, where each entry $\mathbf{H}(i,j)$ represents $\frac{\partial \mathcal{L}^2}{\partial w_i \partial w_j}$, indicating the dependency between $i-th$ and $j-th$ weights. For HNeRV, we calculate the Hessian matrix across its 7 layers. In the visualization, we show the upper triangular Hessian matrix because it is nearly symmetric around the principal diagonal (the matrix is nearly Positive Semi-Definite (PSD) for a converged model).

Applicability to MLPs: The analysis is fundamentally applicable to MLPs since the illustrated statistics are based on general principles of twice-differentiable neural networks, which is not constrained by the network's specific architecture. We will release the related code upon the acceptance of this paper.

Q3: What is the significance of the equation 5 and 6. Also detail how the mixed-precision quantization is used.

A3: We appreciate the reviewer’s comments and provide the following clarifications:

Eq. 6 formulates the first key sub-problem of NeuroQuant: allocating bitwidths (i.e., mixed precision) to layers given a target bitrate $\mathcal{R} + \epsilon$. However, a specific bitrate can correspond to multiple mixed-precision configurations. Hence, it is crucial to determine the optimal allocation scheme. In Sec. 3.1, we analyze existing mixed-precision criteria and identify their shortcomings for non-generalized INRs due to significant inter-layer dependencies and weight perturbation directionality. We then propose a new criterion incorporating off-diagonal Hessian information and perturbation directionality.

Eq. 5 Once the bitwidths are allocated, the quantization parameters (QPs) can be initialized. Equation 5 describes the second sub-problem: calibrating initialized QPs to minimize the loss caused by quantization. This ensures better reconstruction quality. In Sec. 3.2, we derive a unified calibration formula based on equation 5, and introduce network-wise calibration and channel-wise quantization strategies.

Where is Mixed-Precision Result: (1) Table 1: We first evaluate consistent bitwidths (e.g., INT6 and INT4). Mixed-precision results are included for INT2, marked with *. The results demonstrate NeuroQuant’s superiority across all configurations. (2) Sec. 4.2: The variable-rate coding experiments directly utilize mixed-precision quantization, showcasing its effectiveness in achieving better performance and reduced encoding time. (3) Fig. 5 (left subfigure): This directly compares our mixed-precision method against HAWQ-V2 [2], demonstrating the validity of our theoretical contributions.

Algorithm: we provide a pseudo-algorithm detailing the mixed-precision process in the updated Appendix D.

Thanks for addressing my concerns. The comparison with DeepCABAC is interesting. DeepCABAC determines the quantization step based on the reconstruction loss in the weights space which is different from NeuroQuant. If the DeepCABAC determines the quantization step based on the task loss, then the performance of the deepCABAC will be improved.