Neural B-frame Video Compression with Bi-directional Reference Harmonization

We sincerely appreciate your affirmation and recognition of our work. Here are our detailed responses to your concerns, and we hope they will clarify our work.

Weakness 1: The difference between Baseline and DCVC-B (re-trained).

At the onset of this work, given that DCVC-B had not yet released its source code, we referred to its design to reimplement the Baseline. Subsequently, once the DCVC-B framework became open-source, we re-trained it using our training settings to obtain DCVC-B (re-trained). As a result, there are differences in the implementation details between the two. The main difference is as follows:

The Baseline and BRHVC first dimensionizes the references to match the dimensions of latent features and then feeds them into Encoder/Decoder, while DCVC-B directly input them. Taking the largest size as an example, the process of integrating reference information into the Encoder is as follows:

DCVC-B: $y^1_r=Conv^1(cat[x,C^1_f,C^1_b])$,

Baseline: $C^1_t=RB^1(C^1_f, C^1_b)$ then $y^1_r=Conv^1(cat[x,C^1_t])$,

BRHVC: $C^1_t,...=BCF_E(C^1_f, C^1_b,...)$ then $y^1_r=Conv^1(cat[x,C^1_t])$,

where $y^1_r$ denotes the latent feature integrated with reference information, $Conv^1$ denotes a convolution layer in the Encoder, $cat[\cdot]$ denotes concatenation along the channel dimension, $RB^1$ denotes stacking of some ResBlocks.

It should be noted that tensor $C_t$ has the same dimensions as $C_f$ and $C_b$, which reduces the computational overhead of the Baseline and BRHVC in the Encoder/Decoder compared to DCVC-B.

As a result, the Baseline has fewer kMACs and memory overhead compared to DCVC-B, but its decoding speed is slightly slower. This may indicate that there is still room for optimization in our code, which we will attempt to improve in future work. Moreover, since there is not much difference in the overall mechanism, the performance of the two is also quite close.

Weakness 2: Comparison under equivalent complexity.

This question is similar to Reviewer BaaM's Question 1. As shown in Table 4, we constructed a more complex model (denoted as Baseline w/ Resblocks) for comparison. Experimental results show that merely increasing model complexity to match BRHVC’s encoding/decoding speed yields only a 4.1 % improvement—far below BRHVC’s 12.3 %. It confirms that BRHVC indeed enhances the performance effectively.

We are pleased to see your feedback and thank you for your timely and constructive comments. Below are our explanations in response to your supplementary questions.

Supplementary Question 1: Cases of complex motions.

The experiments in Table 10 demonstrate that BRHVC significantly outperforms DCVC-B on datasets with more complex motions. The sequences BasketballDrill and RaceHorses represent complex motion scenarios, ParkScene represents a relatively ordinary motion scenario, and FourPeople represents a scene with small motion. Specifically, BasketballDrill is a multi-person basketball scene with many moving targets and complex body movements; RaceHorses is a multi-person horse-riding scene with numerous occlusions; ParkScene is a park scene with a few people cycling by, featuring relatively stable motion; and FourPeople is an indoor meeting scene with small motion.

Table 10: Performance comparison between BRHVC and DCVC-B on representative sequences in the HEVC dataset.

Experimental results show that BRHVC achieves the highest gains in the complex motion sequences BasketballDrill and RaceHorses, with improvements of 19.1% and 18.5%, respectively. In contrast, the smallest gain is observed in the small motion sequence FourPeople, with only 2.3%. In the ordinary scene ParkScene, a 9.9% improvement is achieved. Since the URC issue becomes more severe with more complex motions, these results indicate that BRHVC is more effective at addressing the URC issue.

Supplementary Question 2: Impact of frame rate.

Theoretically, for the same content, a lower frame rate implies larger inter-frame motion, which amplifies the impact of the URC issue. Among the 6 datasets, MCL-JCV has the lowest overall frame rate and the highest resolution, which may lead to the most severe URC issue. Table 1 shows that DCVC-B's average BD-rate is -17.2%, but only -2.0% on MCL-JCV. In contrast, BRHVC's average BD-rate is -27.3%, and -16.1% on MCL-JCV. It indicates that the network architecture designed to address the URC issue can effectively compensate for the previous shortcomings.

We sincerely thank your time, effort, and professional expertise. We have carefully considered your concerns and provided detailed answers, hoping these will make our work more solid.

Weakness 1: Enhance collaboration between the two modules.

This is a great idea. Perhaps we can implement a mechanism similar to a 2-pass or coarse-to-fine approach. It would allow the two modules to interact and harmonize with each other, thereby enhancing performance. However, it might introduce some complexity. In any case, thank you for your suggestion. We will try to incorporate it in our future work.

Weakness 2: Is the computational cost "acceptable"?

We believe the overhead is indeed "acceptable" given the performance gains achieved. To verify it, we use the most critical metrics—percentage overhead of decoding time over the baseline and BD-rate gain based on its baseline as the criterion.

Table 6: Gain–overhead comparison across some methods.

References

[s1] Augmented deep contexts for spatially embedded video coding.

[s2] Neural video compression with context modulation.

The results are shown in Table 6. Although the anchor of gains may be not strictly aligned, it still provides partial evidence. It can be seen that our method’s trade-off between gain and overhead falls within an acceptable range. Several additional factors should be considered:

First, coding efficiency exhibits diminishing returns: DCVC-DC (2023) achieves an attractive gain-overhead ratio, whereas the two 2025 works [s1,s2] require substantially higher costs for further improvements.

Second, RA coding inherently processes more information than LD coding, so any enhancement built upon it is more prone to increased overhead. However, unlike LD coding, RA theoretically allows parallel encoding of multiple frames at the same level, which can further reduce the overall complexity when deployed in practice.

Third, RA is not designed for low-latency requirements. Much like the historical development of LD, it may be more appropriate for RA to first explore performance-boosting mechanisms and then seek complexity reduction.

In conclusion, we believe the overhead is indeed "acceptable" given the performance gains achieved.

Weakness 3: Performance of Baseline w/ BMC.

As shown in Table 4, using BMC alone (denoted as Baseline w/ BMC) can save 6.4% bitrate. Moreover, the complexity of each module has been explicitly listed.

Weakness 4 : Other image-quality metrics.

We adopt MS-SSIM, the most commonly used alternative metric besides PSNR, for evaluation. Following the DCVC-B setup, we fine-tune our MS-SSIM model for only 2 epochs, starting from the MSE-optimized model. The results in Table 7 show that BRHVC significantly outperforms VTM-RA in terms of MS-SSIM. Due to the time constraints of the rebuttal period, we can not thoroughly fine-tune the model, which may limit further performance gains relative to DCVC-B.

Table 7: BD-rate comparison for MS-SSIM. DCVC-B and BRHVC are both optimized by MS-SSIM in fine-tuning.

Weakness 5, Quesion 3: Performance on UVG and MCL-JCV.

This Weakness is similar to Reviewer 5ESm's Question 2. The primary reason is the absence of high-resolution training data in our dataset. Please see it for more details.

Weakness 6, Question 2: Experiments with different IPs.

We conducted experiments with different IP settings in Table 8. Since the configuration files provided by HM-RA and VTM-RA have a maximum IP of 32, we performed experiments with IP settings of 8, 16, and 32. For the sake of expediency, we use the first 33 frames on HEVC for testing. Moreover, since the bitrate proportion of I-frames varies significantly across different IP settings, we also provide the BD-rate computed only over B-frames, denoted as BD-rate w/o I-frame.

Table 8: BD-rate variations across different IP settings on the first 33 frames.

The results show that, as IP increases, the frame span also increases in some hierarchical levels, thereby increasing the severity of the URC problem. Under this condition, the improvement of BRHVC over DCVC-B also increases (from -15.3% to -21.6%). This indicates that our method successfully handles URC while DCVC-B does not, thereby aligning with our motivation.

Question 1: The difference between Baseline and DCVC-B (re-trained).

This Question is similar to Reviewer L376's Weakness 1. Please see it for more details.

We sincerely thank you for your positive feedback on our work. Below are our responses to your concerns, and we hope these efforts can better clarify our work.

Weakness 1: Computationally excessive of BMC.

It is indeed the case that invoking SpyNet three times may seem excessive, especially for small motions, as you pointed out. However, as shown in Reviewer BaaM's Question 2, on average, the optical flow at each scale has made an indispensable contribution. Thus, we think retaining three separate flows remains the best compromise for now. In future work, we intend to migrate to a single-pass architecture that delivers all three flows in one inference. Thank you for your comment, it has reinforced this plan. Nevertheless, BRHVC currently still achieves good performance under the same complexity (see Baseline w/ Resblocks in Table 4).

Weakness 2: More detailed computational complexity.

Refer to Table 4, we provide more detailed metrics, including kMACs/pixel and max memory usage. Specifically, BCF adds approximately 700 kMACs/pixel and 50 ms to decoding latency, while BMC adds about 340 kMACs/pixel and 80 ms. Although BRHVC increases complexity, as can be seen from Table 4, our method is more effective for methods of equivalent complexity (i.e., Baseline w/ Resblocks).

Weakness 3: Performance comparison with VTM-RA.

Although BRHVC underperforms VTM-RA on some datasets, it indeed achieves a substantial improvement over previous Nerual Video Compression (NVC) methods, outperforming them on all datasets (see Table 1). Additionally, the gap with VTM-RA is mainly observed on UVG and MCL-JCV, which will be analyzed in Question 2.

Question 1: More details on Contextual-Feature Extraction (CFE).

The CFE module combines Temporal Context Mining [20, 32] with the post-processing Group-Based Offset Diversity mechanism [20]. For conciseness, we merge the two modules into one, denoted as CFE. The role of CFE is to extract reference information from $F_f,F_b$ and perform motion compensation on them with optical flows. To convey this point more clearly, we will refine the wording in this part. Thank you for your feedback.

Question 2: Performance on UVG and MCL-JCV.

We speculate that the main reason for our inferior performance compared to VTM-RA on MCL-JCV and UVG is that these datasets are high-resolution (1920×1080), whereas the Vimeo-90K training set commonly used for Neural Video Compression (NVC) lacks such high-resolution data. Besides BRHVC, this shortcoming results in generally poor performance of NVCs methods on these datasets.

To verify this, we downsampled the two datasets and re-evaluated them against VTM-RA. We resized the images to 480×270 and, for efficiency, selected the first 33 frames for testing. As shown in Table 9, the performances of NVC methods overall show significant improvements in low-resolution settings, with BRHVC even outperforming VTM-RA noticeably.

Table 9: BD-rate comparison on downsampled MCL-JCV and UVG.

Most NVC methods train on cropped 256×256 patches, which aids training stability and conserves computational resources but limits the model's ability to handle larger spatial contexts (e.g., 1080P images). We encourage the NVC community to further explore this issue, which may lead to further improvements in the performance of NVC. Despite these challenges, our method has made considerable progress, bringing NVC closer to VTM-RA than ever before.

Question 3: Comparison with multimodal-based method.

Regrettably, we believe it may not be feasible to make a fully fair comparison between BRHVC and these approaches. First, there are differences in research focus: their methods target subjective metrics (e.g., LPIPS) at extremely low bitrate, while our work aims for objective metrics (i.e., PSNR) and video fidelity at medium to high bitrate. Second, the lack of open-source code and PSNR results for M3-CVC further complicates direct comparison. Nevertheless, we will open-source our model in the future, enabling other researchers to compare it with other video coding methods across different fields.

We sincerely thank you for your detailed and instructive reviews, and for recognizing our contributions to B-frame coding. Below are some responses that we hope will address your concerns.

Weakness 1,2: Additional metrics and comparison under equivalent complexity.

We conducted more comprehensive comparison on additional metrics such as kMACs and Max Memory. Furthermore, to verify whether simply stacking modules can yield sufficient gains, we construct a more complex model (denoted as Baseline w/ Resblocks) for comparison. The results are as follows:

Table 4: Ablation study on different network designs (based on Table 2 in the main text).

Considering that inference speed is influenced by many factors and may not always correlate directly with kMACs or memory usage, as noted in [15] and other studies, we suggest using actual coding time as a more reliable metric in this case. Experimental results show that merely increasing model complexity to match BRHVC’s encoding/decoding speed yields only a 4.1 % improvement—far below BRHVC’s 12.3 %. It indicates that the improvement of BRHVC stems from the motivation behind our design, rather than merely from increased complexity. Moreover, Table 6 in Reviewer iau7's Weakness 2 indicates that the increase in complexity resulting from our method design is acceptable.

Here is a detailed description of Baseline w/ Resblocks. To match the encoding/decoding speed of BRHVC, we stack 9 ResBlocks per scale (three scales in total) for reference fusion in the encoder and 4 ResBlocks per scale in the decoder. In contrast, the baseline employs only 1 ResBlock per scale at both the encoder and decoder. Please refer to Reviewer L376’s Weakness 1 for additional details of the Resblocks (denoted as $RB$).

Further ablation experiments on BMC can be found in Question 2.

Weekness 3: How existing models fail to use reference information.

In the main text, we have provided explanations of our design principles based on existing models and demonstrated the visualization effects. To further validate the effectiveness, we conducted additional comparative experiments. Below are the details.

First, existing methods did not have specialized designs for the URC issue. As stated in lines 170–174, existing methods [33,34] do not explicitly model the weights of the two reference frames; instead, they perform reference fusion directly via concatenate&convolution. The weights of references are learned in training and fixed after training, which can not adapt to the value of references. In contrast, as shown in Equation 3, our BRHVC explicitly models the weights with $s_f$, $s_b$, and $b_t$.

Second, the visualization in Figure 8 shows that our design performs as expected. BRHVC focuses its attention on more valuable regions, and this mechanism enables it to handle the URC issue more effectively.

Third, experiments demonstrate that BRHVC is capable of handling the URC issue more effectively. The experiments in Table 4 show that BRHVC achieves a significant overall performance improvement. Moreover, the experiments in Table 8 (from Reviewer iau7's Weakness 6) show that BRHVC performs better than DCVC-B when the URC problem is more severe.

Question 1: Experiment of removing BCF/BMC.

The newly added Table 4 presents the experimental results for these modules, where Baseline w/ BCF,BMC and Baseline corresponds to our BRHVC and BRHVC w/o BCF,BMC, respectively. For details on the baseline design, please refer to Reviewer L376's Weakness 1.

Question 2: Validate optical flows at different scales.

The optical flows at each scale contribute to a noticeable performance improvement. To investigate the role at each scale, we designed the following experiment: we iteratively replace the optical flows at one scale with the downsampled/upsampled version from another scale and then fine-tune the network until convergence. The results are in Table 5, where "$(\cdot)\uparrow_2$" denotes 2× upsample, "$(\cdot)\downarrow_2$" denotes 2× downsample. "$v^1, v^2, v^3,\hat{v}^1,\hat{v}^2,\hat{v}^3$" are all shorthand for the corresponding scales: specifically, "$v^1$" denotes $v^1_{t,f}, v^1_{t,b}$, and likewise for the others. For example, Method M1 replaces the largest-size flows $v^1,\hat{v}^1$ with the intermediate-scale flows $v^2,\hat{v}^2$, its result measures the performance change caused by the absence of $v^1,\hat{v}^1$. Likewise, M2 and M3 denote replacing the intermediate- and smallest-size optical flows, respectively.

Table 5: Contributions of the optical flows at different scales.

Experimental results demonstrate that the flow at every scale provides a noticeable gain, demonstrating that leveraging additional flows indeed enables more effective motion compensation. Moreover, optical flows at different scales exhibit varying contributions. As the spatial resolution decreases, the contribution of the optical flow at that scale diminishes due to the reduced amount of information it carries.

Question 3:  Carify the two main B-frame coding strategies.

Thank you for your feedback. We will specifically distinguish the type of coding in the manuscript.