3D-LMVIC: Learning-based Multi-View Image Compression with 3D Gaussian Geometric Priors

Thanks to you for the valuable comments. We are grateful for your positive feedback, especially regarding the model performance. We address your remaining concerns as follows:

R1[The training time of the 3D Gaussian]

---

For the Train scene of the Tanks&Temples dataset, which contains 301 images, we conducted experiments on a platform equipped with an Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz and an NVIDIA RTX A6000 GPU. Training the 3D Gaussian representation for 30,000 iterations took approximately 13 minutes and 35 seconds, averaging 2.71 seconds per image. Additionally, the average encoding time per image is 0.19 seconds. Therefore, the total time to compress a single image—including both training and encoding—is approximately 2.90 seconds.

R2[Efficient 3D Gaussian training approach]

---

We tested a reduced training setup with 7,000 iterations, which resulted in a total training time of 2 minutes and 22 seconds, or 0.47 seconds per image. Including the image encoding time, the average total time to compress each image is 0.66 seconds.

Additionally, we assessed the alignment performance of depth estimation using the 3D Gaussian representation trained for 7,000 iterations under the same experimental conditions as detailed in Table 2 of the main text. The results are summarized in the table below:

Despite the significant reduction in training iterations, it is evident that the proposed alignment method still achieves high alignment accuracy.

R3[Explain the role of the image context transfer module]

---

The image context transfer module is designed to incorporate features from the reference view into the current view to enhance the overall feature representation. Its functionality is similar to the joint context transfer (JCT) module proposed in LDMIC [1]. While LDMIC adopts a cross-attention mechanism to fuse reference features, our approach performs pixel-wise alignment for feature integration.

Specifically, due to the disparity between the two views, the module first aligns the reference view features to the current view using the estimated disparity. The aligned features are then concatenated with the current view's features.
Subsequently, a feature mask is applied to the combined features to filter out non-overlapping regions and retain only the relevant contextual information. Finally, the resulting features are passed through a residual block to produce the enhanced feature representation.

[1] Zhang, Xinjie, Jiawei Shao, and Jun Zhang. "LDMIC: Learning-based Distributed Multi-view Image Coding." The Eleventh International Conference on Learning Representations.

R4[Clarify the greedy algorithm for multi-view sequence sorting]

---

Thank you for pointing out this issue. We provide a more detailed explanation of the algorithm below.
Specifically, given a multi-view sequence $v_1, v_2, ..., v_n$, the algorithm starts from the first view $v_1$ and iteratively selects the view that is closest to the current one. For example, if $v_i$ is the closest to $v_1$, it is placed immediately after $v_1$. The process then continues by finding the closest view to $v_i$ among the remaining views and placing it next. This procedure is repeated until all views are ordered into the sequence.

Thank you for your valuable comments! We are grateful for your positive feedback, especially regarding the proposed methods and the ablation experiments. We address your remaining concerns as follows:

R1[Clarify alignment verification and target view selection]

---

Given a reference view and a target view, we first align the reference view to the target view. Specifically, each pixel in the target image is projected to the corresponding position in the reference image based on the estimated disparity. The color at each projected location is then obtained through bilinear interpolation from the surrounding pixels in the reference image.
We then compute the similarity between the aligned reference view and the target view using PSNR and MS-SSIM. A higher similarity score indicates better alignment accuracy.

In our alignment experiments, we assume that the ground-truth target view is known and use it as supervision to evaluate alignment quality. Specifically, for a multi-view sequence $v_1, v_2, ..., v_n$, we take $v_i$ as the target view and $v_{i-1}$ as the reference view.

R2[The training time of the 3D Gaussian]

---

For the Train scene of the Tanks&Temples dataset, which contains 301 images, we conducted tests on a platform equipped with an Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz and an NVIDIA RTX A6000 GPU. Training the 3D Gaussian representation for 30,000 iterations took approximately 13 minutes and 35 seconds, or an average of 2.71 seconds per image. We also tested a reduced training setup with 7,000 iterations, which resulted in a total training time of 2 minutes and 22 seconds, or 0.47 seconds per image.

Additionally, we assessed the alignment performance of depth estimation using the 3D Gaussian representation trained for 7,000 iterations under the same experimental conditions as detailed in Table 2 of the main text. The results are summarized in the table below:

Despite the significant reduction in training iterations, it is evident that the proposed alignment method still achieves high alignment accuracy.

R3[Test the depth estimation method on a large-scale scene]

We further evaluate alignment performance on two large-scale scenes (Temple, Church) and one medium-scale scene (Truck) from the Tanks&Temples dataset under identical experimental conditions (Table 2, main text).

• Scenes:
  • Temple: 302 images (outdoor temple)
  • Church: 600 images (indoor church)
  • Truck: 251 images (mid-scale object)

Results are summarized below:

Key Observations:

1. Consistent Superiority: Our method achieves best PSNR across all scenes and best MS-SSIM in 2/3 cases, demonstrating robust performance scalability.
2. Large-Scale Competitiveness: While FlowFormer++ shows marginal MS-SSIM advantage in Temple (+1.6%), our method still leads in PSNR (+3.4%) and dominates in Church (PSNR: +4.0%, MS-SSIM: +4.1%).
3. Mid-Scale Strength: The significant gains in Truck (PSNR: +5.6%, MS-SSIM: +11.7% over FlowFormer++) validate the method's effectiveness for complex object-level scenes.

Interpretation: The 3D Gaussian-based approach maintains strong competitiveness in large-scale settings, with performance variations likely attributable to scene-specific characteristics (e.g., texture uniformity in Temple). Ongoing work focuses on further optimizing large-scale adaptability without sacrificing mid-scale advantages.

Thank you for your valuable comments! We are grateful for your positive feedback, especially regarding the algorithm's running speed and compression performance. We address your remaining concerns as follows:

R1[Test on Cityspace, KITTI, and Instereo2K]

---

1. Limitations of Stereo Images for Learning Geometry in 3D Gaussian Representation
We found that for stereo image datasets with only two viewpoints, the 3D Gaussian representation struggles to learn accurate geometric information. In contrast, multi-view data under wide-baseline setups provides rich scene information sampled from widely varying viewpoints, which helps the 3D Gaussian representation learn accurate geometry. In stereo image datasets, the cameras are often positioned close to each other with similar orientations, making it difficult for the 3D Gaussian representation to correct geometric errors through large viewpoint variations and learn the correct geometric structure. The table below presents the alignment performance of various disparity estimation methods on the Cityscapes and KITTI Stereo datasets under the same experimental settings as the alignment experiments described in Table 2 of the main text.

Experiments have shown that the 3D Gaussian representation cannot learn accurate geometry based on stereo image data with only two viewpoints. The disparity estimation modules in current multi-view image codecs, such as Patch Matching (PM), already perform well on stereo image datasets.

2. Motivation and Practical Applications
We aim to design a multi-view image compression framework tailored for wide-baseline setups to achieve accurate disparity estimation and effectively eliminate inter-view redundancy. Unlike KITTI and InStereo2K datasets, which contain stereo images with small, mostly horizontal disparities captured by closely positioned cameras, wide-baseline setups in datasets like Mip-NeRF 360 and TnT feature irregular view relationships and less consistent disparities. These characteristics make existing disparity estimation methods, such as homography transformation and patch matching, less effective, as shown in Table 2 on the TnT dataset. Wide-baseline setups are also critical for practical applications, where scenes often consist of dozens to hundreds of images, creating significant challenges for storage and transmission.

R2[Questions regarding HAC]

---

1. Reason for including HAC in the comparison:
Although HAC is a 3DGS-based compression method, it inherently encodes multi-view information. Therefore, we believe it is meaningful to compare its performance with dedicated multi-view image compression methods under the same multi-view compression setting.

2. Explanation for HAC’s poor performance:
For evaluation, we use one-third of the images from each scene as test samples. For example, in the Train scene of the Tanks&Temples dataset (301 images), we used 101 images for testing.
Increasing the number of test images is likely to benefit HAC's performance. We will consider this in future work.

The table below presents a comparison between HAC and our proposed 3D-LMVIC method on the Tanks&Temples dataset:

As we can see, HAC's PSNR and MS-SSIM in terms of rendering quality reach a plateau around 30dB and 0.97. Therefore, based on the experimental results, it seems that HAC might perform better with more test images, such as when the number of test images reaches 1,000. On the other hand, HAC does have a rendering quality bottleneck.

3. Comparison between HAC and dual-branch VAE architectures (e.g., HESIC and BiSIC):
HAC compresses the entire 3D scene, not just multi-view images. While this enables novel view synthesis and richer scene understanding, it also leads to larger model sizes.
In contrast, methods like HESIC and BiSIC (dual-branch VAE-based) focus more on efficient representation and compression for the given views.

Thank you for your valuable comments! We are grateful for your positive feedback, especially regarding the quality of writing, theoretical justifications, and experimental results. We address your remaining concerns as follows:

R1[Use of median over weighted average for depth estimation]

---

Since 3D Gaussians are trained under the supervision of real RGB images, some "noisy" Gaussians may emerge that do not affect rendering quality but can bias weighted average depth estimation [1]. In contrast, median depth estimation inherently provides a denoising effect: instead of aggregating depth over all Gaussians along a ray, we select the depth of the Gaussian most likely to represent the true 3D world point. This allows us to ignore many background Gaussians that are occluded by foreground Gaussians and invisible in the rendered image, yet would otherwise negatively influence weighted depth estimation.

[1] Chung, Jaeyoung, Jeongtaek Oh, and Kyoung Mu Lee. "Depth-regularized optimization for 3d gaussian splatting in few-shot images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

R2[Runtime of the 3D Gaussian optimization]

---

For the Train scene of the Tanks&Temples dataset, which contains 301 images, we conducted tests on a platform equipped with an Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz and an NVIDIA RTX A6000 GPU. Training the 3D Gaussian representation for 30,000 iterations took approximately 13 minutes and 35 seconds, or an average of 2.71 seconds per image. We also tested a reduced training setup with 7,000 iterations, which resulted in a total training time of 2 minutes and 22 seconds, or 0.47 seconds per image.

Additionally, we assessed the alignment performance of depth estimation using the 3D Gaussian representation trained for 7,000 iterations under the same experimental conditions as detailed in Table 2 of the main text. The results are summarized in the table below:

Despite the significant reduction in training iterations, it is evident that the proposed alignment method still achieves high alignment accuracy.