Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion

Thank you for your kind recognition and these insightful questions—they are indeed crucial for understanding our work.

Q1: Regarding the computational cost of PFO and practical use cases

A1: For PFO, our goal is to demonstrate the potential of prompt optimization in further improving the overall fidelity, even though it is currently computationally intensive.
Additionally, the PFO introduces no additional latency during decoding, making it acceptable for performance-critical scenarios, such as cloud-based restoration services where computational resources are readily available and the encoding latency is not strictly constrained. However, even without PFO, our method still achieves superior performance compared to existing approaches (as demonstrated in Table 1). This underscores the effectiveness of our core framework. Furthermore, we are actively working on optimizing PFO's efficiency to facilitate its wider practical deployment in future applications.

Q2: Reproduction vs. official PerCo implementation

A2: Since the official PerCo code is not publicly available, we mainly compared with a publicly avaiable version. This reproduced version provides some drop in metrics like LPIPS/MS-SSIM, but better at FID/KID and visual quality. Notbaly, to ensure a fair comparison with both versions: We have also benchmarked against PerCo’s official objective results (in Table 9) and subjective comparisons (in Fig. 20) in Appendix D. We also list the table below. Our method also consistently outperforms PerCo (offical), even when comparing with its strongest metrics like PSNR/LPIPS/MS-SSIM.

We sincerely appreciate the reviewer’s insightful comments and recognition of our work’s potential. The questions raised are highly valuable for improving the robustness and impact of our research.

Q1: Implementation details about compared methods

A1: Thank you for your reminder. To ensure a fair comparison, we evaluated all methods as follows: For open-sourced approaches (e.g., DiffEIC, PerCo, CDC, MS-ILLM, Text-Sketch), we used their publicly available pretrained models. For non-open-sourced methods, we relied on the official results reported in their respective publications (e.g GLC, CorrDiff). Additionally, since the open-sourced HiFiC model operates at a higher bitrate, our data was sourced from the reproduced version by the DiffEIC's author. For PerCo, the performance of the open-sourced version is slightly different with the offical results, so we provided comparison with both version.

Q2: Choice of Stable Diffusion v2.1 as Backbone

A2: Thank you for raising this insightful question. It is possible and in line with expectations that larger and better diffusion models can have improved performance. The choice of Stable Diffusion v2.1 as the backbone was primarily driven by two key considerations:

• Fair Comparison with Existing Works: Current state-of-the-art methods (e.g., PerCo, DiffEIC) adopt SD v2.1 as their base model. To ensure a direct and equitable evaluation of our method’s performance improvements, we maintained consistency in the backbone architecture.
• Complexity-Performance Balance: While advanced models like FLUX could potentially enhance compression performance, their large parameter size (12B) and high GPU memory usage make them impractical for real-world applications for image compression. Therefore, we opted to use smaller models to validate the reliability of our method.

We fully acknowledge that leveraging newer diffusion models is a promising direction for further improvements. In fact, we are actively working to secure additional computational resources to explore such extensions in future research. At the same time, we are also investigating how to better utilize the quantized version of FLUX to enhance our performance. Your suggestion aligns perfectly with our roadmap for advancing this work.

Q3: User Studies and Perceptual Validation

A3: To fully demonstrate the comparison of subjective quality, we have included some example images in the main text and appendix. However, we fully acknowledge that user studies are indispensable for comprehensive validation.

After concentrated testing efforts, we're pleased to share some initial subjective user study results(https://anonymous.4open.science/api/repo/public-E45F/file/user_study.png?v=0cb25072). Due to time constraints, we conducted a preliminary evaluation at two bitrate points with 20 participants. Each participant assessed randomly selected images from the DIV2K test set (30 images per bitrate point). The final results were calculated as the average of all votes. Subjective tests confirm our method's clear visual advantage over baselines. We will keep working on providng a more comprehensive user studies.

Thank you again for the constructive feedback—we will incorporate these suggestions to strengthen our research further.

Q1: What is the difference between conditioning the diffusion model directly with visual cues instead of explicit texts

A1: In our method, explicit texts primarily compensate for semantic gaps in the visual condition, especially at ultra-low bitrates. For example:

• Low-bitrate regime: In extreme low-bitrate scenarios where compressed features may retain only basic structural information (e.g., object locations and edges), relying solely on these limited visual cues can lead to vital semantic mismatch (e.g cat to dog). Our method adds text descriptions to fill in missing details, bridging the semantic gap from heavy compression. The subjective results can be viewed at: https://anonymous.4open.science/api/repo/public-2D3D/file/test2.jpg?v=1ef61402
• High-bitrate regime: As the bitrate increases, the visual condition retains sufficient information, and the reliance on text gradually diminishes. This is empirically validated in Figures 9,10 of our paper, where the residual text length drops to zero at higher bitrates. Thus, visual conditions and texts are synergistic: the former anchors structural fidelity, while the latter contains compact high-level semantics (visual cues contain richer and preciser information, while texts are more compact). Our hybrid solution can utilize both sources to achieve overall improved performance.

Q2: Would there be information loss during semantic residual retrieval

A2: During the process of semantic residual retrieval, what we extract is not directly the semantic content of residual features, but rather, as illustrated in Figure 6, we use MLLM to extract the residual information between the original image $x$'s caption and caption of the compressed image $x'$: $caption_{res}=MLLM(x)-MLLM(x')$, instead of $MLLM(x-x')$ . The table below shows the results of semantic residual retrieval for different models (GPT-4o, Llama-3.2-11B) in Kodak.

Therefore, the semantic residual in the form the text is usually conveyable as long as the text description matches with the input image. In terms of text-level information loss, it mainly depends on the capability of the underlying MLLM models as shown here in the table.

From the table, we can observe that since GPT-4o possesses stronger capabilities, the LPIPS between reconstructions after SRR and those from the original captions remains nearly identical. In contrast, Llama-3.2-11B shows some degradation, suggesting potential loss of certain keywords in the process.

However, as you have rightly considered, the semantic residual represented in texts may not perfectly align with the pixel-level residual signal. Hence, we proposed the PFO module to compensate for the feature information that natural language cannot describe. Please correct us if we misunderstand the question you raised.

Q3: Why would such “gibberish” prompts be beneficial to guiding the denoising process

A3: Thank you for raising this insightful question. Regarding this issue, we provide the following clarification of the PFO pipeline: A word is first converted into an embedded token space, and the PFO performs search in the token space through gradient backpropagation guided by output image reconstruction quality. The token vector is iterally updated and finally converted back as a new word. Semantic control in diffusion models operates through CLIP's text embedding space rather than surface-level language.

PFO performs gradient-based search in this latent token space, optimized solely for image reconstruction quality without explicit linguistic constraints. Therefore, the PFO may break the grammar and readability of the word. Consequently, the generated captions may not always exhibit high readability (though still retaining interpretable tokens such as "DBZ (Dragon Ball Z)," "pagoda," and "temples," as shown in Figure 23).

However, they can achieve higher semantic alignment with the original image in CLIP's embedding space. The high-dimension token space encapsulates information beyond the expressive capacity of human-readable language, thereby enabling more effective guidance.

Q4: Figure 5 is not properly referred to in the main text

A4: Thank you for your reminder. The principle of Figure 5 is indeed consistent with our response to Q3, and we will subsequently add a detailed explanation in the main text.

Thank you for your thoughtful feedback. Regarding your inquiry, we would like to provide the following clarification:

Q1: Comparing with GAN-based GLC

A1: We highlight our advantage over GLC as below:

1. Ultra-low Rate Support: GLC is a very competitive and representative GAN-based compression. However, similar to other GAN-based approaches (e.g., MS-ILLM, HiFiC), it faces challenges at ultra-low bitrates. Although GLC achieved bitrates below 0.01 bpp on datasets like CelebAHQ, such performance requires task-specific retraining for facial images, limiting its generalizability. For general natura images (e.g. Kodak, CLIC), GLC's minimum achievable bitrate is approximately 0.02 bpp, whereas our method remains effective even at an ultra-low bitrate of 0.0026 bpp. As the bitrate decreases further, the effectiveness of feature-only optimization gradually diminishes.
2. Simplified and Stable Optimization: GLC employs GAN and a complex 3-stage training for refining compressed features. In contrast, our approach achieves superior visual quality and competitive objective results currently with simple MSE-optimized latent compressor, and mainly focusing on how diffusion models can effectively enhance these features. From the perspective of feature compression, GAN based GLC, diffusion based PerCO, and our method share similar principle—using a pretrained VQGAN/VAE to perform image compression the latent features. However, the optimization strategies diverge.

In essence, GAN-based and Diffusion-based compression frameworks are two competitve pathes. These two approaches are highly complementary and can mutually reinforce each other. Combining GLC's feature refinement with diffusion recovery could further advance ultra low-bitrate image compression performance.

Q2: Full resolution Memory Cosumption

A2: We fully understand this concern and would like to provide the following clarifications:

1. Test Setting: As mentioned by the reviewer, our current test setting is to align with existing diffusion-based methods such as CDC and DiffEIC for fair comparison.
2. Memory Benchmark: On full-resolution CLIC images (2048×1365):

• while GLC's closed-source nature prevented direct evaluation, its backbone VQGAN's peak memory usage is 11.2GB (w/o compression module of GLC)
• Our method's peak usage is 17.8GB (at the Diffusion's VAE decoder).

This marginal increase in memory usage represents a reasonable tradeoff. Considering our method's unique capability of achieving ultra-low bitrate compression and the potential of diffusion-based frameworks, we believe they hold promise for substantial advancements in both performance and efficiency.

Q3: Full resolution results and Comparing with GLC about FID

A3: As mentioned, our method demonstrates more promising performance at ultra low rates (<0.01bpp). GLC’s paper evaluates only a narrow bitrate range (0.02-0.03bpp), we matched this range for fair comparison on CLIC and DIV2K (full resolution) and MS-COCO 30K (256×256, following GLC’s setup). The results can be seen at https://anonymous.4open.science/api/repo/public-5C7B/file/comparison.png?v=f3d457fd. As can be seen from the figure, our method still provides better or competitive results over GLC for full resolution tests.

Q4: Question about Table 4: Fewer steps but better performance

A4: In Table 4, we provided two baselines for comparison. One is a strategy ANS used in previous works, the other is our ResULIC without the CDM strategy. The results showed that with CDM and our Predicted-$x_0$ training strategy, better performance can be achieved with fewer steps.

More explanation of these two modules:

CDM: CDM accelerates denoising by initiating diffusion from a distribution conditioned on compressed feature means $z_c$ rather than pure noise (Eq.8). This intermediate starting point reduces sampling steps while enhancing fidelity.

Predicted-$x_0$ Strategy: Our sampling process (Algorithm 2 in the Appendix) suffers from error accumulation when deriving $z_0$ from noise predictions. The training strategy predicts $x_0$ to reduce error accumulation from noise prediction, improving accuracy and enabling high-fidelity reconstruction with fewer steps.

Additional experiments are provided in the table to demonstrate reduced decoding delay and improved performance（In "Denoising Step", the numbers represents sampling steps (50，20，4) for <0.05bpp and (20，10，3) for ≥0.05bpp）:

We appreciate your insights and hope this clarifies the trade-offs and unique contributions of our work.