Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior

Rebuttal to Weaknesses

• About "Novelty". To our knowledge, we are the first to explore a single codec with a single bitstream that adapts to different intelligent tasks and human perception. By combining generative priors, we designed SC Loss and TGM to achieve this goal. Besides, we have carefully reviewed the DDCM paper. This work primarily focuses on compression algorithms for perceptual quality and does not include any exploration on intelligent tasks. We will cite this paper and provide distinction and discussion. Finally, we show the pipeline of adapting traditional codecs to different tasks in the first diagram of Figure 1 and discuss it in L30-L35. Although the traditional codec itself is general, the bitstream is task-specific. This is because they often optimize for specific tasks, such as some RoI bit allocation methods, whereas Diff-ICMH does not require any additional design or separate bitstreams.

• About "Motivation". We sincerely apologize for any misunderstanding. We must emphasize that in the paper, we do not make any claim that traditional image compression methods "harm" semantic fidelity; they simply do not focus on semantic fidelity. Nor do we claim that perceptual-oriented methods can inherently better preserve semantic accuracy. In the motivation section, our claim is that two key points for intelligent tasks are semantic fidelity and perceptual realism. The former directly relates to whether tasks can be completed normally, while the latter can cause error accumulation and further semantic distortion due to training-inference mismatch. Traditional signal fidelity-oriented image compression methods only optimize for signal fidelity itself and do not focus on semantic fidelity and perceptual realism. These methods include both traditional codecs and learning-based codecs. Perceptual-oriented methods can achieve better performance compared to pure signal fidelity-oriented image compression methods because they better ensure perceptual realism. Diff-ICMH aims to simultaneously address both semantic fidelity and perceptual realism mentioned above. We will further emphasize this statement in the paper.

• About "Clarity Issues and Structural Organization".

  • "Necessity of Figure 1". We must emphasize that Figure 1 is a very important illustration. As we introduce in L30-L40, previous codecs for intelligent tasks either require multiple codecs to achieve this purpose, or different intelligent tasks need different bitstreams. Therefore, the original starting point of this work, as shown in the last diagram of Figure 1, is to use a unified codec with a generalized bitstream to adapt to different intelligent tasks as well as human vision.
  • "generative process follows the standard diffusion process?" Yes. We will describe this more clearly in the paper.
  • "not explicitly stated what information is transmitted". As illustrated in Figure 4, the transmitted bitstream consists of the compressed latent space representation and tag bits. We will accurately describe this linguistically in Section 4.1.
  • "Figure 7 lacks organization." Thank you for your suggestion. We will reorganize the arrangement of these figures to present the results more clearly.
  • "where the SC loss is conducted". Thank you very much for your careful review. The semantic space refers to the feature space of the diffusion model, as illustrated in Figure 5. We will revise the wording for greater clarity in the paper.
  • "Ablations in Section 5.4". Regarding the experimental setup for ablation studies. As stated in L262, the experimental details for the ablation study correspond to section A.2 in the appendix: "The experimental results were rigorously evaluated on the COCO 2017 object detection benchmark dataset." Regarding the results, as illustrated in Figure 8 (b), results of the middle block achieves the best RD performance (in the Rate-Distortion performance graph, closer to the upper left indicates better performance), which demonstrates our claim.

• "Did the authors train their own PerCo version, or used the open sourced version?" We adopted Perco(SD): https://github.com/Nikolai10/PerCo. We will cite and acknowledge it.

• "The DDIM step count differs between methods". Experiments with steps=20 of Diff-ICMH and PerCo are shown below. It can be observed that our method also significantly outperforms PerCo with the same denoising steps. | Method | bpp | mAP | |-----------|--------|-------| | Diff-ICMH | 0.1092 | 34.68 | | | 0.0350 | 27.88 | | PerCo | 0.1274 | 33.53 |
  | | 0.0332 | 21.50 |

• "The ablations are lacking with regards to the necessity of TGM". We sincerely apologize for any misunderstanding. We must emphasize that we did not omit the ablation study for TGM. Figure 9 and L263-L272 correspond to the ablation study experimental results and analysis for TGM and SC Loss. We will change "tag guidance" to "TGM" in Figure 9 to avoid similar misunderstandings.

• "Visualizations"

  • "A more thorough presentation in the main paper ... along with additional examples in the appendix". Excellent suggestion. We will reorganize to include key perceptual results in the main paper and expand the Appendix with additional examples.
  • "perceptual comparisons (Fig 12) do not compare to SOTA diffusion based methods such as DiffEIC, PerCo or DDCM". Thanks for your suggestion. We will add comparison results with those SOTA diffusion based methods.
  • "in the first image the proposed method's BPP is one of the highest in the figure". Thank you for your insightful feedback. We will adjust the bitrates in our comparisons for improved clarity.
  • "zoomed region in Figure 12 ... additional row". We appreciate this constructive suggestion. We will add additional rows of zoomed-in images for each comparison group to enable more precise visual analysis.

• "data reuse". Yes, the experimental results in Figure 7 are consistent with DiffEIC. We will explicitly acknowledge and cite it.

• "lacks discussion on trained, partially trained, and zero-shot compression methods". We seek clarification on the comment about "trained, partially trained, and zero-shot compression methods." Standard neural codec evaluation requires fully trained models for meaningful comparison. Partially trained or zero-shot approaches would significantly degrade performance and provide uninformative comparisons.

• "speed comparisons across different hardware settings". Yes, the evaluation is conducted across different hardware configurations. However, we emphasize that the primary focus of this paper is to demonstrate the generalizability and significant potential of Diff-ICMH for diverse intelligent tasks and human perceptual quality assessment. Therefore, computational complexity analysis was not the central emphasis of our current work.

• "requires training for every new desired bit-rate." Yes, each rate point currently requires a separate trained model. However, most methods not specially designed for variable-rate compression also use one model per bit-rate, which is the standard validation approach for novel neural compression techniques. Our primary contribution is demonstrating Diff-ICMH's generalizability across diverse intelligent tasks and human perceptual quality assessment. While codecs equipped with variable-rate adjustment can achieve multiple rates with a single model, such engineering implementations are beyond our current research scope.

Answers to Questions

• "details of explicit numbers of tag conditions". Thanks for your constructive suggestion. We are using the RAM++ version from Recognize Anything (https://github.com/xinyu1205/recognize-anything). Its maximum default vocabulary is 4585, and we do not perform any entropy coding but use fixed-length encoding, so each ID requires 13 bits for compression (max as 8192). Additionally, we counted that each image on COCO predicts 8.7 tags on average under RAM++'s default setting, so the final average bit count is 13×8.7=113.1 bits. We will supplement these specific details in the article and supplementary materials. Thank you again for your suggestion.
• "resolution mismatch". Thank you for this insightful observation. You're correct that stable diffusion models trained on fixed resolutions can suffer from training-inference mismatch when generating different resolutions, often leading to quality degradation and structural collapse. However, our approach incorporates ControlNet, which provides additional control signals that effectively address these structural issues. This enables robust adaptation across different resolutions. As demonstrated in the original ControlNet paper, the method successfully handles various aspect ratios and resolutions while using Stable Diffusion 1.5 (pretrained at 512×512) as the base model.
• "why optimizing similarity in this latent space helps enforce actual similarity". Essentially, the approach can be viewed as compressesing a compact representation that filters out irrelevant textural details. Specifically, Stable Diffusion's 4-channel VAE, optimized under MSE+LPIPS+GAN loss, functions as a perceptual-oriented lossy compression process rather than simple dimensionality reduction. As discussed in the appendix (L131-L136), VAE latent space provides a compact, perceptually meaningful representation that filters semantically irrelevant information. Comparing to compressing the original pixel-level signal, compressing VAE's latent variables allows the compressed bitstream to prioritize semantically salient information and avoid wasting bits on those hard-to-code and complex textures, achieving superior rate-distortion performance across downstream machine vision applications. The results in Appendix Section B.5 and Figure 14 also confirm our statement. Besides, to address potential convergence issues from extreme values of VAE latent variables, we compress the latents after normalization. We will add this implementation detail to the Appendix.

Thank you for your valuable feedback. Here are our responses to your comments.

---

1. Definition of "intelligent task"

We sincerely apologize for any misunderstanding caused by our unclear expression. "Intelligent task" [1] here specifically refers to neural network-based image understanding tasks, also known as "machine vision," [2,3] including object detection, instance segmentation, and pose estimation.

2. Distinction of codecs designed for use with intelligent tasks

The core differences in Figure 1 lie in the generalization capability of the codecs and the adaptability of the bitstreams to diverse downstream intelligent tasks and human visual perception needs. In subfigure (a), traditional codec-based methods employ a unified encoder-decoder for different tasks but require task-specific bitstreams due to targeted bit allocation or quantization for specific task. In subfigure (b), end-to-end neural codec methods require separate optimization of encoder-decoders for different intelligent tasks, necessitating task-specific codecs and bitstreams. In subfigure (c), feature compression-based methods require training separate codecs for compressing features of different tasks, making both the codec and bitstream task-dependent. In subfigure (d), representing our proposed Diff-ICMH, one unified codec and the corresponding bitstream are needed to meet different downstream requirements, encompassing both various intelligent tasks and human visual perception.

3. Results of ablation study

Yes, the mAP in the ablation study was conducted on the object detection task. Thank you for your suggestion. We provide additional ablation study results for Instance Segmentation and Pose Estimation here. Results will also be incorporated into the Appendix.

Instance Segmentation:

Pose Estimation:

4. Discussion with zero-shot compression methods

We apologize for the ambiguity in our earlier response. The "trained" methods we referenced specifically refer to neural codecs based on VAE architectures that are trained from scratch. We acknowledge that DDCM is indeed a solid training-free and zero-shot compression method based on diffusion models. However, the focus in this paper is on the generalization capability of codecs and bitstreams—namely, whether a single established codec and bitstream can adapt to different intelligent tasks and human perceptual needs without task-specific modifications. Under this scenario, DDCM, Diff-ICMH, and PerCo all represent generalized methods that do not require adaptive optimization for specific tasks (such as detection and segmentation), while methods like TransTIC [2] and Adapter-ICMH [3] require task-specific adaptive training to obtain specialized codecs and bitstreams. We will cite DDCM and provide clearer exposition of this categorization in the revised paper.

5. Complexity

Thank you for your valuable suggestion. The computational complexity of different methods is illustrated below. All experiments are conducted on the Kodak dataset using an A100 GPU with NFE=20 for PerCo, DiffEIC, and Diff-ICMH. Timing is measured in seconds.

[1] Feng R, Jin X, Guo Z, et al. Image coding for machines with omnipotent feature learning[C]. In ECCV, 2022.

[2] Chen Y H, Weng Y C, Kao C H, et al. Transtic: Transferring transformer-based image compression from human perception to machine perception[C]. In ICCV, 2023.

[3] Li H, Li S, Ding S, et al. Image compression for machine and human vision with spatial-frequency adaptation[C]. In ECCV, 2024.

---

Thank you for your valuable comments to our paper. Your constructive suggestions are greatly appreciated.

Rebuttal to Weaknesses

1. "Most of the modules in the framework (e.g., autoencoder, tag extractor and stable diffusion) are pretrained". We sincerely apologize for any confusion. As shown in Figure 4 and described in L199-L200, the tag extractor and stable diffusion model are indeed pre-trained and frozen, but the autoencoder and control module are completely trained from scratch.
2. "Poor inference efficiency". Yes, you are absolutely right, and this is a very important question. We have also provided corresponding experimental results and related analysis in the Appendix Section B.4. The slow decoding speed is mainly caused by the iterative denoising process. To more clearly address this concern, we will add a dedicated Limitations section in the main text to elaborate on this problem. However, it is worth noting that the core motivation of this work is to demonstrate the significant value of Diff-ICMH in unifying human perception and machine vision. The earliest autoregressive-based codecs [1,2] also required tens of seconds to decode a single image, but later, with the development of better entropy models, decoding speed improved from tens of seconds to tens of milliseconds. In recent years, inference speed optimization algorithms for diffusion models have also been rapidly developing, with numerous model distillation [3] and inference step optimization [4,5] methods emerging continuously. We believe that in the near future, combining diffusion models for image encoding and decoding will no longer be constrained by complexity and lengthy encoding/decoding times.

Answers to questions

1. "More ablation studies evaluated on FID and LPIPS metrics". Thank you for your constructive suggestions. We conducted ablation studies of LPIPS and FID respectively on CLIC2020 test set to explore the impact of our proposed SC loss and TGM (tag guidance module) on subjective quality metrics. The results are shown in the table below. It can be observed that SC loss and TGM have minimal impact on LPIPS. We speculate this is because LPIPS primarily focuses on perceptual accuracy of reconstructed images, particularly low-level features, while SC loss and TGM mainly optimizes semantic consistency, which is not directly correlated with the LPIPS metric. On the other hand, SC loss and TGM significantly help reduce FID, as they implicitly and explicitly enhance semantic information in the bitstream respectively, thereby better activating the pretrained generative priors. | Method | bpp | LPIPS | FID | |--------|-----|-------|-----| | w/o SC loss, w/o TGM | 0.1073 | 0.1013 | 7.439 | | | 0.0319 | 0.1979 | 14.81 | | w/ SC loss, w/o TGM | 0.1069 | 0.0967 | 7.293 | | | 0.0322 | 0.1949 | 13.95 | | w/o SC loss, w/ TGM | 0.1071 | 0.1007 | 6.947 | | | 0.0315 | 0.1986 | 13.31 | | w/ SC loss, w/ TGM | 0.1066 | 0.0961 | 6.821 | | | 0.0318 | 0.1966 | 12.73 |
2. "Why do you adopt the middle feature in the diffusion model to calculate the SC Loss ?" Thank you for your constructive feedback. This is an excellent question. We initially also believed that shallower representation spaces might preserve more comprehensive information, which would be more beneficial for model training and supporting subsequent intelligent analysis. However, as shown in Figure 8(b), we found that computing the SC loss only at the middle features yields the best results, which we analyzed in L276-L278. Overall, this is likely because deeper features, i.e., middle block features, can capture higher-level semantic information while filtering out some noisy low-level perceptual information. Computing the SC loss in this representation space is more conducive to preserving high-level semantics while filtering out irrelevant information that should not be compressed and transmitted, thereby achieving better rate-distortion performance.

​[1] Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 7939-7948.

[2] Minnen D, Ballé J, Toderici G D. Joint autoregressive and hierarchical priors for learned image compression[J]. Advances in neural information processing systems, 2018, 31.

[3] Sauer A, Lorenz D, Blattmann A, et al. Adversarial diffusion distillation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 87-103.

[4] Sauer A, Boesel F, Dockhorn T, et al. Fast high-resolution image synthesis with latent adversarial diffusion distillation[C]//SIGGRAPH Asia 2024 Conference Papers. 2024: 1-11.

[5] Meng C, Rombach R, Gao R, et al. On distillation of guided diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 14297-14306.

Rebuttal to Weaknesses and Questions

1. "lack of discussion of the limitations of the proposed approach". Thank you very much for your feedback. Indeed, we have included experimental results on computational complexity and a discussion of limitations in Section B.4 of the Appendix. To enhance clarity, we will reorganize this content into a dedicated section in the main text to more clearly present the limitations of our approach.
2. "Concerns of Task-Independent Support". We sincerely apologize for any misunderstanding. We must emphasize that the entire training process does not involve any explicit or implicit optimization targeting any downstream task. Our model is optimized solely on the LSDIR dataset using diffusion-based ε-prediction loss, rate optimization, and our proposed semantic consistency loss, then directly tested on different downstream intelligent tasks. Therefore, Diff-ICMH is a general and generalizable codec that can zero-shot support different intelligent tasks once training is completed.
3. "Tag Guidance Module (TGM) is simple". Thank you very much for your suggestion. This is indeed a highly valuable question for discussion. In fact, the design of TGM follows a simple yet efficient philosophy, grounded in two core design principles: First, leveraging the strong generative prior of T2I models. As is well known, T2I models are inherently designed to generate images conditioned on text. Therefore, choosing text as the conditioning modality is a natural and reasonable choice. Methods such as CLIP image representations or learned prompt optimization approaches would deviate from the original input format of T2I models, potentially hindering the effective activation of the model's prior knowledge. We have experimented with using the CLIP grid image features of the image as the condition instead of tags. However, we found that even with CLIP features free from any compression distortion, their performance was still inferior to that of tags. As a result, we abandoned this approach. Second, achieving negligible bitrate overhead. Another core design principle of TGM is encoding/decoding tags, which requires extremely minimal bitrate to compress the corresponding tags as stimulation factors for activating the generative prior. As described in the paper, each image requires approximately only 100 bits. Specifically, we utilize the RAM++ version of Recognize Anything (https://github.com/xinyu1205/recognize-anything), which has a maximum default vocabulary of 4,585 tags. We employ fixed-length encoding to compress tag IDs, requiring 13 bits per ID (with a maximum capacity of 8,192). Our statistics on COCO show that each image predicts an average of 8.7 tags under RAM++'s default settings, resulting in an average bitrate of 13×8.7=113.1 bits. In contrast, compressing text would require significantly more bits, not to mention CLIP-based representations or learned prompt optimization methods, which typically involve high-dimensional floating-point continuous representations (e.g., 2,048, 3,072, or 4,096 channels). However, the CLIP-based conditioning or learned prompt optimization methods you mentioned are indeed promising directions for future research, though they would necessarily require targeted designs to address the high bitrate requirements of such representations and how to well stimulate the generative prior, which exceeds the scope of this paper. The core motivation of our work is to demonstrate that diffusion-based perceptual coding methods can effectively unify human perception and machine vision tasks. For this motivation, our designed TGM serves as a simple and practical module.
4. "Concerns of Generalizability". First, this work primarily validates the effectiveness of our method through common vision benchmarks, which is consistent with how other methods (Omni-ICM[1], TransTIC[2], Adapt-ICMH[3]) conduct their testing and validation. To our knowledge, we are the first to encompass such a diverse range of vision tasks and datasets, totaling 10 intelligent tasks. Other domains such as medical imaging, real-time video comprehension, and non-visual language settings are beyond the scope of this paper, though they could be explored in future work. Additionally, we have also tested on out-of-distribution (OOD) datasets, specifically RefCOCO, corresponding to the seventh subplot in Figure 6 and the related discussion at L241, and demonstrated the generalization ability of our method on OOD data.

[1] Feng R, Jin X, Guo Z, et al. Image coding for machines with omnipotent feature learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 510-528.

[2] Chen Y H, Weng Y C, Kao C H, et al. Transtic: Transferring transformer-based image compression from human perception to machine perception[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23297-23307.

[3] Li H, Li S, Ding S, et al. Image compression for machine and human vision with spatial-frequency adaptation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 382-399.

Rebuttal to Mixed

"It would be helpful if the authors elaborated on their choice of similarity metric for enforcing semantic consistency—e.g., whether they used cosine similarity, Euclidean distance, or another measure—and discussed its impact on performance." Thanks for your constructive suggestion. Below are the ablative experiments of the choice of distortion metric to calculate SC loss, including cosine similarity and Euclidean distance. Experiments are conducted on MS COCO object detection. We follow the default training setting for both metrics. It can be observed that the results of cosine similarity achieve a better rate-distortion trade-off. We conjecture that this is because Euclidean distance is sensitive to both the direction and magnitude of vectors, making it susceptible to being dominated by large-magnitude dimensions in the features, which introduces bias in loss computation and hinders overall feature alignment. In contrast, cosine similarity eliminates the influence of vector magnitude through normalization and focuses solely on the directional relationship between vectors, thereby better capturing structural similarities between features and achieving more balanced cross-dimensional feature alignment. We will add the experimental results and corresponding analysis to the Appendix.

Rebuttal to Cons

1. About "The empirical performance of the proposed method appears limited". First, a major motivation of this paper is to explore a codec that can uniformly adapt to diverse machine vision tasks and human visual perception requirements. Moreover, we must emphasize that Figure 6 includes ten different tasks in total. Our method only performs comparably to the second-best approach on the third and seventh tasks, while demonstrating significant advantages across all other tasks.
2. About "Figure 7 indicates no significant improvement in perceptual quality". First, the core objective of this paper is to develop a general codec that unifies human vision and machine vision. Therefore, the inherent goal is to achieve state-of-the-art or comparable performance simultaneously in both perceptual quality and intelligent tasks. Second, our method achieves superior perceptual quality at lower bitrates compared to other approaches, such as DiffEIC.
3. About relationship to "Matsubara et al., WACV 2022" and "comparisons with task-specific or multitask compression approaches". First, regarding the relationship between the two papers, the WACV paper performs adaptive optimization for multiple downstream tasks to achieve better rate-distortion performance on those specific tasks, while our work does not perform any task-specific adaptive optimization and additionally supports human visual perception, making it a generalized codec. We will incorporate the citation of this work in the main text and provide a clear clarification. Furthermore, regarding comparisons with task-specific optimized codecs, we emphasize that our evaluation includes TransTIC and Adapt-ICMH—task-specific methods specifically designed and optimized for particular tasks, as demonstrated in Figure 6 and the corresponding analysis. Notably, Diff-ICMH achieves significantly superior performance compared to these specialized approaches while requiring no task-specific adaptive optimization. This demonstrates not only the exceptional performance of our method but also its superior generalization capability and practical ease of deployment.

Feedback to Limitations

We greatly appreciate your valuable suggestions. We will add a dedicated Limitations section to comprehensively address these concerns, including the trade-offs you mentioned regarding hallucinated details and applications in medical imaging and scientific analysis.