We thank the reviewer for recognizing and acknowledging the strengths of the proposed approach. We will address the raised concerns below:

Response to the Weaknesses

1. Comparisons to TinyLIC and other SOTA baselines.

[Reply] We thank the reviewer for the valuable suggestion. It is indeed necessary to provide a more comprehensive evaluation by comparing TinyLIC as a baseline and comparing the complexity with other SOTA methods. To address this, we have added the relevant comparisons, as shown in Figure 1 and Table 3 of the global response attachment. TinyLIC baseline is a variable-rate version optimized for MSE, while the perception-optimized version is equivalent to MPA (α=0). Since CRDR is also a modification based on MRIC, we include MRIC as a baseline for complexity comparison. As shown, our implementation has lower parameter and computational complexity compared to MRIC, while maintaining similar latency. This similarity in latency is because MRIC, being composed solely of convolutions, is more GPU-friendly than TinyLIC, which consists of transformer structures that are not yet well optimized for GPU. We promise that these modifications will be presented in the final version.

2. Effectiveness of the predictor.

[Reply] We understand the reviewer's concerns regarding the effectiveness of the predictor. In fact, Table 3 in our submission only evaluates the encoder, primarily to verify the enhancement of coding efficiency by MPA. However, this table shows only a small part of the effectiveness of the predictor, and the predictor's role in the decoder is even more important which cannot be overlooked, as it enables the base model to achieve coding for multi-task. Our experiments in the submission have already demonstrated its effectiveness. As shown in Figure 3 of our submission, our approach achieves comparable performance to methods like MRIC and CRDR, which are optimized solely for distortion and realism, and matches the vision task performance of fully fine-tuned models, all within a unified model. The key to achieving this performance is that the predictor decouples features across different tasks by predicting their importance for each task, thereby making side paths always focus on the features that are most important for task optimization at any ratio, and easing the complexity of optimizing coding for multi-task. Figures 4, 5, and Appx. E.2 in the submission show that the predictors assign different importance scores for different tasks, demonstrating that the predictor enables the additional side paths to capture the most critical features for each task. Related explanations have been given in Section 5.3 of the submission.

Response to the Questions

1. Effectiveness of the predictor.

[Reply] Yes. To address the concerns and the question, we have explained the effectiveness of the predictor module in our response to Weakness #2.

2. Applying to other base models.

[Reply] Yes. In recent years, similar MetaFormer structures [1] (including Swin Transformer [2] and ConvNeXt [3] which have channel MLPs) have been widely applied as general backbones in the latest learned image coding research [4-8], demonstrating their effectiveness and generalization. Our MPA is an innovation based on the channel MLPs within such MetaFormer structures, enabling learned image coding models to support coding for multi-task in a natural way. Therefore, the proposed MPA can be applied to any model using a MetaFormer backbone.

Response to the Limitations

1. Not all-in-one.

[Reply] We sincerely thank the reviewer for the insightful feedback. We respectfully acknowledge that our method needs to fine-tune some of the parameters as the reviewer mentioned which is indeed a common practice to optimize coding for multi-task. We would like to clarify that our understanding of an all-in-one model is in the context of practical applications. As the reviewer mentioned, even a unified model still requires fine-tuning for specific tasks. But during the actual deployment phase, it is sufficient for the optimized unified model to handle all required tasks using a single encoder and decoder pair. Our MPA can achieve this by performing one encoding and user-controllable decoding, allowing a unified model to reconstruct an image for the target we want. This is what we mean by "all-in-one" coding. We appreciate the reviewer's understanding, and if the reviewer still finds this term unsuitable, we are open to adjusting the wording in the final version.

References

[1] W. Yu, et al. MetaFormer Is Actually What You Need for Vision. In CVPR 2022.

[2] Z. Liu, et al. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. In ICCV 2021.

[3] Z. Liu, et al. A ConvNet for the 2020s. In CVPR 2022.

[4] Y. Zhu, et al. Transformer-based Transform Coding. In ICLR 2022.

[5] R. Zou, et al. The Devil Is in the Details: Window-based Attention for Image Compression. In CVPR 2022.

[6] J, Liu, et al. Learned Image Compression With Mixed Transformer-CNN Architectures. In CVPR 2023.

[7] Z. Duan, et al. Lossy Image Compression With Quantized Hierarchical VAEs. In WACV 2023.

[8] H. Li, et al. Frequency-Aware Transformer for Learned Image Compression. In ICLR 2024.

We would like to express our sincere gratitude to you for the considerate feedback. We appreciate your recognition of the clarity and quality of our paper, particularly in articulating the problem, proposed method, and significance of our contributions.

We want to affirm that your understanding of the advantages of our proposed method aligns well with the insights provided by other reviewers. Your perspective is valuable and contributes to a well-rounded evaluation of our work.

To further assist you in understanding the contributions and significance of our work, we encourage you to review our responses to the other reviewers' comments. These responses provide additional context and detailed explanations that might help in appreciating the strengths and innovations of our approach.

Thank you again for your supportive comments and for taking the time to review our work. We hope this information enhances your confidence in your initial assessment.

First of all, we really appreciate the reviewer's careful comments. We offer the following response to the reviewer's concerns:

Response to the Weaknesses

1. Comparison to SpotTune.

[Reply] We appreciate the reviewer's attention to the differences between MPA and SpotTune. It is essential to emphasize that feature decoupling is central to MPA's effectiveness, an aspect that should not be overlooked, especially crucial for coding for multi-task scenarios. By considering feature decoupling, MPA supports smooth transitions between tasks, allowing for controllable and flexible reconstruction orientations, catering to diverse coding applications. In the encoding phase, feature decoupling enhances compression efficiency. In the decoding phase, it enables controllable reconstruction, which is vital for task-controllable image coding. In contrast, SpotTune only considers the impact of fine-tuning across different layers, applying indiscriminate routing to all features, which restricts its utility in task-controllable image coding.

2. Complexity of training strategy.

[Reply] We thank the reviewer for the concerns regarding the generalization and usability of MPA. In fact, our proposed strategy originates from the widely-used multi-stage training strategy for generative image compression [2], which has been broadly validated for its stability and generalization, while not being overly complex. Here, we provide a further explanation. First, the initial stage involves training a general basic model. In this stage, the losses and hyperparameters we use strictly follow MRIC, with the only difference being that we fix MRIC's β at 2.56 (cf. Eq. (4) in [3]) and add a ratio loss to optimize the predictor in the encoder. Second, the subsequent stage of training only optimizes the side path and predictor in the decoder. When optimizing for MSE, the loss we use is the MRIC loss with β fixed at 0 (cf. Eq. (4) in [3]), plus the ratio loss. For vision task optimization, the loss we use removes the GAN loss from the first stage and adds a vision task-specific loss. Therefore, our proposed method ensures the generalizability and usability.

Response to the Questions

1. Applications for other fundamental tasks and cross-domain tasks.

[Reply] We thank the reviewer for the insightful question regarding the application of MPA to other tasks. As we mentioned in our response to Weakness #2, our method demonstrates the generalizability and can adapt to various application scenarios. Here, we conducted additional tests on MPA's performance in object detection shown in Figure 1 of the global response attachment, using the same object detection model and task loss as TransTIC [4]. Regarding knowledge transfer in cross-domain tasks, our exploration mainly focuses on task-specific optimization and has not yet extended to domain-specific features. This is a promising direction for future research, and we will continue to follow up on this.

2. Hyperparameters and loss terms.

[Reply] We thank the reviewer for the insightful question. Regarding the hyperparameters, we use the same settings as in [3] which have been thoroughly evaluated by the authors (cf. Section 5.2 in [3]). It's hard to reproduce the ablations on hyperparameters in such a limited period of rebuttal phase, so we respectfully suggest you refer to [3] for details. As for the loss terms, we provide detailed ablation studies in Table 1 of the global response attachment to demonstrate that our current combination can achieve a competitive trade-off. Regarding the role of each term, LPIPS enriches the semantic information of the reconstructed images, improving generalization, MSE constrains the pixel value consistency between the reconstructed and original images, and the task loss directly optimizes accuracy. The rate is determined by the encoder and entropy model, which are frozen and do not affect the second-stage optimization. We hope our ablations on the loss terms should help you more directly understand the role of each loss and the potential results of adjusting hyperparameters.

3. Performance gap.

[Reply] There are two main reasons for the observed performance differences. First, the model size limits the performance especially at relatively higher bitrates. As shown in Table 3 of the global response attachment, TinyLIC, the base model we use to implement MPA, is much smaller than MRIC which is the base model of CRDR. Second, as a common sense in the community of image coding, optimizing with a relatively narrower bitrate coverage range can facilitate the coding performance for variable-rate models. As shown in Figure 3 of our submission, the bitrate coverage range of CRDR (0.08~0.72bpp) is much smaller than that of our implementation (0.07~1.20bpp). Thus the performance gap is reasonable and acceptable.

4. Comparison to LoRA.

[Reply] We appreciate the reviewer's suggestion regarding LoRA. Our experiments recognize the effectiveness of LoRA in Table 2 of the global response attachment. Although MPA involves more fine-tuning parameters, it also achieves better performance. Moreover, our work has distinct features. We utilize predictors to support coding for multitask and smooth transitions between tasks within an all-in-one framework. The low-rank structure design of LoRA inspires us to consider improvements for the side path in our future work.

References

[1] Y. Guo, et al. SpotTune: Transfer Learning through Adaptive Fine-tuning. In CVPR 2019.

[2] F. Mentzer, et al. High-Fidelity Generative Image Compression. In NeurIPS 2020.

[3] E. Agustsson, et al. Multi-Realism Image Compression with a Conditional Generator. In CVPR 2023.

[4] Y. Chen, et al. TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception. In ICCV 2023.