Unpaired Image-to-Image Translation via Neural Schrödinger Bridge

We thank the Reviewer for the insightful review, which has been incorporated into the revised paper. We address the Reviewer's concern below.

Q1 : It could be valuable to include the reverse translations (such as Zebra to Horse, Winter to Summer) in the supplementary material, providing more comprehensive experimental results.

We kindly refer the Reviewer to Appendix C.5, where we provided an additional result on Zebra2Horse translation, and observed UNSB beats CycleGAN. We plan to include more results in the future.

The response adequately addressed my concerns, and I am inclined to keep my rating as 6.

We thank the Reviewer for the insightful review, which has been incorporated into the revised paper. We address the Reviewer's concerns and questions below.

Q1 : I think that some relevant baselines were not included in the experimental section. Could the author compare the proposed method to EGSDE [1] (Diffusion-based model)?

We kindly refer the Reviewer to Appendix C.3, where we compared UNSB with EGSDE, StarGAN v2, and NOT on the Male2Female translation task, and observed UNSB beats all baselines.

Q2 : The authors attribute the failure of SB for unpaired image-to-image translation to the curse of dimensionality. However, methods such as EGSDE and SDEdit do achieve successful translations in translating high resolution images. Could the authors elaborate on the difference between the cases?

We kindly refer the Reviewer to Appendix D, where we added a discussion on the Reviewer’s question.

Q3 : Following my previous point, the datasets considered in the paper are all of “scene-level" image to image translation (in which mostly color and texture need to be altered during translation). In most of other I2I translation works, many "object-level" datasets are evaluated e.g. CelebA-HQ Male-to-Female, Cat-to-dog, etc. Do the authors expect different performance for the latter case? I believe it would be helpful to include benchmarks for at least some of these additional datasets to understand the strengths and the limitations of the proposed method.

We kindly refer the Reviewer to Appendix C.3, where we performed an additional experiment on Male2Female, and observed good results for UNSB.

Q4 : The description of the Stochasticity analysis (Fig. 7) is not clear to me. Could the author describe this ablation experiment and what does the generated map tell us? I understand that high pixel-wise std implies greater diversity, but why is it surprising that the std is greater for the foreground pixels? What is the scale of the std?

The following discussion has been added to Appendix C.2.

Fig. 7 serves as evidence that UNSB indeed solves the SB problem. Specifically, a SB between two domains is stochastic, so it does one-to-many translation. Thus, a necessary condition for a model to be a SB is stochasticity. Fig. 7 shows that UNSB satisfies this necessary condition.

Greater std for foreground pixels tells us UNSB does one-to-many generation, and generated images lie in the target domain. For instance, a model which simply adds Gaussian noise also does one-to-many generation, but is not meaningful. UNSB output shows high variation for locations which are relevant to the target domain (Zebra).

Minimum and maximum values of pixel standard deviation in Figure 7 are 0.0031 and 0.7940, respectively. We note that all pixel values are normalized into [0,1].

We thank the Reviewer for the insightful review, which has been incorporated into the revised paper. We address the Reviewer's concerns and questions below.

Q1 : I have some doubts regarding the statement (end of page 1) that “no work has successfully trained SBs for direct translation between high-res images” and (end of page 2) “our work represents the first endeavor on this problem”. For example, [1], [2] deals with 64x64 images and tackles unpaired image-to-image problems. I am not sure, if the methods from [1], [2] works with 256x256, but, anyway, [1] and [2] should be covered as the related works and the competitive methods.

Thank you for informing us about [1] and [2]. We added a discussion regarding [1] and [2] in Section 2. We also attempted to implement [1,2] on our experiments, but faced scalability issues. Concretely, as noted in [1], [1] takes about 1 day to learn a SB between 28x28 grayscale images, with 2 GPUs. As noted in [2], [2] takes 7 days to learn a SB between 64x64 color images, with 2 GPUs. Compare this to UNSB: for instance, on Summer2Winter, UNSB takes 20 hours with 1 GPU to learn a SB between 256 x 256 color images.

Q2 : Regarding the competitive methods (GANs), the authors miss the StarGAN and StarGAN v2 [3]. The latter model demonstrates a good quality on the unpaired image2image and I think this model is a good baseline method. Please, add it to the comparison.

We kindly refer the Reviewer to Appendix C.3, where we compared UNSB with StarGAN v2, EGSDE, and NOT on the Male2Female translation task, and observed UNSB beats all baselines.

Q3 : I ask the authors to add the clarifications how to estimate the entropy with samples, a technique introduced at the beginning of page 6. It will facilitate the comprehension and smoothness of the reading.

Thank you for the constructive feedback. We provided more detail on entropy calculation in Appendix B of the revised paper.

Q4 : I have one concern about objective (14). The transition from constrained problem (9)-(10) to the combination of losses (14) is non-trivial and not fully theoretically justified. I am not sure, that the objective (14) indeed solves the SB problem. Some comments/clarifications will contribute to the clearness of the manuscript flow. I strongly encourage the authors to evaluate how does their approach solve SB problem on a nontrivial setup beyond gaussian2gaussian case. In particular, there is a recent paper [4] which proposes benchmark pairs of distributions with known GT EOT/SB solutions, even for images data. The validation on such a benchmark will highly contribute to the quality of the paper and will probably answer this question.

Thanks for the important comment. The transition from constrained problem to an unconstrained formulation arises from using the Langragian and dual formulation. Furthermore, we would like to refer the Reviewer to Appendix C.4, where we added a result on the suggested benchmark [4], and observed positive results.

Q5 : I have one more question regarding the baseline methods. The authors choose the work [5] (called them as NOT) “as the representative of OT [methods]”. This choice really surprised me because [5] also deals with SB problem, and, as I understand, their method is not designed for image data (at least, the authors of [5] does not consider image data use cases). That is why comparing with [5] seems strange. I recommend authors take [6, 7, 8] as the representative of OT methods instead. These methods also has a possibility to generate the samples of controllable diversity, have GAN-resembling objective. Moreover, they are designed specifically for unpaired image2image problems.

Thanks for pointing out the typo. We attached a wrong reference to NOT in our paper. NOT in our paper refers to [6] mentioned by the Reviewer, and all experiment results for NOT are carried out with [6]. We apologize for the mistake, and we fixed this reference in our revised paper.

Dear Reviewer f3t2,

As the deadline for the Reviewer-Author discussion phase is fast approaching (there is only a day left), we respectfully ask whether we have addressed your questions and concerns adequately.

Sincerely,

The Authors.

Dear reviewer f3t2,

We would like to gently remind you that the discussion period ends in approximately 10 hours. We would appreciate it if you could let us know whether our comments addressed your concerns, as summarized below.

• In Section 2 we covered [1] and [2] as related works and competitive methods in the revised paper.

• In Section 5 we fixed a wrong reference for Neural Optimal Transport (NOT) [3], and clarified [3] is used as a representative for OT in all our experiments.

• In Appendix C.3 we conducted an additional experiment on Male2Female translation with CelebA-HQ-256 images with baselines EGSDE, StarGAN v2, and NOT, and observed UNSB beats the baselines.

• In Appendix B we clarified how to approximate entropy in detail.

• In Appendix C.4 we validated UNSB on the suggested benchmark [4], and observed positive results.

• In Appendix C.7 we validated UNSB on Cat2Dog with AFHQ $512 \times 512$, which further demonstrates the scalability of our proposed approach.

Best regards, Authors

[1] Diffusion Schrödinger Bridge Matching, NeurIPS’2023

[2] Entropic Neural Optimal Transport via Diffusion Processes, NeurIPS’2023

[3] Neural Optimal Transport, ICLR’2023

[4] Building the Bridge of Schrödinger: A Continuous Entropic Optimal Transport Benchmark, NeurIPS’2023

We thank the Reviewer for the insightful review, which has been incorporated into the revised paper. We address the Reviewer's concerns and questions below.

Q1 : The paper claimed that "none of SB models so far have been successful at unpaired translation between high resolution images", however, the experiments in this paper do not include high resolution images yet (256x256 cannot be regarded as high resolution). How will the proposed method perform on this setting is still unclear.

We kindly remind the Reviewer that multiple works [1,2,3] view 256x256 as high-resolution. Moreover, numerous representative methods for unpaired image-to-image translation [4,5] use 256x256 resolution images. Although we believe that our method can be scaled to higher resolution such as 512x512 or 1024x1024, and are running the experiments now, due to the limiting computational resource, it is a pity that we cannot provide the results during the rebuttal period.

[1] High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al., CVPR, 2022.

[2] EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations, Zhao et al., NeurIPS, 2022.

[3] Dual Diffusion Implicit Bridges for Image-to-Image translation, Su et al., ICLR, 2023.

[4] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, Zhu et al., ICCV, 2017.

[5] Contrastive Learning for Unpaired Image-to-Image Translation, Park et al., ECCV, 2020.

Q2 : It is unclear that the findings by ablation study are still valid for other datasets?

We would like to refer the Reviewer to Appendix C.6, where we performed an additional ablation study on the Summer2Winter translation task, and observe similar trends as before.

Q3 : When presenting the stochasticity analysis, it only gives results for one image. It would be better if the paper can provide statistical numbers across many different images/cases.

We kindly refer the Reviewer to Appendix C.2, where we provide more stochasticity analysis results.

Dear Reviewer ek8v,

As the deadline for Reviewer-Author discussion is fast approaching (there is only a day left), we respectfully ask whether we have addressed your questions and concerns adequately.

Sincerely,

The Authors.