Response W3EV

We thank the reviewer for their encouraging comments: “extensive analysis to demonstrate the validity of LPL”, “This level of extensive ablation study is rare.” Below we address the points raised in the review individually.

W1 “Efficiency” and relation to Kang et al., ECCV’24.
Thanks for bringing this paper of Kang et al. $1$ under our attention. We consider it concurrent to our work as the ICLR deadline coincided with the ECCV conference.

We discuss the efficiency aspects and computational overhead in the new Section A.5 of the supplementary material.

• Figure 15 shows that LPL can boost performance after as little as 20k iterations when finetuning.
• Additionally, we show that the gains remain significant even when comparing with a baseline trained with a maxed-out batch size for the same time period.

We discuss the relation with Kang et al in the new Appendix A.4 and compare with their approach experimentally, results are reported in Table 5. We explored training all the timesteps and only earlier timesteps, similar to our work. In both cases they obtain worse performance than the L2 baseline.

Below, we summarize the main difference between our LPL and E-LatentLPIPS:

• E-LatentLPIPS was only tested for distilling diffusion models into GANs and not for training diffusion models.
• E-LatentLPIPS requires training of a classifier latent space for every new autoencoder, while LPL uses the already available autoencoder decoder.
• In terms of interpretation, E-LatentLPIPS does not tackle the mismatch between pixel-space and latent-space diffusion but provides an LPIPS model that can help in latent distillation tasks. As this latent representation is more compressed than the latents, it is unlikely that it will help with high-frequency information similarly to ours.
• When tested for diffusion training, we obtain results that are degraded with respect to the L2 baseline. For more detail we refer to Section A.4 of the supplementary material.

W3 “Novelty”
Our work tackles the mismatch between pixel diffusion and latent diffusion, hence we believe it provides valuable insight about the training dynamics of generative models in latent space, as the latent structure is often an overlooked component in the literature. From our point of view, the novelty comes from identifying the issue of latent-pixel structure mismatch and how it results in suboptimal results.
We provide extensive experimental results and ablations, showcasing its effectiveness in tackling the identified problem as well as improving overall performance on multiple datasets, resolutions, and training paradigms. We believe this is a valuable contribution to the field.
In Section A.2 of the supplementary material we provide evidence highlighting the mismatch issue with the current SOTA latent image generative models. Nevertheless, to further strengthen our contributions, we provide an updated version of Section A.1 in the appendix to showcase that LPL can be interpreted as an image-space penalty on the forward process posterior distribution.
Furthermore, we added a new Section A.4 to the supplementary material with comparisons to other recent works that develop perceptual losses in latent space, showing that just plugging other perceptual losses in the training hurts performance while our approach is more general and improves over the latest SOTA baselines.

Q1 “How does the result compare when using FLOPS as the x-axis?”
We added an extensive analysis on efficiency and computational overhead in the new Section A.6 of the supplementary material.

• Figure 15 shows that LPL can boost performance after as little as 20k iterations when finetuning.
• Additionally, we show in Table 7 that the gains remain significant even when comparing with a baseline trained with a maxed-out batch size for the same time period.

Q2 “Is there a specific reason why LPL was only used in post-training?”
The reason LPL was used in post-training is two-fold:

• Reducing computational overhead.
• We hypothesize that LPL provides a better signal when denoised predictions are more accurate (this is reflected in Section A.1 of the supplementary material where the condition for the Taylor expansion becomes reasonable), which is also the reason why we impose a threshold on the timesteps for which LPL is applied.

In the new Section A.6 of the appendix, we perform the requested experiments.

• When training from scratch, the model using LPL converges faster than the baseline (Figure 14).
• When finetuning with LPL, the adaptation period is of less than 20k iterations (Figure 15).

$1$ Kang et al., Distilling Diffusion Models into Conditional GANs. In ECCV 2024

Thank you for your dedicated effort on the rebuttal. I really appreciate it.

---

There were two main issues I pointed out in my review.

1. Efficiency

How does the result compare when using FLOPS as the x-axis? Comparing by iteration seems somewhat unfair.

The authors responded to this in the rebuttal with two strategies:

1. Even with only 20k iterations in post-training, they already surpass in performance. (Figure 15)
2. They outperform when measured using the same GPU and wall-clock time. (Table 7)

Firstly, it's unfortunate that although many reviewers requested comparisons based on FLOPS, the authors did not provide exact FLOPS-based comparisons but instead used iteration (Figure 15). Unfortunately, when comparing based on FLOPS, LPL seems to perform worse (since LPL requires additional x2 more FLOPS). Given this, the technical contribution appears somewhat reduced. If the authors think that comparing LPL based on FLOPS is particularly unfair or disadvantageous, please let me know. If so, I would like to consider that in my evaluation.

Regarding the Table 7, I wonder if the learning rate in the experiment without LPL was appropriately tuned according to the batch size. If so, it helps to convince that the LPL loss helps improve performance under realistic constraints.

2. Novelty

I believe the novelty of LPL is somewhat lacking.

The authors claim that their paper "provides valuable insight about the training dynamics of generative models in latent space." Indeed, as they assert, the role of latent space in latent diffusion models (LDMs) is a relatively less explored area.

However, I am not sure what effective insights the use of LPL provides about the training dynamics of LDMs. From a reader's perspective, the LPL loss is fundamentally a perceptual loss, as indicated by its name, and it's hard to perceive that it solves a fundamental problem between latent space and pixel space. For instance, as other reviewers besides myself have mentioned, they ask whether there are experiments using latent-LPIPS. The underlying meaning of this question is that, in the current flow of the paper, the motivation for using perceptual loss using the VAE decoder is due to perceptual representation, not because of a latent space–pixel space mismatch. Therefore, if this paper wants to assert novelty, it needs to reorganize the storyline focusing on the mismatch between latent space and pixel space.

Below are my thoughts on the experiments mentioned in the rebuttal.

Section A.1

I think it provides really good justification. Perhaps this theory could be used to support that it prevents mismatch.

Section A.2

I believe this experiment could serve as a good early demonstration in the paper showing the problem with LDMs. It intuitively shows that while the L2 norm may be appropriate in pixel space, it is not in latent space. It would be beneficial to provide this as initial motivation.

Section A.4

Table 5 shows that LPL is promising compared to the baseline. It's a good result.

We provide extensive experimental results and ablations, showcasing its effectiveness in tackling the identified problem as well as improving overall performance on multiple datasets, resolutions, and training paradigms. We believe this is a valuable contribution to the field.

I indeed agree that this paper brings some improvements in performance. Therefore, even if this paper is accepted, I would not object. And I believe that the contribution of performance-oriented papers is very important. However, I feel that this paper relies heavily on existing methods, w/o giving new insights, at least in current form.

---

In conclusion, after the rebuttal, the two issues I had—efficiency and novelty—still seem unresolved. It's unclear whether the proposed LPL shows better performance than the baseline w.r.t. FLOPS, and the novel insight that LPL resolves the mismatch between image space and latent space does not seem appropriately conveyed in its current form. I think it would be better to focus less on perceptual loss and discuss the mismatch more thoroughly.

It's painful to say this after you've put so much effort into the rebuttal. If there's any part you want to refute or if I'm misunderstanding and you need to persuade me, please let me know.

We thank the reviewer for their valuable feedback.

1. Efficiency.

We updated the appendix with FLOPs-based comparisons.

Table 8 shows the total FLOPs during a training iteration of the different phases present in Table 2. We find the FLOP count to increase roughly by a factor 1.5 when training with LPL across all configurations. However, when comparing the complete training runs, i.e. taking into account also the (long) pre-training without LPL, we find that the total FLOP count increases by less than 15% (256 resolution models) or less than 10% (512 resolution models), while achieving better FID (shown in Table 2).

Table 7 now shows the FLOPs/iteration for the training with and w/o LPL, where the batch size has been scaled in both cases to fill the GPU memory. We find that in this case the FLOP count to be actually slightly higher (1.6%) for the baseline model without LPL but with larger batch size.

Regarding the learning rate in Table 7, we did scale the baseline’s learning rate linearly with the batch size (the LPL model uses a learning rate of 10^(-4) while the baseline uses a learning rate of 1.6*10^(-4). We now explicitly state this in the manuscript.

Figures 16,17 and 18 show the performance in terms of FID when adopting FLOPs in the x-axis for the baseline model without LPL and the same model with LPL included. In this case the tendency when training from scratch (Fig 17) is in favour of the baseline which converges faster for similar amounts of FLOPs. For finetuning (Fig 18), however, we can see that LPL helps the model converge faster and achieves better performance, especially when incorporating EMA, we see that the LPL model converges to a lower point than the baseline. We updated Figures 17/18 with a longer runtime for the baseline in order to equalize the FLOP count.

We hope that these clarifications adequately address the reviewer’s concerns.

2. Novelty.

We thank the reviewer for their suggestion. We follow the reviewer’s advice and reorganize the storyline in the main manuscript in order to focus more on the motivation for our loss (L2 blurriness, pixel-latent mismatch, decoder as perceptual network), while deferring certain implementation details to the appendix (such as the outlier detection approach), we additionally include comparisons with other perceptual losses for diffusion in the main manuscript.

We will continue improving the storyline and the flow of the paper accordingly.

We thank the reviewer for their thoughtful feedback.

1 - One straightforward way to look at this is through the development in A.1. There we show that for the image-space divergence penalty to be approximated by our LPL loss, the Taylor expansion rule (l800-809) requires that the predicted denoised latent and the target latent should be similar, such a condition holds better later on during training when the denoiser becomes more accurate.

Broadly speaking, the effect of LPL is a polishing effect on the predicted image, however if the model hasn’t yet learned to generalize well because it didn’t see enough images $2$ then the effect of LPL can be diluted in distributional metrics such as FID.

Another way to see this effect is through the exposure bias problem [3].

Broadly speaking, the effect of LPL is a polishing effect on the later reverse processes $p_t^\Theta, t> \tau_\sigma$.

However, the exposure bias being significantly higher earlier during training ensures that the optimized distribution is practically not seen during sampling (Let $z_1 \sim \mathcal{N}(0, I)$ then $p_\Theta(z_t|z_1)$ is too different from $q(z_t|z_1)$ for LPL to have a meaningful effect early during training).

Similarly, if the denoiser prediction of the latent is of very bad quality, then the gradient from LPL can be very noisy because of the non-linearities of the decoder dealing with out-of-distribution samples, thereby hampering convergence.

Regarding $1$ , the method improves representation learning ability of the denoiser which can help it in better modelling object structures and semantic properties for example, but this does not address the latent-space/pixel-space mismatch tackled by LPL.

Intuitively, it makes sense that such a method works better in pretraining, as it pushes the internal representation of the model towards a local minima induced by the pretrained teacher network, which works as a sort of knowledge distillation that is known to improve convergence properties.

Also it will suffer less from the noisy gradient problem because their method does not require backpropagating through the teacher network.

2 - We thank the reviewer for their remark, we will spend time improving the writing to better frame novelty, emphasize LDMs weaknesses and why this makes our LPL worthwhile.

We thank the reviewer for their dedication in reviewing our work and providing valuable feedback and discussions.

$2$ GENERALIZATION IN DIFFUSION MODELS ARISES FROM GEOMETRY-ADAPTIVE HARMONIC REPRESENTATIONS, Kadkhodaie et. al, ICLR 2024.

$3$ M. Ning et al., Elucidating the Exposure Bias in Diffusion Models, ICLR 2024.

Response UNn8

We are glad the reviewer found our qualitative and quantitative results promising, and our ablations carried out in a systematic way. In the following, we respond to the points raised one by one.

W1+W2+Q3 “paper seems unpolished”, “the notation is a bit confusing”, “better if the authors could stick to the canonical DDPM/EDM $3$ notations”
We updated our notation to be similar to the canonical notation as suggested, and proofread the text in order to improve its coherence and formatting. If the reviewer would like us to further clarify or otherwise improve specific passages of the manuscript we would be happy to do so.

W2 “Applying perceptual loss in T2I generation isn’t novel, see Lin & Yang.”
We thank the reviewer for bringing this reference under our attention. We note that the paper by Lin & Yang $1$ has not been peer-reviewed/published. However we added a discussion of this paper and compare with it experimentally in the new Section A.4 of the supplementary material. While Lin & Yang found their loss to deteriorate performance when using classifier-free guidance for sampling, we have observed consistently improved generations with LPL on different datasets and resolutions. For more detail we refer to Section A.4 of the supplementary material.

Q1 “only a VAE citation is given to explain blurriness, could you add citations…”
Another perspective is provided by Ning et al $1$ , which attributes the smoothness/blurriness effect to exposure bias during sampling (denoiser input mismatch between training and sampling). We added a discussion of this work in the relevant work (L307-310).

Q2 “there is already accepted literature $2$ that train another model which efficiently compute LPIPS losses in the latent space.”
We were referring to the usual LPIPS which employs a VGG-16 or AlexNet backbone trained in image space, such models are not suitable in our case.
We compare LPL with E-LatentLPIPS as a training objective, results are reported in Table 5 of the appendix. We explore two scenarios: (1) training all the timesteps, and (2) training on only earlier timesteps. We observe that both approaches perform worse than the L2 baseline.
Below, we summarize the main difference between our LPL and E-LatentLPIPS:

• E-LatentLPIPS was only tested for distilling diffusion models into GANs and not for training diffusion models.
• E-LatentLPIPS requires training of a classifier latent space for every new autoencoder, while LPL uses the already available autoencoder decoder.
• In terms of interpretation, E-LatentLPIPS does not tackle the mismatch between pixel-space and latent-space diffusion but provides an LPIPS model that can help in latent distillation tasks. As this latent representation is more compressed than the latents, it is unlikely that it will help with high-frequency information similarly to ours.
• It makes more sense that E-LatentLPIPS can help with later timesteps (high noise ratio) by providing semantic structures but this was not verified in our experiments.
• When tested for diffusion training, we obtain results that are degraded with respect to the L2 baseline.

Q3: addressed above

Q4 “do you enforce zero-terminal SNR here? It is known that the SD diffusion schedules are flawed $4$ .”
Yes, our velocity models (DDPM-v and flow) all use a zero terminal SNR state, we now state this in lines 300-302.

$1$ M. Ning et al., Elucidating the Exposure Bias in Diffusion Models, ICLR 2024

$2$ Kang et al., Distilling Diffusion Models into Conditional GANs. In ECCV 2024

Q1 FLOPS and VRAM usage
We have added a new section A.6 to the supplementary material to study the effect of the computational and memory overheads induced by our Latent Perceptual Loss (LPL).
Results in Table 7 show that when maximizing the batch size for the available memory, and training for the same wallclock time of 48h, LPL still achieves improved FID scores despite a reduced throughput. A detailed tracking of FID over training iterations in Fig 15 shows that LPL leads to consistent improvements over the baseline after as little as 20k iterations.

Q2 “Why do you use a CFG scale of 2?“
We follow the standard approaches when choosing the guidance scale, a value 2 of is a defacto standard when comparing ImageNet models at 512 resolution, at 256 resolution we use a value of 1.375 which is the standard (such values give the best FID results for the baseline) $4,5$ , higher values can produce more visually appealing images but at the cost of reduced diversity.

Q3 “Why is the frequency cut off at 360 in Fig. 6?”
For the frequency graph, we plot the errors for different frequency ranges, for images that are 512x512. The frequency range corresponds to circles with radius between 0 and the maximum radius in the image, i.e., the distance from the center to one of the corners, this distance is given by sqrt(256**2 + 256**2) = 362.

**Q4 Suggested discussion of Lin & Yang, arXiv 2023 $6$ **
We thank the reviewer for this suggestion, we discuss this paper and compare with it experimentally in the new Section A.4 of the supplementary material. While $6$ found their loss to deteriorate performance when using classifier-free guidance for sampling, we have observed consistently improved generations with LPL on different dataset, resolutions, and datasets. For more detail we refer to Section A.4 of the supplementary material.

$1$ Self-Perceptual, Lin & Yang 2024, https://arxiv.org/abs/2401.00110.

$2$ DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, Fu et al., NeurIPS 2023. https://openreview.net/forum?id=DEiNSfh1k7

$3$ Efficient Diffusion Training via Min-SNR Weighting Strategy, Hang et al., ECCV 2023.

$4$ Scalable Diffusion Models with Transformers, Peebles & Xie, ICCV 2023.

$5$ On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models, Berrada et. al., NeurIPS 2024

$6$ Diffusion Model with Perceptual Loss. Lin & Yang, arXiv 2023.

Response iXeA

We were happy to see that the reviewer states that our extensive experiments “clearly show that the proposed latent perceptual loss objective improves metrics across different diffusion formulations and datasets”.
We respond to the different points raised by the reviewer point-by-point below.

W1 “framing the proposed loss as a perceptual loss”
Our motivation to use the term perceptual is the conceptual similarity to other losses based on deep features, which are typically based on classifiers. The Self-Perceptual approach of $1$ Lin & Yang, is also not based on classifier features. Our motivation to name our approach Latent Perceptual Loss allows us to make the connection to this related work, rather than to claim a connection between our work and human perception. Note that other “perceptual metrics”, such as DreamSim $2$ , rely on deep features from non-discriminative networks such as DINO and CLIP, see Section 4 of $2$ . We can discuss this point in the introduction to avoid any confusion on this point if the reviewer feels this would be helpful.

W2 “Could it be that the improvement actually… $comes from$ … different implicit weighting of timesteps”
Thank you for this interesting question, we address it in the newly added Section A.5 of the supplementary material. In Table 6, we verify that it is our Latent Perceptual Loss (LPL) rather than timestep weighting that underlies the performance improvements by comparing it to two different baselines that do not use LPL: (a) reweighting the loss of later timesteps to be similar to the implicit reweighting induced by using our LPL for later timesteps, (b) the state-of-the-art reweighting technique of Hang et al. $3$ . We find that neither of these baselines brings the improvements over the vanilla epsilon weighting, and that the best performing approach is our LPL. Thus, we conclude that the hypothesis that the improvements are not coming from LPL is unlikely.

W3 “It would be nice to see better backing up… such as introducing small (random) deviations and showing how their effect on the decoded image is disproportionally large compared to larger ones”

Similar to the discussion on the huggingface forum cited by the reviewer, we can fit a linear projection to directly predict the downscaled images from the latents, and then compare the decoding results. Please keep in mind that such a linear decoding produces images 8x smaller than the original, hence many details are lost. We provide two new experiments in section A.2 of the appendix, see paragraph “Linearity of the latents”. First, we find that the linear decoding idea from the cited discussion does not work well: the optimal linear mapping varies with the input image, and they introduce both high-frequency noise artifacts, as well as low-frequency color tone mismatches.
Additionally, such reconstructions are very approximative and are 8x smaller than the original image, hence the details can not accurately be represented with this method.

Finally, regarding our claim that the AE latent is “highly irregular”, we view this as a consequence of the compression rate of the autoencoder which we intuit relies on high frequencies to encode high level information.
Our experiment in section A.2, in which the interpolation operation simulates a low-pass filter on the latents, showcases this hypothesis in the case of SD autoencoders.
When taking into consideration the tendency of l2 to produce blurry results on regression tasks, we can conclude that the training objective is not ideal for the nature of the latent space.
We are open to accommodate reviewer’s suggestions on how to soften this claim based on the provided evidence.

Response sAd7

We thank the reviewer for their encouraging comments: “... motivation is clear …” and “method is simple and can be easily applied”. Below we address the different points raised in the review one-by-one.

W1 “clearly demonstrate the effectiveness and performance gain over the previous state-of-the-art methods”
Our baseline is based on the state-of-the-art SD3: we replicate their multimodal DiT (MMDiT) architecture and flow matching training procedure, and train the model on two public datasets (for reproducibility) as well as one proprietary dataset. Thus, we argue that the reported results should be considered as improvements over state-of-the-art methods. To better show that the baseline models we trained are competitive with SotA models, we provide a quantitative comparison on ImageNet at 512 resolution in Table 8 in the new Section A.7 of the supplementary material, showing that our Flow model, using RK4 ODE sampler already achieves SoTA results in terms of FID.

W2+W3 “perceptual loss would increase the computation cost … provide a clear comparison on this”, “performance comparison for the same training time instead of the same training iterations”
We thank the reviewer for this suggestion. We have added a new section A.6 to the supplementary material to study the effect of the computational and memory overheads induced by our Latent Perceptual Loss (LPL).
Results in Table 7 show that when maximizing the batch size for the available memory, and training for the same wallclock time of 48h, LPL still achieves improved FID scores despite a reduced throughput. A detailed tracking of FID over training iterations in Fig 15 shows that LPL leads to consistent improvements over the baseline after as little as 20k iterations.

W4 “wondering if the authors have tried other ways instead of simply masking … Or has the authors tried using some other models to compute the perceptual loss to avoid those outliers?”
(a) In our initial experiments we tried other approaches than masking, e.g. training the AE with L1 loss and clipping large activations. However, none of these approaches worked as well as the introduced outlier detection approach.

(b) We found outlier features in both AE architectures we explored: the asymmetric autoencoder from Zhu et al. (2023) and the autoencoder from SDXL (Podell et al., 2024). Recent papers (Kang et al. ECCV’24, Lin & Yang arXiv’24) consider other models to compute a perceptual loss: an image classifier train in autoencoder latent space, and a diffusion model Unet features, respectively. In the newly added Section A.4 of the supplementary material we provide a discussion-of and comparison-to these approaches. Experimentally we find LPL to lead to better results than these alternative approaches.