Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models

We sincerely appreciate your time and thoughtful feedback. We are especially grateful for your recognition that our method is well-motivated, theoretically grounded, and addresses the critical challenge of reward hacking. We also value your observation regarding the complementary roles of spatial and spectral domain regularization in enforcing Gaussianity.

We would like to take this opportunity to re-emphasize our contributions:

• Moment-based loss (Section 3.1): Unifies independently developed regularization terms such as KL divergence, norm, and kurtosis losses within a single framework.

• Power-spectrum-based loss (Section 3.2): Defined via the DFT in the spectral domain, it regularizes DFT coefficients to match their theoretical distribution. Unlike PRNO, which removes spatial correlation by aligning the empirical covariance matrix with the identity and incurs quadratic complexity, our method uses FFT for efficient computation with $O(D \log D)$ complexity and operates via a fundamentally different mechanism.

• Dual-domain regularization (Section 3.3): We apply spatial and spectral losses jointly, leveraging their complementary strengths—the spatial loss ensures marginal distribution alignment, while the spectral loss enforces spatial neutrality. (L116)

Below, we address the specific questions and concerns raised:

Q: This paper is mainly evaluated only on a single text-to-image model (FLUX) with limited prompts. Broader evaluation on other architectures or diffusion models would strengthen the paper

A: To assess the generality of our approach, we additionally evaluated our method using the Stable Diffusion XL-Turbo (SDXL-Turbo) model for text-aligned image generation:

Our method outperforms the two strongest baselines on PickScore and ImageReward, demonstrating its effectiveness beyond the original FLUX model. We will include these results in the revised version of the paper.

Q: It would help if authors can discuss more regard whether matching higher-order moments could yield further improvements or not.

A: We report additional results using different $\mathcal{K}$ values in text-aligned image generation:

The performance with {1,2} shows a clear performance improvement over both $\emptyset$ and {1}. However, we did not observe further improvements when extending to higher-order moment losses. The first two terms were sufficient to surpass baseline performance, as our regularization loss was designed not only to incorporate moment-based losses but also to include a power-spectrum-based loss that effectively suppresses residual spatial structure. (L187) Additionally, we provide a unified framework that integrates regularization losses which had previously been used independently.

Q: It assumes that optimizing the latent alone suffices to align rewards, but I didn't see it explore interactions with jointly optimizing text or conditioning variables, which might be relevant in practice.

A: While we could not find prior work exploring joint optimization of conditional variables in test-time reward alignment for text-to-image models, we investigated this direction by updating both the text embedding and the latent variable during optimization. The results are summarized below:

Reg. denotes regularization, and Emb. Opt. denotes text embedding optimization.

We did not observe any performance improvement from text embedding joint optimization.

Additionally, we would greatly appreciate it if you could kindly elaborate the following point you raised as a limitation:

“it would be good to expand on how latent regularization might propagate biases from the pre-trained generator.”

We are eager to understand your concern more deeply so that we may address it properly in the revised version.

We sincerely thank you for acknowledging our rebuttal and for indicating that our clarifications addressed the concerns, and we truly appreciate the decision to raise the final score.

Thank you for taking the time to review our work and provide thoughtful and constructive feedback. We organize our responses into four categories:

Q: While the paper provides theoretical foundations for moment conditions and spectral properties, it lacks deeper theoretical analysis of why dual-domain regularization is necessary.

A: Our core rationale for dual-domain regularization is rooted in the complementary limitations and strengths of moment-based and power-spectrum-based losses—an insight supported theoretically.

Moment-based losses are permutation-invariant and therefore do not account for the ordering of elements in a sequence. (L185) For instance, consider a sorted sequence of 10K samples drawn from a standard Gaussian distribution. Although this sequence does not resemble a natural realization of random Gaussian samples—since such perfect ordering is highly improbable—the moment-based loss cannot distinguish it from a genuinely random Gaussian sample because it only matches marginal statistics. This limitation is theoretical, not merely empirical.

To address this, we propose a power-spectrum-based loss, which is permutation-variant and sensitive to spatial correlations. (L43, L186) By analyzing the DFT coefficients, it captures deviations from white Gaussian noise and penalizes residual spatial structure. In Lemma 1, we provide a theoretical characterization of the expected distribution of the power spectrum under the assumption of independently sampled Gaussian noise in the spatial domain. This formal foundation justifies the use of spectral loss as a principled way to enforce spatial neutrality.

However, the spectral loss alone does not ensure alignment with the target marginal distribution, as it focuses only on spatial structure. Therefore, both losses must be used together:

• The moment-based loss ensures the correct marginal distribution.

• The spectral loss enforces the absence of spatial correlation.

These two losses play complementary roles in fully regularizing toward Gaussianity (L116). To further support this, we provide visual evidence in Figure 1, which illustrates why regularization in both domains is necessary.

If you have a specific theoretical direction or additional depth in mind, we would greatly appreciate further guidance and are open to incorporating it into the revision.

Q: How were key parameters like λpower=25.0 chosen?

A: We present results on text-aligned image generation using different values of $\lambda_{\text{power}}$:

We observe that performance remains stable across a wide range of $\lambda_{\text{power}}$ values and consistently improves over the case with $\lambda_{\text{power}} = 0$. These results suggest that our loss is not sensitive to this parameter. We chose 25.0 considering the trade-off.

Q: Loss function values cannot reliably indicate Gaussianity, could you design better Gaussianity evaluation metrics?

A: We would like to clarify the intent of our statement in L327. The key point of L327 is that while our loss is effective for regularization—optimizing it guides the latent toward the Gaussian manifold (Figure 3) and prevents deviation—its absolute value is not meant to serve as a global Gaussianity score for comparing arbitrarily distant points. As the title "Gaussianity Regularization" suggests, our focus is on designing an effective regularization, not a general-purpose metric. This distinction does not diminish its utility, as our experimental results consistently support its effectiveness in practice.

Q: How does the method perform on other generative models?

A: We evaluated our method using the Stable Diffusion XL-Turbo (SDXL-Turbo) model for text-aligned image generation:

Our method achieves improved performance over the two strongest baselines in both PickScore and ImageReward metrics, demonstrating its effectiveness beyond the originally tested model, FLUX. We will include this result in the revised version of the paper.

Thank you for taking the time to carefully review our work and provide valuable feedback. We address the concerns below.

Q: After reading the abstract, as a reader, I wasn't sure what was the application of the current method.; The paper would read very well if it is rewritten as a latent regularization loss for reward alignment instead of its current form.

A: We appreciate your comment regarding the clarity of our paper. Our intention was to introduce the proposed regularization loss in a broader context, as Gaussianity is a fundamental concept with wide applicability across various domains. However, we understand that this broader framing may have made it difficult to identify the specific focus of our work. In response to your helpful suggestion, we will revise the paper to more clearly position it as a latent regularization method for reward alignment.

Q: Why not enforce more terms since matching more moments implies getting much closer to the gaussian manifold.

A: We report additional results using different $\mathcal{K}$ values in text-aligned image generation:

The performance with {1,2} shows a clear performance improvement over both $\emptyset$ and {1}. However, we did not observe further improvements when extending to higher-order moment losses. The first two terms were sufficient to surpass baseline performance, as our regularization loss was designed not only to incorporate moment-based losses but also to include a power-spectrum-based loss that effectively suppresses residual spatial structure. (L42, L186) Additionally, we provide a unified framework that integrates regularization losses which had previously been used independently. (Section 3.1)

Q: In L314 authors claim that their method achieves higher rewards and outperforms baselines with fewer updates. Please provide experimental evidence to support this statement.

A: We acknowledge that this expression may be misleading. Our intended meaning is that the proposed regularization helps to reach comparable reward with fewer optimization steps. We will revise the sentence to improve clarity as follows:

“As a result, it achieves higher rewards over baselines for the same iterations.”

Thank you for your thoughtful review and valuable comments. We address the concerns in detail below.

Q: How sensitive is $\mathcal{K}$ in the joint objective?; Did you experiment with moments higher than 2?

A: We report additional results with different $\mathcal{K}$ values in text-aligned image generation:

The performance with {1,2} is clearly improved compared to $\emptyset$ and {1}. However, we did not observe further improvements when extending to higher-order moment losses. The first two terms were sufficient to surpass baseline performance, as our regularization loss was designed not only to incorporate moment-based losses but also to include a power-spectrum-based loss that effectively suppresses residual spatial structure. (Section 3.2) Additionally, we provide a unified framework that integrates regularization losses which had previously been used independently. (Section 3.1)

Q: How sensitive are the experiments to $\lambda_\text{power}$?

A: We present the following results on text-aligned image generation with varying $\lambda_{\text{power}}$:

We observe that the performance remains stable across a wide range of $\lambda_{\text{power}}$ values, and is consistently better than the case with $\lambda_{\text{power}} = 0$. These results indicate that the method is not sensitive to this parameter.

Q: whether the primary contribution of the experiments is in adding the constraint within the spectrum domain

A: We would like to clarify that our contributions extend beyond simply introducing a spectral-domain constraint.

• Moment-based loss (Section 3.1): Unifies independently developed regularization terms such as KL divergence, norm, and kurtosis losses within a single framework.

• Power-spectrum-based loss (Section 3.2): Defined via the DFT in the spectral domain, it regularizes DFT coefficients to match their theoretical distribution. Unlike PRNO, which removes spatial correlation by aligning the empirical covariance matrix with the identity and incurs quadratic complexity, our method uses FFT for efficient computation with $O(D \log D)$ complexity and operates via a fundamentally different mechanism.

• Dual-domain regularization (Section 3.3): We apply spatial and spectral losses jointly, leveraging their complementary strengths—the spatial loss ensures marginal distribution alignment, while the spectral loss enforces spatial neutrality. (L116)

Therefore, we emphasize that our contribution is not limited to the addition of a spectral loss, but lies in the integration and synergy of both domains within a coherent and efficient framework.

Q: Is the joint objective just a combination of KL and PRNO?

A: Our loss is not simply a combination of KL and PRNO. We would like to clarify two key points:

• The primary distinction lies between PRNO and our power-spectrum-based loss. PRNO suppresses spatial correlation by encouraging the empirical covariance matrix to approximate the identity matrix (L80), which incurs quadratic complexity. In contrast, our method leverages the FFT to compute DFT coefficients and regularizes them to match the theoretical distribution. This approach results in a significantly lower complexity ($O(D \log D)$) (L48). As shown in Figure 3, PRNO takes 14.1 seconds for 100 iterations, whereas our method requires only 0.26 seconds, achieving over 50× speed-up—while more effectively matching the Gaussian distribution.

• We also emphasize the complementary roles of spatial and spectral regularization, as discussed in Section 3 and illustrated in Figure 1. While both KL and PRNO do not address regularization in the spectral domain, our formulation incorporates spectral-domain regularization to more effectively enforce Gaussianity. Importantly, our regularization loss shows superior performance compared to these baselines in the experiments.

Q: Why regularization is needed in both the spectral and spatial domains simultaneously; How spectral and spatial domain regularization work together

A: Moment-based losses permutation-invariant and therefore do not account for the ordering of elements in a sequence. (L185) For example, consider a set of 10K Gaussian samples that have been sorted. Although this sequence does not resemble a natural realization of random Gaussian samples—since such perfect ordering is highly improbable—the moment-based loss cannot distinguish it from a genuinely random Gaussian sample because it only matches marginal statistics.

To address this, we propose a power-spectrum-based loss, which is permutation-variant and sensitive to spatial correlations. (L43, L186) By analyzing the DFT coefficients, it captures deviations from white Gaussian noise and penalizes residual spatial structure. In Lemma 1, we provide a theoretical characterization of the expected distribution of the power spectrum under the assumption of independently sampled Gaussian noise in the spatial domain. This formal foundation justifies the use of spectral loss as a principled way to enforce spatial neutrality.

However, the spectral loss alone does not ensure alignment with the target marginal distribution, as it focuses only on spatial structure. Therefore, both losses must be used together:

• The moment-based loss ensures the correct marginal distribution.

• The spectral loss enforces the absence of spatial correlation.

These two losses play complementary roles in fully regularizing toward Gaussianity (L116). To further support this, we provide visual evidence in Figure 1, which illustrates why regularization in both domains is necessary.