Learning Parametric Distributions from Samples and Preferences

We thank Reviewer yquN for the time spent and the positive feedback. We address the reviewer’s questions below.

1. Reward models Except for Theorem 4.3, all the derivations in Section 4 hold for general (hence reward-based) preference models provided Assumptions 4.2,4.4,4.5 and 4.7 hold. Characterizing the expressivity of parametric rewards satisfying those assumptions is interesting, yet challenging. We provide two positive and one negative examples.

• Positive: monotonic reward. Suppose that $\tilde \ell_{\theta}(x,y) = f(p_{\theta}(x)) - f(p_{\theta}(x))$ where $f$ is increasing on $[0,1]$. Since $sign(\tilde \ell_{\theta}) = sign(\ell_{\theta})$, hence the parameters with zero classification loss and our estimators are the same. Therefore, our results hold for this class of rewards when our assumptions hold for the log-likelihood reward. When $f$ is decreasing, the preferences are “reversed”, and similar arguments can be made. This example includes (1) normalization by a multiplicative constant (e.g., temperature $\beta$) and (2) the odds-ratio reward-based preference based on $f(x) = \log(x/(1-x))$ and defined by Hong et al. (2024, ORPO: Monolithic Preference Optimization without Reference Model).
• Positive: margin with Gaussian. Suppose that $\tilde \ell_{\theta} = \ell_{\theta} + c$ where $c$ is a constant and $\ell_{\theta}$ is the Gaussian log-likelihood preference. By extending our computations from Appendix E, Assumptions 4.2, 4.4, 4.5, and 4.7 hold with $c$-dependent positive constants. Margins are used by Meng et al. (2024, SimPO: Simple Preference Optimization with a Reference-Free Reward) and IPO from Azar et al. (2023, A General Theoretical Paradigm to Understand Learning from Human Preferences).
• Negative: reference model with Gaussian. Suppose that $\tilde \ell_{\theta} = \ell_{\theta} - \ell_{\theta_0}$ where $\theta_0$ is known and $\ell_{\theta}$ is the Gaussian log-likelihood preference. Since $\tilde \ell_{\theta}(x,y) = \langle x-y, \theta - \theta_0 \rangle$ and $\nabla_{\theta} \tilde \ell_{\theta}(x,y) = x-y$, Assumption 4.5 is violated for $u=\theta^\star - \theta_0$. Not all direct alignment algorithms rely on a reference model (see SimPO or ORPO).

2. Accelerated rate Accelerated rates arise when accumulating random variables having a positive density at a specific point through a minimum (or maximum) operator.

• When estimating the location parameter $\theta$ of a uniform distribution over $[\theta,\theta+1]$, the optimal estimator achieving the accelerated rate is the minimum of uniform observations whose density is positive at $\theta$.
• For deterministic preferences with log-likelihood rewards, we observe the true ordering between likelihoods. This enforces a hard constraint on the admissible parameters, which can be expressed with a minimum operator. More precisely, the maximal deviation $R_{n,u}$ along direction $u$ is upper bounded by the minimum of positive random variables whose density is positive at zero under Assumption 4.7. With high probability, this min operator is upper bounded by $O(1/n)$. The proof combines Lemma 4.6 and an upper bound on the inverse of the cdf based on a Taylor expansion around $0$.

3. Misspecification Under misspecification, the deterministic preferences might not provide separability within $\Theta$ since $\theta^\star \notin \Theta$. Then, DP MLE should be defined as SP MLE by using the 0-1 loss $1(u < 0)$ instead of the logistic loss $-\log \sigma(u)$. This combines a cross-entropy loss and a classification 0-1 loss, reweighted by a regularization $\lambda > 0$. This objective is reminiscent of single-stage alignment procedures such as ORPO and ASFT, see Gorbatovski et al. (2025, The Differences Between Direct Alignment Algorithms are a Blur). Without separability, computing DP MLE can be NP-hard. Under sufficient regularity, DP MLE converges to $\theta_{0} \in argmin_{\theta \in \Theta} KL(\theta^\star,\theta) + \lambda m(\theta)$ where $m$ as in line 270, where $\theta_0 \ne \theta^\star$. This minimization is challenging, as $\theta \to m(\theta)$ might not be convex. Deriving a tractable ELBO method for this optimization is an interesting direction to obtain tractable and robust estimators. As $\theta_0$ lies in the boundary of $\Theta$, we should control the maximal deviation wrt to $\theta_0$ for directions that point towards the interior of $\Theta$ to prove an accelerated rate. While some elements of our analysis might be salvaged, we believe that finer technical arguments should be derived to capture this interesting setting.

Restrictive assumptions See the answer to Reviewer vgrP for a detailed discussion on how to weaken them to local conditions.

We thank Reviewer Gth4 for the time spent and the encouraging feedback. We address the reviewer’s questions below.

Iterative human preference alignment We investigate the case where pairs of observations and their preferences are tied together, which includes the log-likelihood ratio as preference. We detail the connections with iterative human preference alignment.

• Many human preference alignment methods build on the Bradley-Terry (BT) model for preference, based on rewards. Direct alignment algorithms use variants of the log-likelihood to define the implicit reward of a policy. Choosing $\ell_{\theta}(x,y) = \log p_{\theta}(x) - \log p_{\theta}(y)$ coincides with the optimal policy for maximum entropy RL (see, e.g., Swamy et al, 2025, All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning).
• For offline preference data, the assumption $(X,Y) \sim p_{\theta^\star}^{\otimes 2}$ is unrealistic as $\ell_{\theta^\star}$ is collected from a fixed data set of pairs of observations. Recent LLMs are built on iterative alignment procedures. At stage $N$, the model $p_{\theta_N}$ is trained based on the preference data for generations by the previous model, i.e., $(X,Y) \sim p_{\theta_{N-1}}^{\otimes 2}$. Under the realizability assumption and without mode collapse, this self-refinement paradigm converges towards the true model $p_{\theta^\star}$. Our setting characterizes the limiting behavior of this iterative process, i.e., preference based on $\ell_{\theta^\star}$ for observations from $p_{\theta^\star}$.

1. RU versus DP Figure 1(a) provides evidence suggesting that the randomized estimator (RU) and the worst-case estimator (WE) perform on par with DP MLE: RU is slightly better than DP, itself slightly better than WE. Figures 1(b) and (2) highlight that DP outperforms WE and AE for larger dimensions, where the gap increases when $d$ is nonnegligible compared to $n$. Therefore, only DP obtains the best-of-both world estimation error rate. For large $d$, implementing RU is challenging. We conjecture it suffers from the same limitation as AE. This is supported by additional experiments on new estimators using the setting of Section 6, see anonymous plots at https://anonymous.4open.science/r/ICML25SuppExp .

• Figure 1(a) extended. The center estimator (CE) returns the center of the interval $\mathcal C_n$. The truncated Gaussian estimator (TrG) returns a realization from a Gaussian distribution with mean CE and variance $4/n$, which is truncated to $\mathcal C_n$. TrG performs on par with RU. CE outperforms both TrG and RU. This suggests that being far away from the boundary of $\mathcal C_n$ improves performance compared to DP that lies on the boundary of $\mathcal C_n$ as observed empirically. Moreover, randomization on $\mathcal C_n$ worsens performance compared to CE. Using the derivation in lines 55-63, it is coherent that CE improves on DP by a multiplicative constant: the average of those two (non-independent) random variables decreases faster. This can be proved by refining the proof of Lemma 4.6 to account that $n = N_{\theta^\star,-1} + N_{\theta^\star,1}$ (defined in Line 658).
• Figures 1(b) and 2 extended. For $d>1$, multiple centers exist and we use the Chebyshev center estimator (CCE) of $\mathcal C_n$. While CCE outperforms AE by a constant margin, CCE only outperforms DP in the regime of large $n$ compared to $d$. It performs worse than SO for small $n$. Geometrically, for small $n$ and large $d$, the random polytope $\mathcal C_n$ is more likely to be “spiky” along some directions. Due to those distant vertices, the center becomes a worse estimator than DP, since the “average” is intuitively less robust to outliers. In contrast, DP dominates SO statistically (Lemma 4.1), hence it achieves rate $O(\sqrt{d/n})$ when $n$ is small compared to $d$.

2. Line style The dashed line is shorter than the solid line, see SP (sto) and SO, yet others are not distinguishable. We will correct this.

We thank Reviewer vgrP for the time spent and the detailed comments. Due to the limited space, we only address some of the reviewer’s concerns.

Restrictive assumptions While our research question is inspired by iterative human preference alignment (see the answer to Reviewer Gth4), we do not claim the direct applicability of DP MLE for realistic LLM training. When studying DP MLE only, we conjecture that the “global” assumptions 4.2 and 4.4 can be weakened to local versions. Using time-uniform concentration results, we can build a sequence of shrinking confidence regions $(R_n)_n$ around SO MLE that contains $\theta^\star$ for all time $n$ with high probability (whp). Then, we modify DP MLE to be constrained on $R_n \cap C_n$ that contains $\theta^\star$ whp. For $n$ large enough, $R_n \cap C_n$ will be included in a local neighborhood of $\theta^\star$ under which the “local” assumptions 4.2 and 4.4 are satisfied. Given that Assumption 4.4 is based on “ignoring” the reminder term in a first-order Taylor expansion, assuming a local version is a significantly weaker requirement.

1. Misspecification There are two possible sources of misspecification not taken into account by our current analysis.

• Misspecified observations. See answer to Reviewer yquN when $\theta^\star \notin \Theta$. When $p^\star \notin F$, $L_n(\theta)$ is a quasi-log-likelihood term, as $F$ doesn’t contain the true structure. Under sufficient regularity, SO quasi-MLE converges towards $\theta_{0} \in argmin_{\theta \in \Theta} KL(p^{\star}, p_{\theta})$ where $p^\star \ne p_{\theta_0} \in F$. Without the separability from well-specified deterministic preference, we define DP quasi-MLE as in the answer to Reviewer yquN. Under sufficient regularity, this estimator converges towards the minimizer of a similar optimization problem based on the above KL and a misspecified equivalent of $m(\theta)$.
• Misspecified preferences. The Bradley-Terry (BT) model that uses reward-based preferences has limited expressivity as it doesn’t allow for intransitive preferences. Even when individuals exhibit transitive preferences, their averaged preferences might be intransitive due to disagreements. See Munos et al. (2024, Nash Learning from Human Feedback) or Swamy et al. (2024, A Minimaximalist Approach to Reinforcement Learning from Human Feedback).

2. Verifying our assumptions In all generality, it is challenging to give a general recipe to formally verify those assumptions. A formal verifier (Lean) or software (SageMath) might be useful given a closed-form definition. Numerically, those assumptions can be confirmed or rejected by sampling from $p_{\theta^\star}^{\otimes 2}$. Assumption 4.4 is rejected by exhibiting $(X_i,Y_i) \in \tilde D(\theta^\star,\theta) \setminus D(\theta^\star,\theta)$. Assumptions 4.2 and 4.5 are confirmed by finding $(X_i,Y_i) \in D(\theta^\star,\theta)$ and $(X_i,Y_i) \in G_{1}(\theta^\star,u)$. The sampling complexity of such tests scales as the inverse event’s probability. Using Dvoretzky–Kiefer–Wolfowitz inequality, $F_{\theta^\star,u}$ can be estimated to verify Assumption 4.7 hold. Our additional experiments with accelerated rate include Laplace and Rayleigh distributions, see the anonymous plots at https://anonymous.4open.science/r/ICML25SuppExp .

RU versus DP See answer to Reviewer Gth4.

Covariance Gap Our simulations suggest that the asymptotic gaps between SO and SP are mild. The empirical gap is also small for moderate $n$.

0-1 loss See answer to Reviewer yquN. For non-separable data, the minimization of the 0-1 classification loss can be NP-hard even for the simple class of linear classifiers, e.g., Feldman et al. (2018, Agnostic Learning of Monomials by Halfspaces is Hard). Inspired by (Tang et al., 2024, Generalized Preference Optimization: A Unified Approach to Offline Alignment), we implement estimators based on other convex surrogates: Hinge, Square, Truncated square, Savage, and Exponential. All estimators perform on par with the logistic loss, see the plot at https://anonymous.4open.science/r/ICML25SuppExp .

Intuition on $V_{\theta^\star,u}$ It quantifies the amount of information in $(X_i,Y_i)$ to discriminate $\theta^\star$ from other parameters on the half-line directed by $u$. The lower $V_{\theta^\star,u}(X_i,Y_i)$ is, the more discriminative $(X_i,Y_i)$ is.

Informative preferences For observations with null preference gradient, parameters close to $\theta^\star$ could have similar preferences. Therefore, those samples are not sufficient to discriminate between them.

Negative examples Those claims are a direct consequence of the definitions and will be proved in Appendix for completeness.

Typo It will be fixed.