Differentially Private Analysis for Binary Response Models: Optimality, Estimation, and Inference

Validity of Definition 3.1, Theoretical Claims and Question 1.

To satisfy $(\epsilon, \delta)$-LabelDP (and similarly for $\epsilon$-DP), the conditional probabilities $p_{00}=P(Y^*=0 \mid Y=0)$ and $p_{11}=P(Y^*=1 \mid Y=1)$, which lie in $(0,1)$, must meet the following privacy constraints: $$ P[Y^=0 \mid Y=0] \leq e^{\epsilon} P[Y^=0 \mid Y=1]+\delta, \; \; P[Y^=1 \mid Y=1] \leq e^{\epsilon} P[Y^=1 \mid Y=0]+\delta,

P[Y^=0 \mid Y=1] \leq e^{\epsilon} P[Y^=0 \mid Y=0]+\delta, \; \; P[Y^=1 \mid Y=0] \leq e^{\epsilon} P[Y^=1 \mid Y=1]+\delta.

Experimental Designs or Analyses 1.

We appreciate the reviewer pointing out this observation. The abrupt jumps around small $\varepsilon$ values (particularly near $\varepsilon=0.1$) observed in Figures 2 and 4 arise primarily from the specific data generation mechanism of our simulation study for the parameter $\boldsymbol{\beta}$. To clarify, after revising and slightly adjusting our data-generating procedure (especially for $\boldsymbol{\beta}$), we observed that these jumps significantly diminish and the curves become noticeably smoother. This smoother property aligns well with the results of our real data analysis (Figure 6) in our paper, where the curves demonstrate a much more stable and smooth pattern. This smoothness of the real data further supports that the initial observed fluctuations were due to the original simulation setup rather than an inherent instability of the proposed methodology.

Following your suggestion, we will increase the simulation runs and provide smoother simulation results to better illustrate the effectiveness and robustness of our approach.

Experimental Designs or Analyses 2.

We thank the reviewer for noting the ambiguity with respect to the scaling of the $x$-axis in Figures $2-6$. In fact, the chosen privacy budget values $(\varepsilon)$-specifically $\\{0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.7, 1\\}$-were not evenly spaced or set according to a log-scale. Instead, these values were deliberately selected in an irregular and somewhat random manner to extensively test the robustness of our method across a wide range of privacy constraints.

This deliberate randomness and irregularity in the choice of $\varepsilon$ values ensure our results are not sensitive to systematic or uniformly spaced intervals. Despite such randomness, our method consistently demonstrates strong performance, validating its robustness across varied privacy settings

Essential References Not Discussed

The reviewer raised an insightful question about the connection between our optimal LabelDP setting with covariates and traditional DP scenarios (without covariates). Although both papers share the goal of optimal privacy-utility trade-offs for binary responses, our work makes two key advances beyond their framework.

• Covariate Integration: Awan \\& Slavković optimizes the RR mechanisms solely for the binary response $Y$ (minimizing $\operatorname{Var}(\widehat{\sum{Y}})$, where $\sum{Y}$ is the sample sum, which is a complete sufficient statistic for the binomial model in their paper) without considering covariate information $X$, while our Fisher information maximization (Eq. 5) explicitly incorporates covariate effects through the model $\mathbb {E} \left( {Y} _ i \mid { {X} } _ i \right) =1-p _ {00}+\left( p _ {00} + p _ {11} - 1 \right) G \left( {\beta} _ * ^ {\top} X _ {i} \right)$. This criterion, known as T-optimality, explicitly leverages the covariate structure, enhancing the estimation efficiency when covariates are present.

• Inferential Guarantees: Beyond point estimation, our framework enables formal statistical inference with privacy guarantees. Specifically, Corollary 4.11 establishes the asymptotic normality of each coefficient $\beta_ { * j}$ under $\varepsilon$- and $(\varepsilon,\delta)$-LabelDP mechanisms, yielding valid confidence intervals that:

  • Achieve nominal coverage (e.g., 95% intervals contain $\beta_{* j}$ with probability 0.95 as $n \rightarrow \infty$).
  • Preserve privacy through the optimized RR mechanism (Theorems 4.4 and 4.7).
  • Account for covariate effects via the Fisher information matrix.

We will add a discussion in Section 2 comparing these approaches, citing Awan \\& Slavković's marginal optimality as motivation for our novel optimality under the regression framework.

Other Comments or Suggestions.

We agree with the reviewer on the readability issue. In our revised manuscript, we will significantly improve the readability of all figures, especially Figure 1, by:

• Increasing font sizes to match the main text consistently.
• Clarifying annotations and axis labels.
• Enhancing visual clarity through improved formatting and spacing.

Question under Experimental Designs Or Analyses

We acknowledge the concern of the reviewer about the large simulation sample size ($n=10^5$) for asymptotic approximations. This choice was intentional to validate our theoretical results (Theorem 4.7), where the guarantees of consistency and normality are held as $n \rightarrow \infty$. However, we emphasize that our method is equally applicable to smaller samples, as demonstrated in the real-data analysis (Section 6) with $n=474$ - a realistic sample size for sensitive surveys. Here, ORRbR achieved a 30% lower MSE than $\operatorname{RRbR}$ at $\varepsilon=0.1$ (Figure 7), proving its practical utility beyond asymptotic regimes.

To further address this point, we will add simulations for $n \in \\{500,2000,5000\\}$ in revision, showing that ORRbR maintains superior coverage (>92% versus RRbR's <85% at $n=500, \varepsilon=0.5$). Then we will clarify in Section 5 that the Fisher information optimization remains valid for small $n$, although variance estimates may require finite-sample adjustments (e.g., sandwich estimators).

Weakness 1 and Question 1.

We appreciate the insightful suggestion of the reviewer on multiclass extensions. Indeed, our method can be naturally extended to multiclass outcomes using the $k$-RR mechanism. The optimization strategy of maximizing the trace of the Fisher Information Matrix remains valid for multiclass scenarios, as discussed in Yao and Wang (2019), Optimal subsampling for softmax regression. Specifically, for $k$-class responses, the design matrix in Definition 2.2 of our paper becomes a $k \times k$ stochastic matrix, and the optimality criterion would involve maximizing the Fisher Information's trace across multiple response probabilities under LabelDP constraints. This is an exciting direction that we are actively exploring, and preliminary theoretical analysis indicates promising scalability. We plan to provide more detailed theoretical developments and numerical evaluations in future work.

We acknowledge the reviewer's observation regarding the large sample size of our simulation study ($n=10^5$). Our initial choice aimed to clearly demonstrate the theoretical properties under asymptotic conditions. However, we want to emphasize that our real data analysis employs a relatively small sample size of 474 German and Swiss students. This practical application highlights the robustness and applicability of our method even in small-sample contexts.

Weakness 2 and Question 2.

We thank the reviewer for highlighting the potential for other optimality criteria. It is important to note that while both $T$-optimality and $D$-optimality aim at maximizing information, $D$-optimality involves maximizing the determinant of the Fisher Information Matrix, which is computationally challenging and more complex due to the involvement of determinant calculations, especially for high-dimensional parameter spaces. However, $T$-optimality is computationally simpler, making it significantly easier to implement in practical scenarios without sacrificing the quality of inference. Moreover, in many cases, the two criteria are equivalent or closely related, and maximizing the trace often provides a satisfactory approximation to maximizing the determinant.

In detail, our $T$-optimality criterion (maximizing the trace) is equivalent in asymptotic efficiency but far more practical:

1. Computational tractability: The trace decomposes into a sum of variances (diagonal elements), reducing the problem to scalar optimization (Lemma 4.4). This allows closed-form solutions for $\left(p_{00}, p_{11}\right)$ on the boundary of $\mathcal{R}$ (Theorems 4.5 and 4.7).

2. Interpretability: The trace directly corresponds to minimizing the average variance of $\widehat{\boldsymbol{\beta}}$, which aligns with our goal of precise estimation.

3. Equivalence in large samples but different complexity: Both criteria yield consistent estimators, but $T$-optimality achieves this with $O(d)$ complexity versus $O\left(d^3\right)$ for $D$-optimality due to determinant calculations, where $d$ is the number of covariates.