Zipper: Addressing Degeneracy in Algorithm-Agnostic Inference

We express our sincere gratitude for your dedicated time and thoughtful review of our paper. We would like to further elucidate the novelty and significance of our contribution as follows:

Broad applicability and compatibility with flexible training techniques: Our proposed Zipper device is designed for wide-ranging applicability across various goodness-of-fit testing scenarios, including variable importance assessment, specification testing, and model selection. Notably, this device necessitates minimal intervention in the model training process, enhancing its practicality.

Efficient utilization of test data: Unlike the approach by Williamson et al., which divides the test data into two non-overlapping parts (thereby halving the sample size for testing), our overlap scheme in the Zipper device allows for a more efficient use of test data. This results in substantial power improvements while maintaining valid size control, as demonstrated in our theoretical results and finite-sample experiments across various contexts.

We hope these points help to clarify the novelty and significance of our work. We welcome any further comments or questions you may have.

Thank you for your detailed review and constructive feedback on our paper. Your insights are invaluable for refining our manuscript.

Weaknesses:

1. Self-contained discussions:

• On conditions: Condition (C1) pertains to the optimality of the prediction function $f$, eliminating first-order estimation biases. Condition (C2) requires Hadamard differentiability of the predictiveness criterion. Condition (C3) demands accurate estimation of $f$, rendering second-order terms negligible. Condition (C4) controls the remainder terms. Condition (C5) ensures consistent variance estimators. We will add a detailed discussion of these conditions in the main text to improve readability.

• On Williamson et al.: Williamson et al. introduced a framework for algorithm-agnostic variable importance assessments using cross-fitting and demonstrated the asymptotic linearity of the test statistic. They also advocated additional splitting of the test data for null importance testing. We extend this framework to accommodate a wider range of goodness-of-fit tests, including specification testing and model selection. Our method, Zipper, promotes data reuse, significantly augmenting the testing power. We will elaborate on these points in the Related Work section to strengthen narrative coherence.

2. Related work: For variable importance assessments, in addition to LOCO methods within our framework, Shapley value-based measures are commonly used (Lundberg and Lee, 2017; Covert et al., 2020, 2021; Kumar et al., 2020). These measures, which estimate the incremental predictive accuracy contributed by a specific variable across all possible covariate subsets, reveal complex inter-variable relationships but at a considerable computational expense. Furthermore, conditional randomization tests (Candes et al., 2018; Tansey et al., 2021) offer a robust alternative when covariate distributions are known or can be accurately estimated. These methods are especially beneficial in semi-supervised settings with extensive unlabeled data. Additional methods, such as LIME (Ribeiro et al., 2016) and Floodgate (Zhang and Janson, 2020), will be discussed to expand the comprehensiveness of our manuscript's Related Work section.

3. Comprehensiveness of numerical experiments:

• Preference of Zipper in Figure 2: Despite WSGC-2 and DSP-Pert showing superior power in certain settings ($p=1000$), both exhibit considerable size inflation-approximately 0.2 and 0.4 respectively, where the nominal level is 0.05. This size distortion, depicted at $\delta=0$ in Figure 2, renders these methods less reliable for practical application. Conversely, Zipper maintains robust size control and achieves high power, affirming its utility.

• Size and power in Table 2: To clarify, Scenario (i) represents a specific case of $H_0$, while Scenarios (ii) and (iii) exemplify $H_1$ instances. We will rename Scenario (i) to $H_0^{'}$ and Scenarios (ii) and (iii) to $H_1^{'}$ and $H_1^{''}$, respectively.

• Improvement over baseline in Figure 3: Both Zipper and the baseline method (WGSC-3) demonstrate effective size control. In the MNIST handwriting dataset example, we perform sequential variable importance tests to determine the relevance of each region for prediction. We apply Bonferroni corrections to address multiple testing concerns. Figure 3 indicates that Zipper identifies more significant regions (five regions filled in red) compared to the baseline (two regions).

• Alignment with a previous study in Table 3: In our analysis of the Bodyfat dataset, we perform multiple tests with a Bonferroni correction set at $\alpha=0.05/10=0.005$. Table 3 reveals that Zipper successfully identifies both the Abdomen and Hip as significant variables (both $\le 0.005$), whereas WGSC-3 identifies only the Abdomen. Notice that the detection of the Hip variable corroborates results from previous research. We will highlight these outcomes in the revised manuscript.

Questions:

1. Variance estimator: The improvement in testing power is attributed to both the overlapping mechanism of Zipper and the variance estimator. This estimator is consistent under $H_0$, thereby ensuring valid size control. Under $H_1$, it underestimates the actual unknown variance, which is advantageous for power enhancement as demonstrated. However, for confidence interval construction-e dual problem of hypothesis testing-a consistent variance estimator under both $H_0$ and $H_1$ is necessary. We provide such an estimator. See Remark 2.9 and Section D.

2. Preference of Zipper in variable importance testing: This is addressed in our response to Weaknesses 2.

3. Combination of results of Table 1 and Figure 2: Please refer to our response to Weaknesses 3, Point 1.

4. Power and Size in Table 2: This is explained in our response to Weaknesses 3, Point 2.

5. Explanation of Figure 3: Please see our response to Weaknesses 3, Point 3.

Limitations in larger scale settings and locating important subsets: We acknowledge and value this critical feedback. Zipper is specifically designed to evaluate the significance of individual features or subsets in predictions, which paves the way for large-scale comparisons as mentioned at the end of Section 4. Notably, we can conduct a sequence of variable importance tests, each aimed at assessing the relevance of a specific variable $X_j$ in the predictive model while controlling for a global error rate. This procedure necessitates the fitting of $p+1$ models: one that includes all variables and $p$ null models, each excluding a distinct variable. Such a process is computationally demanding. Moreover, accurately controlling error rates presents a considerable challenge due to complex dependency structures among the p-values. We will elucidate this point further in the revised manuscript.

We appreciate the opportunity to enhance our paper based on your feedback and look forward to further discussions.

We extend our sincere gratitude for your thoughtful and detailed review of our paper. We greatly appreciate your positive evaluation and constructive feedback. We are committed to further refining our manuscript and eagerly welcome any additional comments or questions you may have.