We thank the reviewer for their strong approval of the paper and the valuable feedback. Below we address the questions raised by the reviewer.

“Selecting a model because it uses some characteristics rather than others (as accurately represented by SHAP) is not inherently problematic...”:

Choosing a model x ∈ R(t, D) where R(t, D) is the Rashomon set because it meets certain ethical, policy or domain-driven requirements is not problematic. Even if x ∈ C(t, D) (models that change explanations), a confirmatory analysis is legitimate if it is reported transparently. By contrast, malicious X-hacking occurs when an actor withholds other valid models from R(t, D) and only presents a model from the set C(t, D) ∩ R(t, D) that supports a desired narrative, thereby concealing alternative explanations that might contradict the narrative. Also pointed out by reviewer Se87, we appreciate this distinction and intend to put it earlier in the paper (perhaps Section 1) and then later discuss more in Section 7 (Discussion)

“Could you clarify what criteria or guidelines you envision for distinguishing harmful X-hacking from legitimate and beneficial model selection based on SHAP values?”: Our vision is that effective countermeasures or additional requirements from a publication venue to make reporting more transparent can help identify or raise flags for X-hacking which may require further scrutiny. Specifically,

1. Transparent reporting of model choice, clearly mentioning that different models yield different explanations and clarification of why the chosen pipeline/model’s explanation is more appropriate.

2. Pre-specifying a research plan and adherence to it which can be later verified will limit the degrees of freedom that produced the reported model.

3. Justifying the choices of explanations in a selected model may help give valuable insights.

4. A well documented research journey which is open, reproducible, and consistent with methodological standards will help identify X-hacking.

5. Awareness about model multiplicity and X-hacking and the need for effective countermeasures in the research community.

While the above points focus more on what researchers reviewers and peers can look for and raise concerns based on the non-availability of information that threatens transparency, a full-fledged automatic detector for X-hacking is an ideal vision for further development.

We thank the reviewer for their valuable feedback. The following addresses the reviewer’s questions and concerns.

p-hacking and X-hacking: Both p-hacking and X-hacking can arise from scale, chance and collinearity, which are interrelated rather than distinct. For instance, collinearity can cause regression estimates to become unstable, thus exacerbating p-value non-robustness. X-hacking, facilitated by AutoML, leverages model multiplicity and feature redundancy to align with desired explanations, also affected by these factors. X-hacking can be viewed as a generalization of p-hacking, extending the manipulation of statistical significance to the broader manipulation of model explanations; however the focus in our paper is on model-agnostic explanations, not including p-values, which tend to be considered only for a subset of models such as GLMs. Both phenomena can also occur unintentionally, highlighting the need for robust statistical practices and transparent reporting to mitigate these risks.

TimberTrek: As pointed out by Reviewer 9ZtB also, in a revision we will cite TimberTrek in Section 2 as an important work on visualising and understanding Rashomon sets.

Rashomon set and defensible models: The Rashomon set R is defined as the set of models that exhibit nearly equivalent predictive performance within an acceptable threshold $\epsilon$.

R = {$\{ m \in M | perf(m) \geq perf(m^*) - \epsilon \}$}

Given M as the set of all possible models, perf(m) as the predictive performance of model m, and m∗ as the best-performing model, the set of "defensible" models is equivalent to the Rashomon set R. This set allows for a performance decrease ϵ(m) that may vary by model, accommodating those considered 'acceptable' due to their adherence to standard practices in the domain of application/publication. Therefore, the set of defensible models is a Rashomon set: R(t, D) in the paper (section 4.1). Additionally, the set C(t, D) ∩ R(t, D) is the set of those defensible models that changed the explanations. We will make it clearer and keep notation for the set C(t, D) ∩ R(t, D) (Pg. 2 para. 1) where we mention finding defensible models which are the models from the set C(t, D) ∩ R(t, D), i.e., “defensible models that changed explanations”, to avoid confusion.

Figure 2 and Table 1: Figure 2 and Table 1 measure different aspects of the same set, C(t, D) ∩ R(t, D) defined in section 4.1. In figure 2, we see how many (the size) models in this set are found in a post-hoc manner by an off-the-shelf AutoML solution demonstrating that—with no special adjustments—AutoML is susceptible to X-hacking. On the other hand, in Table 1 we see how quickly the first member of this set is found by ad-hoc X-hacking. For a dataset the set C(t, D) ∩ R(t, D) may be large, yet a member of this set appears relatively late in the search process (longer time in Table 1). For another dataset C(t, D) ∩ R(t, D) may be small, but yields one member quickly during directed X-hacking (shorter time in Table 1).

A motivating example: Our following empirical analysis can be added as motivation after background section. Studies have shown a strong relationship between gender and cardiac disease [98, 99, 100, 101], but in an empirical experiment with our ad-hoc X-hacking, the importance of the feature gender was manipulated to drop to Rank 6 from a Rank of 1 in a baseline. The set C(t, D) ∩ R(t, D) had many candidates. First such candidate was found only in 24 seconds. This figure shows the results.

Pre-registration and X-hacking: We mention study pre-registration in Section 2 under p-hacking. Pre-registration is a valuable safeguard against X-hacking but might require more rigorous methodological detail. We will add this in Discussion.

Concern regarding notation: Obviousness and risk of getting caught are both represented by O. It is an arbitrary function in the current context. To avoid confusion with complexity notation, we will change O to another letter, say Z.

In Section 3 para. 3, we mention Q as any quality measure, which can be performance of a model perf(m) or an inferential summary of feature of interest I(m, x). It is only used to define a quality measure and later says (Eq. 1) that generally one optimises for a quality measure of performance during a model search: Q = perf(m).

Balanced view of X-hacking: In Section 3 para. 2 we mention that such hacking can be “deliberate or not”. However, explicitly mentioning it earlier in Section 1 para. 2 can set a broader view at possible reasons behind X-hacking. Specifically, we will add the phrase “unscrupulously, or through lack of time or experience”.

Editing “audacity” section: To maintain the focus of the reader, the subsection 3.3 will be removed to supplementary materials.

We thank the reviewer for their thorough and supportive comments. Furthermore, we will incorporate your suggestions to further strengthen the exposition and clarify details around our adversarial concepts, notation, and experimental setups. Specifically:

We agree that our discussions of adversarial objectives, risk tolerance, and detection were presented at a more conceptual level compared to the more rigorous empirical work. Our focus was to demonstrate and quantify the practicality of manipulating ML explanations, not to propose or experimentally validate a complete “risk model” of adversarial behaviour. In a revised version, we intend to shift parts of Section 6 (Detection and Prevention) to Section 7 (Discussion) to improve clarity.

“… not sure why the authors heavily explored the benefits of Bayesian optimization over random optimization. I am more interested in the gains an adversary would get over the cherry-picking method — i.e., fewer runs to achieve a desired X-hacking effect.”: We wanted to show that a malicious or benign user targeting explanations can exploit a more systematic search. We also wanted to show that given a greater time budget, random sampling will not on average perform better than Bayesian optimization in finding a defensible model. We explore this in an empirical manner and conclude that in our experiments, Bayesian optimization was 3 times faster at random sampling in the ad-hoc X-hacking setting (Section 5.4 (“Time to First Defensible Model”)).

“… you describe the data as collinear. Was it perfectly dependent or just correlated? Also, in Figure 8, is each point a different model? It would help to clarify that each point is a distinct pipeline choice and that for a fixed performance we can shift feature dependencies as we like. In Fig. 3, are the features ordered by baseline importance? It should be mentioned explicitly.”:

Collinearity: We used both moderate and high correlation scenarios, but not perfectly dependent features. Here we use “collinearity” referring to imperfect collinearity rather than perfect collinearity: i.e. the relationship is nearly but not exactly linear. We will clarify that the data is not perfectly dependent, but “redundant enough” to shift how SHAP values are allocated among correlated predictors without significantly affecting accuracy.

Figure 8: Yes, each dot on the plot corresponds to a different trained model/pipeline. In the revision, we will explicitly state that “each point represents one pipeline configuration with its resulting performance and SHAP score for that feature.”

Figure 3: Yes, the features in Figure 3 are shown in descending order of mean absolute SHAP under our baseline model. We will add a sentence in the caption or text noting that “features are ordered based on their baseline SHAP importance,”

We thank the reviewer for their thoughtful feedback and valuable references.

We agree that the phenomenon of having many equally accurate, yet interpretively distinct, models is well established, and we cite several related papers (e.g., Fisher et al., 2019; Brunet et al., 2022) to acknowledge this. However, our paper contributes a new perspective by explicitly demonstrating how modern AutoML pipelines can be adapted (or misused) to exploit those known multiplicities at scale to manipulate XAI metrics (referred to as “X-hacking”). While multiplicity has certainly been studied, we highlight in the related work section how this is limited to certain model families. Our approach highlights automation and practicality—showing that even modest budgets given to AutoML systems can allow for systematically “cherry-picking” explanations to support a predefined narrative, all while preserving acceptable predictive performance. This practical demonstration of how easily multiplicity can be exploited via off-the-shelf or lightly customized AutoML solutions distinguishes our work from purely theoretical or smaller-scale explorations of Rashomon sets. We acknowledge the recent preprint, Ganesh (2025) and add a reference to this in Section 2 (Background).

“More nuanced view on X-hacking is needed, as it can be used as an assistive tool to align the model with domain experts’ intuition.”: We acknowledge that choosing models that align with intuitions of domain experts may not always raise ethical concerns. For example, when domain experts prefer interpretable patterns that reflect established science or policy. However, the distinction lies in intent and transparency. We appreciate this insight and intend to add in Section 7 (Discussion) that model selection for domain alignment may not always attribute to malpractice provided it is transparent and reported in good faith. While our paper emphasizes the opposite, an expanded discussion of both benign and malicious scenarios will enhance the paper’s balance.

“Only one XAI method is used … it is not clear how X-hacking behaves for other XAI methods.”: We worked with SHAP for two reasons: it is a model agnostic approach, and it is a popular metric, making it a strong representative for our demonstration. We show that quantitative XAI metrics like SHAP can be easily manipulated using off-the-shelf AutoML solutions, thus sufficiently establishing the core risk of X-hacking at scale. We believe that any post-hoc explanation method -- especially those susceptible to model multiplicity -- will face similar risks, however, systematic exploration of other XAI metrics is planned as future work.

“The methods can be time-consuming and might be hard to generalize to more complex models.”: We share the concern that scanning a massive analysis space can be computationally expensive, however, we show that current off-the-shelf AutoML solutions and their search spaces can be used to find “defensible” models even under modest resource budgets. For extremely large or complex models, the cost (computational resources and time) of repeated training might be higher, however, it is the choice of the researcher to allot higher budgets in the context of their research. Our current intention is to show that X-hacking can be easily and effectively done at scale, even with small budgets.