Language Models Can Help to Learn High-Performing Cost Functions for Recourse

We thank the reviewer for noting our work is contributing to an important goal under-explored in the literature, and for your suggestions which have improved the paper, we now respond to all your questions and concerns.

---

Why not use Cohan’s kappa: Thank you for the suggestion, but Cohen’s Kappa would not work here, it is used for assessing agreement between 2 raters. Here we have 30 Human raters (and “30 LLM raters”) and are comparing two groups representing two distributions, so the Chi-Squared test of independence was an appropriate choice.

---

What exact test was used and how to interpret p values? We used the Chi-square test of independence (i.e., scipy.stats.chi2_contingency), which is used to compare 2 or more groups. Apologies for the lack of clarity, we are hoping that the p value will be > 0.05, because we want to demonstrate that the distributions of LLM and human are not significantly different. We state this clearly in Figure 2’s caption now. (ln 264)

---

How would the results change if scores where only the accepted pairs by the LLM are used were obtained? If we remove all instances where the LLM was uncertain we would lose 6 questions where the LLM was 100% uncertain. The remaining 12 would have some with uncertain LLM responses. Of these, 8/12 have statistically similar distributions, which is proportionally similar to the 13/18 reported in the paper.

---

Selection of examples: The samples were selected according to their mean values in the dataset and perturbed by standard deviations. This guaranteed they would be reasonable feature values. The questions were purposefully chosen to be very difficult, which is evidenced by the uncertainty shared in the LLM and humans (49% uncertain ln 270). To illustrate this we added additional analysis in Appendix F showing the uncertainty in users and LLM were both equally high and strongly correlated between questions. This shows the questions were not obvious. In total we are evaluating three desiderata across 18 questions, which is not low by the standards of recent recourse publications which sometimes just evaluate as low as six questions with humans [1]. We added the uncertainty results in Appendix F and mention them in Section 4.1

---

What is the advantage of using the LLM: Our LLM method is the only method which can model feature dependencies naturally, this is a significant advantage on all prior work. If we compare it to other methods like Ustun et al. (2019) they will fail on this by definition, so it would be a trivial comparison. Note we already showed this in the human study when the LLM correlated with human judgement on Desideratum 3, and we have added additional computational test in Section 4.3 to show this now more explicitly.

---

The LLM seems to fail to inherently model dependencies. Please see above response. The LLM does naturally model these dependencies, the original results in Table 1 were us purposefully choosing dependencies which were not modelled to show we could fine-tune them into the cost function. We have added computational dependency tests in Section 4.3, and shown how dependencies can be fine-tuned in Section 4.4. with our custom prompting scheme.

---

Apply the cost function in popular recourse frameworks: We have replaced our L1 baseline in Section 4.3 with (1) an SGD-based method by Watchter et al. [3] and (2) a data driven method by Smyth & Keane [2] to test our models, both highly cited and contrasting methods.

---

Can the authors specify more details about the independence test used? We used the Chi-square test of independence for each question to determine if there is a significant difference in the proportion of each choice between the two groups (30 LLM responses and 30 human responses in 18 questions, 540 per group).

---

What are the user study scores without the undecisive cases? See above responses.

---

Why doesn't the LLM seem to have implicit knowledge about feature dependencies in Table 1

We purposefully chose dependencies which did not perform well to illustrate how fine-tuning can help in Table 1. Please note the human studied showed how the LLM picked up on natural dependencies in Desideratum 3 (6/6 had statistically similar distributions or same mode). We have added Section 4.3 which is a new dependency experiment to show where the model can reason about dependencies.

---

I would appreciate additional discussion on how the learned cost function can be integrated in recourse frameworks

We have shown an example in our case study now for two published methods [2,3], thanks for the suggestion.

---

[1] Kenny E, Huang W. The Utility of “Even if” semifactual explanation to optimise positive outcomes

[2] Keane MT, Smyth B. Good counterfactuals and where to find them.

[3] Wachter S, Mittelstadt B, Russell C. Counterfactual explanations without opening the black box

Thank you for your reply which has clarified the following of my points:

• Cohen's Kappa: Yes, the classical kappa only works for 2 raters, but there are extensions to multiple dimensions, e.g., https://en.wikipedia.org/wiki/Fleiss%27_kappa (but that is a minor point for me).

• Chi2-test I understand the type of test that was performed now which should be fine. Maybe mention more explicitly that you use the independence test not for independence testing but for testing a if two samples are from the same distribution. That was what confused me, when reading the draft.

• Selection of non-obvious examples. Thanks for providing the plot in Appendix F which shows that the examples selected were non-trivial.

• Modeling dependencies. I am still a bit confuesed about the selection of dependencies for which some of them are picked up and some are not, but I like the new experiment in Section 4.3. I definitely strengthens this aspect of the work. Some discussion about how the dependencies in Table 1 were chosen and which dependencies the LLM usually does not pick up would be appreciated.

• Baselines. The results in the Case study show that the new cost function can be used with some recourse techniques (of course simple ones), but this is an improvement.

My following concerns remain:

• LLM Uncertainty rate in the study is too high. I agree with the other reviewers, that the high uncertainty rate is problematic. As the authors point out it reduces the small number of 18 examples further to 12 samples for which there is an actual signal.

I thank the authors for their thorough rebuttal, which has addressed most of my concerns. I have updated my score from 5->6.

We thank the reviewer for noting our work is interesting and timely, and for your suggestions which have improved the paper, we now respond to all your questions and concerns.

---

Make the LLM contribution rock solid: We regret not communicating this well. The LLM is necessary to automate the labelling of pairwise comparisons at scale and model feature dependencies automatically. No prior approach can do this. Note that our human study already demonstrated this as all six Desideratum 3 (i.e., the feature dependency questions) materials achieved either statistically similar distributions or the same modal response, but we added more computational evaluation in Section 4.3 to make this even stronger.

---

Human-LLM Comparison Issues: First, in total there are 16 possible feature permutations per dataset, or 48 in total across three, and an additional 2 per dataset when considering mutating the same numerical feature at different ranges as a comparison. We sampled 18, which is 33% of all possible permutations and quite reasonable. Second, our number of participants is double what is observed in some recent recourse studies [1]. Third, LLMs are shown to correlate with large user studies [2], and as our group did correlate with the LLM here, it is reasonable to infer they are generally representative of the population. The six questions were intentionally chosen to use mean values in the dataset, mutate by standard deviations, and be non-trivial, which is evidenced by the uncertainty of the humans and LLMs (see new Appendix F), our questions were all selected before we saw any results, and were not cherry picked from a larger pool.

---

Prove that 49% of the answers are also given randomly by the users: As an aside, this does not mean 9 questions were randomly answered, it means that 49% of the responses were random, which were spread out among questions. We have included extra results on the human responses when they gauged how “far apart” recourse pairs are (i.e., the second question in Figure 6), which represents their uncertainty. We took the mean responses across these questions and the uncertainty of the LLM which was quantified by what percentage of responses were uncertain for each question. Both were then plotted in Appendix F and Figure 7. They show a strong correlation (Person r > 0.5; p < 0.02), providing evidence that not only are close to 49% of human responses also random, but they are equivalently random for the exact same questions. We mention this in Section 4.1 results paragraph (ln 271).

---

Evaluation is hard to understand, what is the ground truth, why not generate human ground truth? The human study showed our standard prompt (i.e., B in Figure 1) did well at approximating human labels (15/18). However there was misalignments, the point of the custom prompt (and our previous computational tests - now Section 4.4) was to fine-tune the labeling process to e.g. reorder feature weightings to fix minor issues in the cost function. We have added the ground truths to the paper in Appendix H. We agree it’s important to compare to a human ground truth, but we did this in the human study, so this is not a valid concern.

---

Redo Section 4 with the desiderata: Agreed, we have written Section 4.4 (Fine Tuning Experiments) to do this, and iterate the desiderata 1-4 to do this as you suggest.

---

Why did you not use a specific algorithm to find recourses: We have replaced this section to use two diverse, highly cited algorithms, thanks for the suggestion.

---

What is the point of pairing the recourses in a graph (Section 3.2)? Having a connected graph improves the reliability of our cost estimates, because it ensures the costs of all recourses can be estimated on a common scale.

---

What is the baseline used to evaluate the approaches in Section 4.2? Yes, we defined the ground truth in our code previously, but neglected to add it to the paper, apologies for this, we have added it to Appendix H.

---

Why do you pick only different recourses to compare against the L1 norm and the trained MLP (Section 4.3)? ...By picking C and A, you might say that the MLP finds cheaper options than the L1 norm).

We have updated this experiment with 6000 comparisons between methods, all validated and compared on exactly the same data. Note we are not concerned with the actual cost in this test, only the recourse ultimately chosen and the actions it results in.

---

[1] Kenny E, Huang W. The Utility of “Even if” semifactual explanation to optimise positive outcomes. Advances in Neural Information Processing Systems. 2024 Feb 13;36.

[2] De Bona FB, Dominici G, Miller T, Langheinrich M, Gjoreski M. Evaluating Explanations Through LLMs: Beyond Traditional User Studies. arXiv preprint arXiv:2410.17781. 2024 Oct 23.

Many thanks for your response, we have addressed the evaluation and presentation concerns of other reviewers as you mention, and now we address your remaining concerns about the human study. Please note that in order to respond adequately to your concerns, we had to split this into two parts (this is part 1).

---

[LLMs unsure 49% of the times] Out of the 18 questions, almost half have random answers, since the authors replace them (line 270). Again, this makes the analysis weaker given the limited sample size.

We appreciate your concern, and have experimented with new prompting approaches. We updated our standard prompt to include a high-level overview of the Desiderata in Section 2, which generalizes across all datasets (see above general response to all reviewers, or Appendix D, standard prompt, red highlight). This has brought the LLM uncertainty from 49% →13.8% on average in the human study., with no questions being 100% random. The alignment results are now even better (17/18 have p > 0.05 or the same mode), here are the full results for your convenience.

---

[Appendix F] I believe the plot in Figure 7 does not show any uncertainty correlation between humans and LLMs (I would have expected points to align on the red line), strengthening the points made in the previous two comments.

Given the amount of variability and noise in real-world data (and the space of possible values), it is often the case that points will not align on the red line for modest samples as we have (18 questions). Please note the objective statistical test Person r was 0.55 (p < 0.02) – [now updated to r=0.48 and p=0.04 with the new data collected from the LLM], which is considered a strong-moderate correlation. In either case, we have updated this plot now with the new data, and we feel it is a stronger visual correlation (as the objective statistical measurements again verify), one point even lies directly on the red line as the reviewer wished to see to be more convinced (although now it is perhaps less relevant since the uncertainty in the LLM has dropped so much).

We thank the reviewer for vouching for the acceptance of our work, and for your suggestions which have improved the paper, we now respond to all your questions and concerns.

---

High uncertainty in LLM Human Study Responses: Thanks, we purposefully chose very difficult questions for the user study which were hard to evaluate, which is why the uncertainty is high. In our main computational experiments (now Section 4.4), the LLM is unsure about the answer (i.e., gives equal cost) 15%, 23%, and 20%, of the time (using the standard prompting scheme) on adult, HELOC, and German Credit, respectively, which is much lower than the user study uncertainty, illustrating this point for us. As an aside, this dropped further to 10%, 5%, and 11% when fine-tuning using the custom prompt schemes. We explicitly mention this in Footnote 4 now, and add additional analyis about the uncertainity between LLM and humans in Appendix F.

---

...the use of a “custom prompt” seems unrealistic as it assumes an oracle can encode information about recourse costs apriori... beginning with existing cost functions only to later recover them may not be particularly meaningful...

Allow us to clarify, there has been a large misunderstanding. The purpose of the custom prompt in Section 4.2 is not to hard-code the cost function (this would be pointless as you point out). It is to show how the original cost function could be “fine-tuned” with small prompt additions. For example, to reorder feature importance, add/exaggerate a single dependency, or manipulate a continuous feature’s cost spectrum, or fairness attributes (how you define them).

Revision: We have re-written some of Section 4 (the evaluation) to be clearer about all this by e.g. titling Section 4.4 “Fine Tuning Experiments”. Moreover, we added additional experiments showing how the LLM naturally picks up feature dependencies, which is a significant advancement over prior work.

---

Similarly, the accuracy comparison in Table 1 - showing how well LLM responses are modeled by either a tree-based model or an MLP - is difficult to interpret in terms of the paper’s objective... the choice of model here should be agnostic to the objective of the paper. The experiments show that pairwise comparisons can be learned with reasonable accuracy, which I think is enough to justify the paper’s merit, but the relative accuracy comparisons confused me - perhaps I missed the underlying point?

We considered two model classes because they have distinct practical advantages. Only MLPs are differentiable (which is needed by many recourse methods), and the tree models have more ability to be transparent, which is important in domains such as medicine or finance where interpretability is a strong requirement, so the comparison is quite important. We are using accuracy to verify how well they are imitating the original LLM labeling process, which is important as we are essentially trying to distill the LLM’s knowledge into smaller cost functions.

Revision: We explain why we compared MLP and tree model accuracy better (ln 303 and Section 4.5).

---

I'm unable to understand the comparisons between the standard and custom prompts, and the bearing they have on the use of LLMs to learn recourse cost functions. Perhaps a search over LLM prompts (without including the desired answer directly) to best match human intuition would be more valuable. Similarly, the accuracy comparison reported in table 1 needs better justification: what does the comparison indicate - are we just checking to ensure that LLM output labels can indeed be learned by the models?

Apologies for the confusion, the standard prompt was used in the human study, and this showed strong results (15/18 mostly aligned). However, there was room for improvement, so the custom prompt is there to fine-tune the LLM responses to match preferences in a specific domain (with minimal instructions). We did not see the need to iterate more LLM prompt variations as we risk overfitting to our human study results, and the standard prompt (i.e., no bias injected in the prompt from us) already did quite well, and we anticipate this will just get better with later LLM models. See above response for accuracy concerns in Table 1.

Revision: We titled Section 4.4. “Fine Tuning Experiments” and made the purpose of the custom prompt clear in Section 4.2 (ln 296).

High uncertainty in LLM Human Study Responses - Upon clarification, it is interesting that the human uncertainties align with the LLM uncertainties. I also see more clearly the intentions behind the use of the MLP and the decision trees.

My question about the unrealistic custom prompt remains. There seems to be a misunderstanding here, as per the authors, but I am still unsure my concerns have been addressed. I think I follow this statement: The purpose of the custom prompt in Section 4.2 is not to hard-code the cost function (this would be pointless as you point out).. However the following, to me, seems to contradict it: the original cost function could be “fine-tuned” with small prompt additions. For example, to reorder feature importance, add/exaggerate a single dependency, or manipulate a continuous feature’s cost spectrum, or fairness attributes (how you define them). Isn't this description doing exactly what my initial question was about - inserting the "answer" in the prompt, and having the LLM recover it in the form of pairwise comparisons used to fit the cost function?

To me, the "fine-tuning" of the prompt by modifying it apriori with properties desired in the cost function, and then observing that the cost function is better attuned to those properties, seems counterproductive. This may not be "hard-coding", but is still essentially leaking information about the cost function to the LLM via the prompt, when the point of the paper (as I understand it) is to elicit the cost function from the internal knowledge of the LLM. If the authors could help clarify if/what i am misunderstanding, I would be like to update my score. Like I mentioned, I think this is a valuable contribution and a unique approach that needs to be shared with the community, but this particular experiment seems to indicate that perhaps LLMs are not indeed a reliable way to learn high-performing cost functions for recourse. If they were, why would we need to include desirable properties about the cost function, known apriori from external domain knowledge, in the prompt?

We thank the reviewer for noting the practical relevance of our work and for your suggestions which have improved the paper, we now respond to all your questions and concerns.

---

Missing Baselines: We did not compare against published baselines because our models already achieved high performance on all of the 4 desiderata (Section 2), including the one about feature dependencies. Hence, because no published baseline considers feature dependencies, it would be a trivial comparison, in which our method would win easily. We make this clearer in what is now Section 4.3.

---

Weak Case-Study: Agreed. We have included two published recourse methods in Section 4.6, and shown how our cost function helps them achieve better recourse.

---

Can you hard-code the cost function, how much does LLM help? You suggestion to use a limited amount of labeled data and hard-coding the function may provide a good cost function, but it could still be quite costly as the problem of defining functions with expert knowledge is quite difficult, can take years, and needs to be constantly updated year-on-year (e.g., see [1]). This is likely why no one has proposed this in recourse, and since LLMs can largely automate the process, it is of enormous practical benefit.

---

Limited Evaluation: Please note we outlined clearly the dimensions along which a cost function should be evaluated as high-performing in Section 2 (which was defined by other researchers) and used it in all of Section 4 pages 4-9 showing positive results (over half the paper is dedicated to evaluation). In direct response to your concern about dependencies, we asked Claude Sonnet 3.5 to define a list of relevant dependencies in each dataset and evaluated this in Section 4.3. as a new addition to the paper.

---

Experiments not motivated in any earlier sections of the paper: We regret the motivation for these experiments were lost in the writing. Allow us to explain. The human study showed positive results, but that there are occasionally discrepancies between LLM and human judgment of cost. The purpose of the custom prompt and Table 1 and Figure 3 (now Fig 4/5) is to show how we can fine-tune it to fix this. For example, we saw the LLM judged certain feature-weightings to be different to humans in Figure 2 (left: Desideratum 1), so our custom prompt showed that this could be fine-tuned to correct it (see Figure 4 (middle) in revised paper). Similar results were shown for Desideratum 2-4. We re-wrote all of Section 4. to make this clearer by e.g. titling Section 4.4 “Fine Tuning Experiments”.

---

Can we specifiy constraints during learning? We could, but this goes back to the problem of hand-defining models [1] discussed above. Our constraints are not so rigid, we mostly use the LLM’s natural labels and only occasionally modify it in our custom prompt to fine-tune the cost function, not define it from scratch, that would not scale well.

---

Adult Data Contamination: This is an interesting point. First, because the recourse points are synthetically generated by us, the LLM will not have seen them during its training. Second, asking the LLM to compare recourse costs is a very different problem from predicting labels (i.e. the typical use of the datasets), and we wouldn’t expect memorization of the latter task to provide any benefit for the former. It is therefore unlikely that pre-training contamination is an issue. We point this out in the main paper and cite the work above (ln 214).

---

Missing References: Thanks for the suggestion, we have done as you ask (ln 504)

---

Why do you use the Adult dataset? We opted to use canonical datasets from the recourse literature to maximize readers’ intuition about the contexts and associated recourse cost considerations. Since the Folktables dataset you suggested is a superset of the data in Adult (drawn from the same underlying census information), we expect the results of our recourse experiments would have been virtually identical.

---

Use of Latin and Phrasing Suggestions: We have replaced “ad libitum” with “at liberty”, and rephrased as you illustrated, thanks for the suggestion.

---

[1] Ruelle E, Hennessy D, Delaby L. Development of the Moorepark St Gilles grass growth model (MoSt GG model): A predictive model for grass growth for pasture based systems. European journal of agronomy. 2018 Sep 1;99:80-91.

We thank the reviewer for their response, and address their remaining concerns now.

Most of the reviewers seem to agree that there are problems with the evaluation in this paper, and I don't believe this is due to misunderstandings. In their response, the authors have already taken steps to improve the evaluation, specifically Table 1 and the revised Sections 4.3 and 4.6. I appreciate this. Table 1 goes into a good direction, I would like to see this evaluation for all the datasets. However, I also tend to agree with Reviewer umaS that there are still gaps in the evaluation: Figure 7 in Supplement F is not convincing at all.

Thanks for the kind words. To address your requests, we have run the case-study on all datasets now, with the extra results now in Appendix J.

We have also addressed the evaluation concerns of Reviewer umasS you referenced by altering our standard prompt to include a high-level overview of the desiderata (see our general response to all reviewers above, or Appendix D, standard prompt, red highlighted text), the uncertainty in the human study has dropped from 49% → 13.8%. Figure 7 has changed now also in accordance with this, and we feel it now looks more convincing (although now it is perhaps less relevant since the uncertainty in the LLM has dropped so much). Table 2 and Appendix F describe all the results in more detail, again, the objective statistical measurements showed a significant correlation.

---

With regard to the point of data contamination, I appreciate that the authors are willing to incorporate this into their paper. To clarify the point from my review, it is not only about the fact that the raw datasets are memorized. It is also about the fact that hundreds of research papers are written about counterfactual explanations on these highly popular datasets, and these are very likely in the training data of the LLM. As such, while it is very hard to say how precisely the LLM becomes able to "naturally model dependencies", it does not seem absurd to believe that dependency modelling is much better for datasets and problems that the model has been heavily exposed to during pre-training (see, for example, https://arxiv.org/abs/2309.13638). This is while it is important to at least make some attempts to study the performance of the LLM for problems that are unlikely to be part of the pre-training data.

Thanks for the further clarification, we understand now. To sincerely address this, we have generated a dataset from scratch which does not exist on the internet with scientifically known causal dependencies in medicine and added this to Section 4.3 (details in Appendix K). Running our standard prompting scheme on this we found that the LLM does model dependencies here with 90% + accuracy. Interestingly, we found that in order to model the causal dependencies correctly, the LLM had to be prompted with (1) chain-of-thought, and (2) our high-level desiderata (detailed in the general response to all reviewers). Thank you a lot for this suggestion, we feel it has massively strengthened the paper.

---

Overall, I still believe that this paper proposes an interesting method that could be of interest to the community. In my view, the evaluation would need to be somewhat more rigorous for a venue like ICLR.

Hopefully our new evaluations are now rigorous enough for the reviewer.

All the best,

The Authors

First and foremost, we want to sincerely thank the reviewers for taking the time to give their first round of responses, which have dramatically improved the paper. As before, we have taken seriously all your concerns and directly addressed them. All changes are again highlighted in red in the revised manuscript for your ease of reading.

Main Change: To summarize the main change, taking to the challenge of most reviewers, we experimented with more creative prompting schemes to resolve the uncertainty concern in our human study (which was the main concern across reviews). Our discovery was that by adding a high-level description of our desiderata in Section 2 to the standard prompting scheme, this had the effect of lowering the uncertainty of the LLM in our human study (and improved its ability to reason about casual dependencies). Note this change to the standard prompt is general, it can be applied to any dataset without modification. Specifically, this is the addition we made to the standard prompt (corresponding to our 4 desiderata), which you can also see in Appendix D:

Remember the following 4 rules and use them in your decision:

1. Some features are naturally harder to change than others, use this logic.
2. For numerical features, the difficulty of changing them can often depend on their starting values.
3. Apart from the mutated features, consider the other features which are different between Alex and Jaden, and how this may affect difficulty.
4. Do not ever use demographic features when considering the difficulty of mutating other features.

To summarize the main changes:

1. In response to the concerns of reviewers 8sk6, umaS, and xrVn, we modified the standard prompting scheme to have a general description of the desiderata in Section 2 (see above or Appendix D). We found this dropped the uncertainty of the LLM in the human study from 49% → 13.8%, with NO questions being 100% uncertain. Moreover, it is now aligned with humans in 17/18 questions, higher than the 15/18 before. We have the full results in Table 2 and have updated Figure 7 and Figure 2 accordingly. Please note this is a general modification to the standard prompting scheme, it still can be applied to any dataset without alterations.
2. In response to Reviewer xrVn, we have extended the case study to consider all three datasets, results are in Appendix J.
3. In response to Reviewer xrVn, we have added another experiment on synthetic data in Section 4.3 (which does not exist anywhere on the internet, and hence not in the LLM’s pre-training data). The data shows that the LLM can model known scientific causal dependencies on unseen data with > 90% accuracy, helping to verify that the LLM’s ability to model dependencies is not limited to popular counterfactual/recourse generation datasets online.

We have updated the paper accordingly. Thanks again for all your help to improve this paper, we would be very excited to share these results with the community.

Best wishes,

The Authors