Thanks for reviewing our work and constructive feedback.

Q1: Advantage beyond the Nearest Neighbor

Yes, you raised an important point. We present below results of additional experiments to justify the need for additional smoothing offered by the trained model. These experiments demonstrate that the nearest neighbor methods ignore density and thus can succumb to outliers. As described in section 2, we want instances that are both near neighbors (small cost) and plausible. It is difficult to trade off the two terms without a smoothing model. We compare with three variants of nearest neighbor methods:

1. NNR: We return the nearest example in the training data which receives the desired label from h, i.e $h(x_i) > 0.5$
2. NNR($\gamma=0.7$): We increase the threshold to 0.7 instead, $h(x_i) > \gamma$,
3. NNR($\gamma=0.7,y=1$): We return the nearest neighbor from $\mathcal{D}^+$ as described in section . $h(x_i) > \gamma, y_i=1$

We note that the last variant corresponds to our method if we set $\lambda = \infty$ in eq 1 but without learning any model.

Best is highlighted as bold and second best is italicized.

We see that NNR has poorer validity and LOF on Adults Income and HELOC datasets. Upon inspection we also note that there is a tendency to return the same recourse instances for different negative instances. GenRe, while learning a smoothly-mixed version

Q2: Interpretability Issue

As noted by previous work(Looveren(2021)), it is difficult to define interpretability in practice and they propose that recourse/explanations are interpretable if they are close to the desired data distribution. In our work, we return recourse instances that are plausible in the positive class, and thus are expected to be more interpretable than those returned by the search-based methods where plausibility is not easily enforced.

Clarification regarding LOF

Yes, by definition, LOF is higher for an outlier point but in the sklearn API, LocalOutlierFactor predict function returns -1 refers to outliers and 1 refers to inlier. We normalize it to 0,1 - effectively making the reported metric as a fraction of recourse instances classified as inliers.

[Looveren 2021] Van Looveren, A., & Klaise, J. (2021, September). Interpretable counterfactual explanations guided by prototypes. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 650-665). Cham: Springer International Publishing.

Dear Reviewer SyMJ,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications.

Thank you very much for your time!

Q1.a: Dataset Diversity

We have tested our method on diverse standard benchmark datasets widely used in recourse literature. Adults Income has 14 features, COMPAS has 7, HELOC has 21 features. We provide more details about the datasets in Appendix B.1

• We have diversity in terms of class imbalance: Adults Income, COMPAS are imbalanced and HELOC is almost perfectly balanced
• Both Adults Income and COMPAS have a mix of categorical and continuous variables.

Q1.b: Different types of Constraints

We use $\ell_1$ norm as the cost function, which encourages the output to be sparse and we also account for constraints such as immutable attributes like age and gender. We also note that this is one particular choice, different cost function and more constraints can be easily incorporated in our framework

Q2: GenRe in a pretrained model

We can use pre trained diffusion models to sample from the target distribution using a guidance method during inference. We ran an experiment using currently the best performing diffusion model for tabular data Tabsyn (Zhang(2024)) and constrained it with a state-of-the-art derivative-free guidance method SVDD(Li(2024)).

Q3: How does it handle biased or limited datasets?

Our sampling strategy ensures that we only show examples from the desired recourse distribution however imbalanced the label distribution is. As observed from results in Table 2, our method performs well on imbalanced datasets such as Adults Income and COMPAS

Q4: Interpretability

As noted by previous work(Looveren(2021)), it is difficult to define interpretability in practice and they propose that recourse/explanations are interpretable if they are close to the desired data distribution. In our work, we return recourse instances that are plausible in the positive class, and thus are expected to be more interpretable than those returned by the search-based methods where plausibility is not easily enforced. LOF metric reported in our paper serves as a measure to the notion of interpretablity by Looveren(2021)

[Looveren 2021] Van Looveren, A., & Klaise, J. (2021, September). Interpretable counterfactual explanations guided by prototypes. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 650-665). Cham: Springer International Publishing.

[Li(2024)] Li, Xiner, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani et al. "Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding." arXiv preprint arXiv:2408.08252 (2024).

[Zhang(2024)] Zhang, H., Zhang, J., Srinivasan, B., Shen, Z., Qin, X., Faloutsos, C., Rangwala, H., & Karypis, G. (2024). Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. In The twelfth International Conference on Learning Representations.

Dear Reviewer sL8f,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications.

Thank you very much for your time!

Q1: Necessity of $V(x)$

Yes you are correct, on ignoring the $V(x)$ part, our formulation converts recourse into a cost-aware sampling problem. Proposing such a formulation is one of our contributions, in contrast to the predominant narrative in the recourse literature of solving an optimization problem during inference to search a recourse instance. The second part of our contribution is harnessing the training data to create instance pairs for implementing the conditional generative model. These two contributions are agnostic to the specific generative model used. We were able to engineer the auto-regressive model for tabular data generation since our motivating applications were all from tabular data. But we also show results with a diffusion model below.

$V(x)$ also has an important role because training labels may be noisy, and this is particularly true for points that are close to the negative instance that seeks recourse. Without $V(x)$, $\exp (−λC(x, x'))P (x|y^+)$ may assign greater mass to these noisy labels simply because they are close to $x'$. A good way to visualize this effect is to think of each class distribution as Gaussian with some overlap. If training examples are sampled from each class independently, the points from $y^+$ class in the overlapping negative region will have low cost. From a Bayesian perspective, we want to sample from $P(x|y, P(y|x) > 0.5)$ instead of $P(x|y)$. Expanding this out we see that it is the normalized form of $P(x|y)V(x)$.

Q2: Conditional Generation using Guidance

We ran an experiment using currently the best performing diffusion model for tabular data Tabsyn (Zhang(2024)) and constrained it with a state-of-the-art derivative-free guidance method SVDD(Li(2024)). We present the results below. We train diffusion models only on $D^+$ as described in section 4.1.

As a semi-ideal, time-inefficient baseline, we implement Tabsyn+Q, in which we sample an dataset Dsyn, which is of the same size as the original dataset. The purpose of this baseline is to verify the generation quality of this diffusion model for our task. For a given negative instance x-, we define the empirical distribution Q on Dsyn as described in Eq. 6. Following our method, we sample 10 recourse instances and return the one with the highest confidence from classifier h.

Tabsyn+Q performs similarly to GenRe on HELOC but performs worse on datasets Adults Income and COMPAS, struggling on the LOF metric. This experiment also sets a skyline that any guidance method should achieve on this diffusion model.

We see that on two of the three datasets the diffusion based generator does not perform well, both in terms of cost and plausibility (low LOF score). We also observed that, in many instances, the model was not able to produce recourse instances which satisfy the immutability constraints.
Having said that, the main crux of our contribution is not the specific generative model used, but the idea of converting recourse into a conditional generation task, and our method of creating instance pairs for training the conditional generative model. It may be possible to train a custom conditional diffusion model in place of our auto-regressive model. However, for these tabular data, auto-regressive models seemed adequate.

[Li(2024)] Li, Xiner et al. "Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding." arXiv preprint arXiv:2408.08252 (2024).

[Zhang(2024)] Zhang, H. et al. (2024). Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. In The twelfth International Conference on Learning Representations.

Dear Reviewer m8hr,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications.

Thank you very much for your time!

Experiments

Efficiency Analysis

• Model parameter count comparison: | #params | Adult Income | COMPAS | HELOC | | :--- | :---| :---| :--- | | CRUDS | 1790260 |1765672|1823044 | | CCHVAE | 574620 |591956 | 591084 | | GenRE | 293608 | 286564 | 303112 |

All methods use the same architecture across all the datasets, parameter counts differ due to differences in data dimensionality. It is notable that in GenRe we have almost half the number of parameters compared to other model-based baselines. It might be possible to achieve the same performance with even fewer parameters but we did not tune hyperparameters to optimize for model size.

• Training and Inference: We provide comparisons with same batch and same number of epochs. During inference, our method has a complexity of $\mathcal{O}(1)$ whereas most baselines perform search during inference and thus can take longer. Below we provide inference time for one method from each class of methods. | Dataset | Adults Income | | | COMPAS | | | HELOC | | | | :--- | :-------: | :----| :----| :----: | :----| :----| :------: | :---- | :--- | | Metric | Training(s) | Inference(s) | Score | Training(s) | Inference(s) | Score | Training(s) | Inference(s) | Score | | DICE | - | 16.97 | 1.33| - | 51.29 | 0.99 | - | 59.11 | 1.29 | | ROAR | - | 1,041.16 | 0.19 | - | 715.97 | 0.19 | - | 673.41 | 0.19 | | CCHVAE | 776.09 | 0.95 | 0.84 | 86.1 | 45.89 | 0.64 | 168.56 | 527.32 | 0.02 | | Tabsyn+SVDD | 11513.9 | 391.79 | 1.63 | 5,304.78 | 250.65 | 1.49 | 6,966.80 | 560.96 | 1.84 | | GenRE | 8464.71 | 24.52 | 1.93 | 1513.9 | 7.49 | 1.89 | 1913.43 | 46.65 | 1.9 |

Note that in our implementation of GenRe, we have a fair amount of CPU processing during training when we sample from $Q$(eq 6). This can be eliminated by substituting with sampling routines on GPU available in libraries like torch.

Ablations

We ran an experiment using currently the best performing diffusion model for tabular data Tabsyn (Zhang(2024)) and constrained it with a state-of-the-art derivative-free guidance method SVDD(Li(2024)). We present the results below.

We see that on two of the three datasets the diffusion based generator does not perform well, both in terms of cost and plausibility (low LOF score). We also observed that, in many instances, the model was not able to produce recourse instances which satisfied the immutable constraints.

Having said that, the main crux of our contribution is not the specific generative model used, but the idea of converting recourse into a conditional generation task, and our method of creating instance pairs for training the conditional generative model. It may be possible to train a custom conditional diffusion model in place of our auto-regressive model. However, for these tabular data, auto-regressive models seemed adequate.

[Hamman(2023)] Faisal Hamman, Erfaun Noorani, Saumitra Mishra, Daniele Magazzeni, and Sanghamitra Dutta. Robust counterfactual explanations for neural networks with probabilistic guarantees. In the International Conference on Machine Learning, pp. 12351–12367. PMLR, 2023.

[Black(2021)] Emily Black, Zifan Wang, Matt Fredrikson, and Anupam Datta. Consistent counterfactuals for deep models. arXiv preprint arXiv:2110.03109, 2021

[Li(2024)] Li, Xiner, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani et al. "Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding." arXiv preprint arXiv:2408.08252 (2024).

[Zhang(2024)] Zhang, H., Zhang, J., Srinivasan, B., Shen, Z., Qin, X., Faloutsos, C., Rangwala, H., & Karypis, G. (2024). Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. In The twelfth International Conference on Learning Representations.

Clarity

Thanks a lot for pointing out notational inconsistencies, we have fixed these in the revised pdf where changes are highlighted in blue. Some other clarifications: We had provided the cost function details in the experiments sections. We have now also added that information in the problem formulation section. We have added a comment explaining projectCategoricals function in Algorithm 2. After step 4, we obtain $p_j$, a categorical distribution over bin indices. To sample $k$, we can directly sample from this distribution. In our implementation, we use torch.multinomial function to do so.

Are you using a random forest or an MLP?

We train a random forest classifier with calibrated probabilities on the original dataset to serve as the gold classifier since for recourse tasks we need to establish validity of generated examples. From this gold classifier, we sample labels for each example in our dataset. These new sampled labels serve as training data for all downstream models, including the MLP classifier. All methods only access the MLP classifier to generate recourse instances. Previous work on robustness under model shifts also use similar setup [Hamman(2023), Black(2021)]. In section B.2, we provide the performance metrics for both the classifiers.

For the baselines, how are the final checkpoints selected?

We use complete implementations from CARLA library(https://github.com/carla-recourse), and the final checkpoints used are from the end of the training run after a fixed number of epochs.

why do you select variance equal to zero (line 376)?

Our method provides a distribution over recourse instances. To compare with the baselines algorithms on the standard benchmark, we used the mode of the sampled bin. This choice corresponds to $\sigma$ = 0. In practice, using our generative approach a user will have the flexibility of inspecting multiple recourse instances by sampling with a non-zero sigma. In the appendix we include results with different temperatures $\tau${5.0,10.0,15.0} and $\sigma${2e-7, 2e-6,..., 2e-2}, below we provide a subset of these results. From this table, observe that GenRe generates diverse recourse instances (as seen by the varying cost values) while being valid across different $\tau$ and $\sigma$ combinations.

Purpose of "TopK"

TopK refers to truncation of $Q$ to top $K$ entries. This is done for efficiency reasons. TopK takes a categorical distribution over $N$ objects and returns a re-normalised distribution over $K$ objects with highest probabilities.

Which dimensions of $x^-$ to freeze? How is the loss function formulated?

This is provided by the dataset. For example, in Adults Income dataset, immutable features are race and sex. Loss function is implemented as cross entropy over bin probabilities

Generative model from scratch, or are you using pre-trained weights?

We train a new generative model from scratch by sampling from $Q$ defined in Eq. 6.

Dear Reviewer 92zH,

Thank you for your constructive feedback on our paper. With the ICLR public discussion phase ending shortly, we wanted to confirm whether our responses have sufficiently addressed your concerns. If there are any remaining issues, we would be happy to provide additional clarifications.

Thank you very much for your time!

Dear Authors,

For the choice of standard deviation (Table 11 in updated pdf), it seems like the best results are given for smaller values of $\sigma$ (moving closer to the deterministic case $\sigma=0$). For the additional evaluations, the authors provided different $\sigma$ values, but their range is really small. Even the largest one will result in a variance of $1e-4$.

Question 1

How different/diverse can the sampled $\mathbf{x}'$ be for such a small standard deviation or variance? Does your model generate really diverse solutions for such a choice of $\sigma$? Or is it more the case that you find one good $\mathbf{x}'$ for each $\mathbf{x}^-$?

Question 2

Can you provide additional evaluations for the above table as you increase the variance or standard deviation until it reaches a value such as 1? Also, can you include a metric to measure the diversity of the sampled $\mathbf{x}'$? How different are these solutions become as we increase the variance? Can we say something about the trade-off between distance from the mean and recourse allocation quality of the sampled $\mathbf{x}'$?