Explaining recommendation systems through contrapositive perturbations

Thank you! Thank you very much for your insightful comment regarding the connection between the contrapositive logic in our Contra+ method and Judea Pearl’s concepts of the probability of necessary and sufficient causation. This is indeed a very interesting connection to causality that we have tried to touch upon in our discussion on connecting our work to Counterfactual backtracking [1]. We appreciate this opportunity to clarify and elaborate on the novel aspects of our approach, particularly on the established frameworks of causal inference.

1. Connection Between Contra+ and Pearl’s Framework of Necessity and Sufficiency

The Contra+ method, while distinct in its application, shares a conceptual connection with Judea Pearl’s framework of necessity and sufficiency in causal inference. This connection lies in the underlying logic that both frameworks employ.

• Logical Foundations (our approach): In propositional logic, necessity and sufficiency are expressed as x → y (sufficiency) and y → x (necessity). The Contra+ method utilises contrapositive logic, where the inversion of these implications (-y → -x for sufficiency and -x → -y for necessity) is central to its explanatory mechanism. For instance, if a recommendation system suggests a movie based on a user's preference for a genre, our method uses contrapositive logic to infer non-preferences from non-recommendations, akin to the logical structure of sufficiency.

• Probabilistic causal Interpretation: Pearl’s work extends these concepts into the probabilistic and causal realm, assessing the likelihood of causation. While Contra+ does not directly engage with probabilistic causation, its logical structure echoes the principles Pearl discusses. The method’s efficiency in explanation generation can be seen as a practical application of these ideas, albeit in a different context and using different assumptions as we will discuss in the following.

2. Delineating the Difference Between Our Method and Pearl’s Framework While there are conceptual similarities, there are key differences in the application and objectives of our Contra+ method and Pearl’s causal inference framework.

• Focus on Explanation vs. Causal Analysis: Pearl’s framework is deeply rooted in understanding and quantifying causal relationships in complex systems, where one has to assume a causal structure of the problem. In contrast, our Contra+ method is primarily focused on enhancing the interpretability and efficiency of recommendations in factorization models without any causal assumptions. Our approach is about using logical inferences to provide clear and computationally feasible explanations for recommendations, not to establish causal relationships as this would require causal assumptions such as the graph (not clear in our case) etc. Nevertheless, we agree that this future research direction of bridging this gap between these two frameworks is interesting and would allow us to potentially provide theoretical guarantees.

3. Future Work Outside the Scope of This Paper Nevertheless, there are several avenues for future research that, while outside the scope of this paper, could enrich the Contra+ method.

• Integrating Causal Inference Principles: One promising direction is the exploration of how causal inference principles can be applied to explaining recommendation systems. This could involve adapting aspects of Pearl’s framework to better understand the causal mechanisms behind user preferences and behaviours, thereby enhancing the depth and robustness of our explanations. We have also, in section 3.3, mentioned a connection to the recent paper on "Counterfactual backtracking" [1] which we believe gives a theoretical understanding of what Contra+ is doing under the hood. However, given that we wish to have a general algorithm without too many causal assumptions, we did not opt to connect them via this route 1-to-1. However, for future work, which is outside the scope of this paper, we will consider making this connection more formal as it would enhance the reliability of the method. This paper primarily focuses on a computationally efficient manner to generate explanations for recommender systems and we have shows on extensive experiments the merits of our proposed method.

In conclusion, while the Contra+ method shares logical underpinnings with Pearl’s framework of necessity and sufficiency, it stands as a distinct approach focused on the specific challenges of explanation in recommendation systems. Future work would aim to explore these theoretical connections further, expanding the method’s capabilities and theoretical foundations.

[1] von Kügelgen, J, Counterfactual Backtracking

First of all, we would like to thank the reviewer for their comprehensive review to improve our paper as well as allowing us an opportunity to clarify any concerns or misunderstandings. We would also like to thank the reviewer for praising our paper for considering an "interesting question (explaining recommendation system) from a contrapositive perspective, which is novel." Here below, we have addressed all your concerns.

W1: The key concern is whether there is another way to get the "explanation". Further, is there infinite number of ways to perturbation the embedding that can achieve the same purpose, i.e., "We did not recommend item i to user u"? In such case, does each way of perturbing embedding correspond to a different h, i.e., "User u would not have interacted with item h"? How can we distinguish merits and drawbacks of each perturbation?

We thank the reviewer for this clarification question. We agree and have acknowledged this observation in our main text of the paper : "We emphasize, that do not claim that we are able to find the one and only explanation, but rather, that we are able to provide a contrapositive explanation which fits our logical statement -B→ -A, which is equivalent to A → B. This is corroborated by our extensive experiments as well." To provide more details, even though there are in theory an infinite number of ways to perturb the representations, in this paper, our main focus is on computationally efficient ways to construct contrapositive explanations. We could use more elaborate ways to construct perturbations, however, given that we are practically oriented, we decided to go with the most natural and efficient way to construct the perturbations, which is our proposed method Contra+. We show the merits of Contra+ on numerous experiments in section 4 and demonstrate that this way of constructing perturbations is effective and even matches the performance of the Influence function in some cases. Note that influence functions, which are computationally much more expensive have only been added for completeness and transparency. Future research would investigate how one could trade off computational efficiency with possibly superior explanations. However, this is outside the scope of this paper as we are mainly focusing on computationally efficient algorithms.

W2: Previous literature like Tan et al. [1], studied cause on a particular aspect, i.e., If the item had been slightly worse on [aspect(s)], then it will not be recommended. This can find the cause on a particular aspect, whereas in this paper, the cause is found on perturbation on all embedding. Is any comments for the difference?

In Tan et al. the authors focus on counterfactual explanations (-A -> -B i.e. if A did happen then B would not happen), which are different to our proposed method, which utilizes the contrapositive (-B -> -A) nature of explanations. However, the reviewer is raising a very interesting point here, whereby one could potentially perturb the embeddings only in a specific direction such that only a given aspect can be affected and subsequently the rating will drop (-B). This perturbation could then be used to determine the contrapositive explanations. However, there are also challenges associated with this approach, such as accurately determining the appropriate direction to push the embeddings in the direction of these aspects. Additionally, it remains uncertain whether this approach would be more effective compared to the perturbation suggested in our paper, which showcased substantial enhancements in our experimental section. Note that in our contrapositive formulation, we only care about -B→ -A, i.e. conditioned on the user not being recommended item j, the user would not have watched item i. Therefore, any method of inducing -B could be used and we leave this interesting avenue for future research as this would be out of the scope of this paper. In summary, we thank the reviewer for their interesting suggestion to incorporate aspect-specific perturbation in the embedding space, a concept that could provide a valuable addition to our paper. However, it is important to acknowledge that our unique contribution lies in the explanation from a different perspective, which is separate from aspect-specific perturbation.

[1] Kaffes et al Model-Agnostic Counterfactual Explanations of Recommendations

[2] Yao et al, Counterfactually Evaluating Explanations in Recommender Systems

[3] Tran et al, Counterfactual Explanations for Neural Recommenders

First of all, we would like to thank the reviewer for their time reviewing our manuscript and giving thought-provoking clarification questions. We would also like to thank the reviewer for praising our paper for being an "interesting explanation method to explain recommendation systems through contrapositive perturbations". In the following, we will address and clarify all the questions raised in the review.

Q1: in Figure (2) and (3), as the number pf experiments increase, in particular at 5, it seems that the contrapositive approach is non-stats sign from other baselines, especially Item Similarity or Influence. Was this the expected behavior?

We thank the reviewer for pointing out this interesting experimental observation in our paper. The fact that the results are similar to influence function and item similarity for ML-100K, when the number of explanations increases to 5 is somewhat expected because the contrapositive and the counterfactual metrics are correlated. This has been shown in the experimental section as well as mentioned at the bottom of section 4, where we state that "Both contrapositive and counterfactual metrics make use of the quantity in the top left corner in Figure 1 in their computation and hence there is a clear correlation between the metrics". Hence, as we increase the number of explanations, quadrant 1 will increase and quadrants 2 and 3 will decrease for the counterfactual and contrapositive metric respectively.

Intuitively, as we increase the number of explanations, we are more likely from the counterfactual perspective, to select a historical item that, if removed, will change the recommendation. Therefore, as we increase the number of explanations, we expect both counterfactual and contrapositive metrics to increase up to a certain point and we observe that at around 5 explanations they both plateau for ML-100K.

Lastly, we want to emphasize that in real application settings (such as social media apps), the goal is to keep the explanations to a minimum to reduce user confusion. Hence, we only added the experiments with 5 explanations as ablation to understand the behaviour and large explanation sizes. Previous works such as [1] only considered explanation sizes of 3.

Q2: in Figure 4, it seems that "Influence" is comparable or have higher M_contra value as "Contrapositive" approach in Dataset ML-100k, is that expected.

We thank the reviewer for this observation and we will add a detailed discussion in the final version of the paper. In particular, we want to first highlight that influence functions have only been added for completeness and transparency, as they are not computationally feasible in large recommender systems. The main focus of our paper is on computationally efficient methods for explanations in the realm of recommender systems. Therefore computational cost of influence functions is not directly comparable, as influence functions require the computation of a Hessian matrix inverse. Nevertheless, for completeness and transparency, we show that even despite these computational differences, our proposed method Contra+ can perform on par with influence functions, not only on the contrapositive but also on counterfactual metric, which is a very promising result. Secondly, in terms of the results being higher or comparable to our proposed method, we would like to highlight that for all the methods that are computationally comparable to our proposed methods (Random, item similarity), we perform significantly better for explanation sizes 1 to 3 for the contrapositive metric and do significantly better on the counterfactual metric across explanation sizes.

We thank the reviewer again for their time reviewing our paper and hope that the above has clarified all of the reviewer's questions. We would appreciate it if the reviewer could increase their score if the above has appropriately answered all your concerns and we are more than happy to answer any remaining questions the reviewer might have.

[1] Yao et al, Counterfactually Evaluating Explanations in Recommender Systems