Explaining Black-box Model Predictions via Two-level Nested Feature Attributions with Consistency Property

Thank you for taking the time to review our paper and for your thoughtful comments. Below, we provide detailed responses to each of your questions.

Just a brainfart thought, why 2 levels? How does the performance of C2FA change as the depth of the nested structure increases (e.g., to 3 or more levels)? Are there theoretical or practical limitations to extending beyond two levels? C3FA can make sense in an NLP context, Word-level to sentence level to paragraph level.

The decision to focus on two levels was driven by the practical applications we were targeting, such as multiple instance learning (MIL) and sentence-level text classification, where the input naturally follows a two-level nested structure (e.g., instances within bags or words within sentences). However, it is theoretically possible to extend our method to deeper nested structures (e.g., three levels: word → sentence → paragraph in NLP tasks) by introducing additional consistency constraints. In this case, we would incorporate extra consistency terms between the attributions at each level, ensuring alignment across all levels.

While our current formulation is optimized for two levels, adding more levels would increase the complexity of the model, potentially leading to higher computational costs and making convergence more challenging. Nevertheless, by extending the consistency constraints to deeper levels, it is feasible to handle more complex nested structures. We plan to explore these extensions in future research to assess both the theoretical implications and practical scalability of our approach.

The paper mentions that C2FA may perform worse when the consistency property is inherently not satisfied. I feel this can be a more realistic setup, which makes C2FA less pragmatic, any justification here?

You raise a valid point. Indeed, there are realistic scenarios where high-level feature attributions (HiFAs) and low-level feature attributions (LoFAs) may not always be consistent. For example, it is possible for a high-level feature to appear significant while its constituent low-level features do not, or vice versa. In such cases, enforcing the consistency constraint could potentially lead to suboptimal explanations. However, our method is designed to strike a balance between consistency and faithfulness. The consistency constraint is weighted using hyperparameters ($λ_H$, $λ_L$), allowing flexibility in scenarios where strict consistency may not be realistic. Additionally, monitoring the optimization loss during training can help detect when the consistency assumption is not being satisfied, allowing for adjustments. In future work, we aim to explore adaptive mechanisms that can dynamically adjust the importance of consistency based on the nature of the data.

The current formulation assumes a fixed nested structure. How might C2FA be adapted to handle variable-length or dynamic nested structures, such as those often encountered in NLP tasks?

Our proposed method constructs explanations on a per-sample basis, which means it can naturally handle inputs with varying lengths and nested structures across different samples. Since C2FA estimates feature attributions for each sample independently, it is not limited by fixed-length inputs or rigid nested structures. Therefore, the applicability of C2FA to variable-length or dynamic nested structures largely depends on whether the black-box model used for predictions can accommodate such data flexibly. As long as the underlying model can handle inputs of varying lengths, C2FA can effectively generate consistent explanations by leveraging the flexibility of its surrogate model approach.

The experiments focus on classification tasks. How well do you expect C2FA to generalize to regression problems or other types of model outputs?

Similar to LIME, our proposed method, C2FA, can be applied to both regression and classification tasks using the same formulation. The process of estimating high-level and low-level feature attributions remains consistent, regardless of whether the output is categorical (classification) or continuous (regression). The key difference lies in how the surrogate model interprets the predictions: in regression, the explanations would focus on the continuous output values rather than class probabilities. This flexibility allows C2FA to generalize across different types of model outputs without requiring significant modifications.

Thank you for taking the time to review our paper. We appreciate your constructive feedback. Below, we provide responses to each of your questions.

Lines 89-91: The paper states that estimating HiFAs and LoFAs separately can lead to inconsistent explanations between them. How might this mismatch reduce users' trust in the model?

When the high-level feature attributions (HiFAs) and low-level feature attributions (LoFAs) are estimated separately, there is a risk that the explanations they provide may not align with each other. For instance, if the HiFAs indicate that a certain image contributes significantly to the model's prediction, but the corresponding LoFAs do not highlight any meaningful features within that image, users may perceive the explanations as contradictory or unreliable. This inconsistency can lead to confusion, thereby reducing users' trust in the model, especially in critical applications such as healthcare or finance, where users rely on model explanations to make informed decisions. By ensuring consistency between HiFAs and LoFAs, our method aims to provide more coherent and trustworthy explanations​.

What would the explanation be if a bag contains more than two images featuring "cats"?

In our experiments, it is common for a bag to contain multiple images featuring "cats." In such cases, our method estimates that the images containing cats will have high HiFAs. However, the values of these HiFAs will be distributed based on the probability output by the model for the label "cats." The LoFAs are then estimated in a way that is consistent with the HiFAs, ensuring that the explanations align across both levels. Since we cannot attach images here, we plan to include additional examples of such cases in the Appendix to illustrate how our method handles bags with multiple relevant images.

Thank you for taking the time to review our paper. We appreciate your thoughtful comments. Below, we provide responses to each of your questions.

line 237. Why does ADMM have the advantage of estimating and independently even though there is the interdependence?

The advantage of using ADMM in our method lies in its use of auxiliary variables to decouple the optimization of interdependent variables, such as high-level feature attributions (HiFAs) and low-level feature attributions (LoFAs). By introducing auxiliary variables and Lagrange multipliers, ADMM reformulates the joint optimization problem into separate subproblems for HiFAs and LoFAs. This allows us to solve for HiFAs and LoFAs independently in each step, while ensuring that they converge to a consistent solution through the consistency constraint. Essentially, even though HiFAs and LoFAs are interdependent due to the consistency requirement, ADMM leverages auxiliary variables to split the optimization into manageable parts, thereby enabling efficient and independent updates. The concrete algorithm is described in Appendix A in the supplementary material.

line 301-303. Does it mean the LoFAs for their higher level feature attribution are just randomly selected based on the corresponding HiFA? What if you take the independently computed LoFA (without the constraint), and distribute each HiFA proportional to the independently computed LoFA?

Such an approach is indeed possible as a baseline. However, if the goal is to estimate HiFAs consistently using the derived LoFAs, a bottom-up approach like BU-LIME would likely be more effective. This is because BU-LIME inherently maintains consistency by aggregating LoFAs to obtain HiFAs, ensuring that the attributions align well between the two levels.

How do bottom-up version of other methods like BU-SHAP, BU-RISE, BU-IntGrad do?

We did not compare our method with bottom-up versions of other methods like BU-SHAP, BU-RISE, or BU-IntGrad because our proposed approach is specifically designed for surrogate model-based methods like LIME. For a fair comparison, we focused on methods aligned with this framework. Notably, (Kernel) SHAP is also a type of surrogate model-based method similar to LIME, with the primary difference being in the use of a kernel function. Therefore, our proposed method, which is optimized for consistency between high-level and low-level feature attributions, would also likely work well with SHAP. However, for methods like RISE and Integrated Gradients, which do not inherently use surrogate models, additional modifications would be necessary to enforce the consistency property, making direct comparisons less straightforward.