Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization

Weakness1 (UNIREX already discussed comprehensiveness). We agree that UNIREX has discussed comprehensiveness and its excellent work has inspired many subsequent studies, including ours.

However, UNIREX and we contribute to the issue of "comprehensiveness" from different perspectives. UNIREX identifies (from the outcome perspective) that the rationales found by the model are not comprehensive enough, and has designed a comprehensiveness regularizer to fix this problem. However, the remaining untouched question is, why existing models fail to find comprehensive rationales. Our unique contribution is the analysis of the cause of this issue, specifically the diminishing marginal returns characteristic of MMI (as claimed in L112-115). Understanding the cause of this issue can help us find better solutions.

In summary, UNIREX's contribution is the observation and identification of this important issue, while our contribution is the analysis of the cause of this issue.

Weakness2 (not trained end-to-end). We are sorry, but this is a misunderstanding. Our method was trained end-to-end. The non-differentiability problem was solved with gumbel-softmax (the same as A2I).

Weakness3 (task performance). Thank you for your suggestion; now the new Tables 1-4 all include task performance (i.e., Accuracy).

Weakness4 (Visualization examples). Thank you for your suggestion, we now add the examples in Appendix A.9.

Weakness5 (hyperparameters). In our initial version, we only provided the learning rate, batch size, and their search strategies (Appendix A.2), but forgot to mention the sparsity and continuity constraints in Eq. 5. This is because these two hyperparameters were directly inherited from the A2I code and were not modified. In Eq.5, $\lambda_1=11$, and $\lambda_2=12$.

Q1 (what are the metrics when you pass gold rationales to the predictor directly). We are sorry but we are not sure what you mean by "metrics". Could you please be more specific? The points on the far right of the x-axis in Figure 3 represent the results when the gold rationale is directly input into the RNP predictor.

Q2 (OOD data and statistical shortcuts). Thank you for your suggestion. We now add a dataset CUB, where OOD data is a challenging problem. The results and discussions are in Appendix A.10. Our N2R still outperforms MMI-based RNP in terms of both rationale quality and classification accuracy. From the visualized example in Fig.9, we find that RNP selects the background water (shortcut) as the rationale for "waterbird", while our N2R does not.

Q3 (evaluating on token level rationales ). Although the rationales in BeerAdvocate are annotated at the sentence level, our extractor actually performs selection at the token level. Additionally, the hotel-related datasets we used have span-level rationales. Perhaps these clarifications can address your concerns to some extent.

For e-SNLI, we do not find a sufficient number of suitable baselines in a short time. Therefore, we add another dataset with token-level rationales, MovieReview. The results are in Appendix A.11. N2R still outperforms recent MMI-based methods.

We are truly grateful for your detailed review and the thoughtful suggestions you provided.

Weakness1&Q1 (classification quality in real-world scenarios). We are sorry, but we are not sure about the specific concern of you by mentioning "real-world scenarios". Could you please be more specific?

Except for BA2Motif, all the other datasets we use are real-world datasets. And in the experiments of Table 1 and 2, our method indeed has a higher prediction accuracy compared to the MMI-based methods.

Do you mean that we need to consider more challenging datasets? Now we add two challenging datasets, CUB and MovieReview, for image classification and text classification respectively. The CUB (image classification) dataset is considered challenging by previous studies due to its overfitting problem. And the Movie dataset is challenging due to its long texts. More details and results are in Appendix A.10 and A.11.

Aside from rationalization methods, we also implement a vanilla non-interpretable classifier on the above two datasets. And our N2R gets even better classification results as compared to the vanilla classifier. Detailed discussion is in Appendix A.10.

And for the datasets previously used in our submission, we also implement a vanilla classifier with BERT for the Beer-Appearance and Beer-Aroma datasets. The results are in Appendix A.11. We still get comparable results with BERT classifier.

Weakness2&Q2 (experiments on more domains aside from text and graph). Thank you for your suggestion. We now extend our method to the image domain. We implement the extractor with a U-net and the predictor with ResNet18. The extractor selects a set of pixels to be the rationale. We evaluate our method on a image classification dataset CUB. Our N2R is still effective on this domain. More results and discussions are in Appendix A.10.

Weakness3&Q3 (computational resources, encoder architectures, and scalability).

Computational resources. A typical predictor consists of a feature encoder and an MLP head. The encoder first transforms the input into a feature vector, which is then fed into the MLP to produce the output. The only additional operation we perform is calculating the norm of the feature vector, which has a very minor computational cost, making the extra overhead during forward propagation negligible. The extra computation may be more apparent during backward propagation. In RNP, the cross-entropy backpropagation goes through both the predictor and extractor, thus we denote the computational load to be 2. N2R's cross-entropy loss only passes through the predictor, making this part of the computation a 1, while the norm loss first passes through the predictor and then the extractor, making its computation a 2, thus the total computation of N2R is 3. Therefore, the backward propagation cost of N2R is about 1.5 times that of RNP. However, although our method has higher costs than the original RNP, it remains lower than other more advanced methods, as they introduce additional auxiliary modules as regularization terms, with computational loads more than twice that of RNP.

Besides, we find our N2R converges much faster than RNP, which means that we can train N2R with fewer steps (N2R needs less than half steps) and thus saving the computational costs. The comparison of convergence speed between RNP and N2R are in Appendix A.13.

The memory usage (MB) for the GRU-based settings (Table 1) are as follows:

The training time (on a RTX4090 GPU) for the Beer-Appearance dataset is

Since the backpropagation of N2R (Eq.8 and 9) is conducted in two steps, the memory usage does not increase compared to RNP. Given that N2R converges faster (Fig.10 shows that N2R requires less than half the training epochs of RNP), the actual training time is shorter than that of RNP. Moreover, because each step of our two-step backpropagation consumes less computational resources than RNP, the advantages of N2R become more pronounced when increases in computational load (such as increased batch size) make GPU power the bottleneck.

Encoder architectures. A typical predictor consists of a feature encoder and an MLP head. The encoder first transforms the input into a feature vector, which is then fed into the MLP to produce the output. The only additional operation we perform is calculating the norm of the feature vector. Thus, our method is encoder-agnostic, as we do not directly manipulate the encoder, but instead operate on the output vectors of the encoder. There are also no specific limitations regarding the extractor architecture, as our extractor is the same as RNP.

Our experiments are conducted on different encoder architectures, including RNN-based (GRUs), Transformer-based (BERT), and GNN-based (GCN). And we have now added a CNN-based (ResNet18) experiment (see Weakness2&Q2). We are not sure if we have correctly understood your concern; could you please specify a bit more?

Scalabilit (more model parameters and longer texts). Thank you for your valuable suggestions. We now add an additional MovieReview dataset which consists of longer texts. The results are in Appendix A.11. N2R still outperforms MMI-based methods on both rationale quality and classification accuracy.

As for the impact of model size, our current experiments have covered small models (GRU and GCN), medium-sized models (ResNet18), and relatively large models (BERT). We observe that as the model size increases, the improvement of N2R over RNP also grows (the improvement is minor when using GCN and GRU, more significant in experiments using ResNet18, and even greater when using BERT). This may be due to larger models being more likely to correctly classify based on very subtle clues, which exacerbates the difficulty for MMI to identify complete clues. However, an increase in model size tends to make the distinction between informative rationale and uninformative noise more pronounced in terms of norm (the weight matrix can be viewed as a series of column vectors. Mathematically, given a k-dimensional vector, the probability that it is orthogonal to a randomly sampled k-dimensional noise vector increases with the dimension k).

Could you please specify your concerns more with more details so that we can identify areas that need clarification or additional experiments?

Weakness4 (theoretical support). Thank you for your suggestion. We have now added theoretical support for our method in Appendix A.14, which covers two aspects.

First, why the norm can be used to distinguish between informative rationale and uninformative noise: the overall idea is that in high-dimensional space, two random vectors are highly likely to be orthogonal, thus the norm of the noise input multiplied by the weight matrix approaches zero.

Second, why our method is not affected by the problem of diminishing marginal returns. We have shown that, under certain ideal assumptions, the norm increases almost linearly with the increase in useful information in the input, which also matches the trend observed in Figure 7(c)(f)(i).

Minor issues.

1. Using a logarithm is an empirical attempt, aimed at ensuring sufficient gradient in the initial phase of training (as the early norm growth in Fig.3 is not significant, adding a log can amplify the gradient), and preventing gradient explosion in the later stages of training. The logarithm is also a common trick in other fields.
2. We apologize for the confusion caused. The legend in the figure already indicates the meaning of the y-coordinates, so we omitted the y-label. We have now added references to (a), (b), and (c) in the caption.
3. We will revise it in the camera-ready version (for now, to avoid confusion in referencing during review and revision, we have not made changes on it).

Overall clairfication. We are so sorry to make such a huge misunderstanding. But we do refer to weight matrices (e.g., the green RNN/Transformer in Fig.1) and not to the embedding matrix. We do not mention the embedding matrix because it is just an untrainable dictionary and thus a part of the input itself (i.e., the embedding matrix is considered as a pre-processing tool of the $X$ in Fig.1 and $X$ can be considered as a sequence of dense word vectors rather than one-hot vectors).

Consider that the rationale candidate $Z$ (Fig.1) is a sequence of word vectors (i.e., the output of a embedding matrix), after sending $Z$ to an encoder (e.g., RNN/Transformer), we get a sequence of word representations $B=RNN(Z)$. We then get a single vector $C$ by pooling: $C=maxpooling(B)$. Finally , the $||Enc(Z)||_2$ in our method (L354) is obtained by $||Enc(Z)||_2=||C||_2$.

W1 (weight matrices cannot be used for estimating the parts of inputs). We agree that weight matrices are not dependent on the particular input, but we have different oppinions about "they cannot be used for estimating the parts of inputs".

What we mean isn't that the weight matrix itself can be used to estimate inputs, but rather the interaction between the weight matrix and the input is used to determine whether an input can be utilized. If the representation norm of a rationale candidate selected by the extractor is close to 0 after passing through the weight matrix, it indicates the input cannot be utilized. Conversely, if the rationale candidate can be effectively utilized, its representation will have a larger norm after passing through the weight matrix. The reason for this property is Sec 3.2 and empirically verified by Fig 3(c).

Here is a toy example for better intuitive understanding.
Consider a network consists of only a linear layer (without bias) and has a low-rank weight matrix $M$ that, after elementary row transformations, takes the following form:

$A$ and $B$ represent inputs with and without the model's learned information respectively. after undergoing corresponding transformations (existing research indicates that after performing certain elementary transformations on the weight matrix, a corresponding transformation can be found for the inputs, resulting in identical outputs [1]), they are likely to take the forms like: $A=[1,0,0,0]$, $B=[0,0,0,1]$. The informative input $A$ usually matches more with the non-zero parts of $M$. And $||AM||_2=\sqrt{3}>||BM||_2=0$. (The column vectors of the weight matrix represent certain directions in high-dimensional space (determined by the learned information), but random noise B generally does not fall along these directions as it does not contain the corresponding information.)

That's how the weight matrice can be used for distinguishing between informative and uninformative rationale candidates. And Fig.3(c) and Fig.7(c)(f)(i) are the practical verifications of the above property.

W2 (GloVe). We used three different encoders (GRU/BERT/GCN) in Sec.5. GloVe is used in conjunction with the GRU encoder, whereas the other two encoders do not require an additional external embedding matrix. We apologize for not mentioning this. And the GRU+GloVe setup follows the baseline Inter_RAT (L420). The code is consistent with the paper, as our method is independent of the embedding matrix.

W3 (confusing term "zero-rank components"). We are sorry to make such a misunderstanding. When we mention "zero rank components", we in fact refer to the subspace that corresponds to the last row of matrix $M$ in response W1--a row with all values being zero. However, $M$ is a transformed matrix; the corresponding positions in the original weight matrix are not necessarily zero, so it's not proper to refer to them as "positions with values of zero." Therefore, we use "zero-rank components" to denote these redundant rows.

W4 (Fig.2). Fig.2 is a qualitative toy example (instead of a quantitative practical experiment) of the Sigmoid function to help intuitively understand the diminishing marginal returns from the perspective of gradient saturation. It was cited in L289-292. The x-axis represents the proportion of genuine gold rationale components in a extractor-selected rationale candidate. The middle of the x-axis indicates the absence of gold rationale components, representing complete noise. The further away from the center, the higher the content of gold rationale components.

W5 (task-irrelevant inputs). Since "John went for a walk" does not contain the model's learned information, it is an OOD input that falls outside the space that the model can handle. Just like the dots in the lower left of Figure 3(c), these dots are the results of inputting sentiment-irrelevant text into the predictor (sentiment classifier).

[1] Git Re-Basin: Merging Models modulo Permutation Symmetries. ICLR 2023.

1. (using "final layer representation") Thank you very much for your valuable suggestion. We have now revised the introduction and Fig.1 accordingly. And we have added the toy example to Appendix A.16.

2. (matrix ranks)

About the references in L93. Although [1] did not directly mention that the weight matrix is low-rank, they noted that the original large number of parameters can be derived from a smaller number of parameters through linear projection (please refer to the screenshot in Appendix A.15). This indicates that many parameters in the original weight matrix are linear combinations of other parameters. When the parameters of some rows (or columns) can all be derived from combinations of other rows (or columns), the matrix is considered low-rank.

As for the second paper Lora, we are grateful for your reminder. we realize that Lora indeed mentioned that the update matrix is low-rank, not that the original matrix is low-rank. We have now changed the supporting literature in L93 to the same as L99 (i.e., [2]).

Apart from the subtle implication in [1], our Section 3.2 is mainly borrowed from [2]. And Lemma D.4 in [2] shows theoretically that neural networks tend to learn low-rank weight matrices (see Fig. 12).

The details of the above discussion are in Appendix A.15.

We greatly appreciate your reminder; to avoid causing any confusion, we have changed the support reference to [2].

About the term "zero rank component". We know it is a confusing term, and we are really grateful for your suggestion. We have now completely removed this term, and accordingly revised the statements in the abstract (L18-25), introduction (L93-98), and Sec 3.2 (L242-249).

Overall, for $XW$, we consider each column vector in the weight matrix $W$ as a direction, and the linear combinations of these column vectors occupy a subspace of the hypersphere (e.g., a low-dimensional manifold). If the input falls within this subspace (i.e., the input vector's direction matches the directions of those column vectors), the model will be able to utilize it. If an input is orthogonal to these directions (according to Lemma 1 in Appendix A.14, for a given vector and a randomly sampled noise vector, they are likely to be orthogonal if the dimension is large enough), then the model cannot utilize this input. We refer to this subspace as the model's "capability subspace", and use the terms 'outside/inside the capability subspace' to replace the previous terms 'zero/non-zero rank components'.

[1] Intrinsic dimensionality explains the effectiveness of language model fine-tuning.
[2] Deep neural networks tend to extrapolate predictably.

Thank you again for your thoughtful suggestions.

Weakness 1-1 (about accuracy).
We have now added the accuracy for BERT experiments (Table 3). Since our main focus was on the rationale quality rather than prediction accuracy, we may have overlooked this metric initially when filling out the forms. Thank you very much for your reminder.

The reason we did not report the prediction accuracy for llama is because we found that there is a significant likelihood that responses from Llama would not contain explicit "positive/negative" classifications (for instance, in the hotel-service dataset, 50.5% of responses from the llama-finetune model did not specify a category, and this figure was 53.0% for hotel-cleanliness dataset). If we were to consider these responses without clear categories as incorrect predictions, the accuracy would appear very low and could be misleading. Considering that our main comparison is on the rationale quality and we do not claim our classification ability is superior to LLMs, hence we did not report this prediction accuracy to avoid unnecessary confusion.

Weakness 1-2 (BERT experiments on other datasets).
We have now added experiments on the other dataset of the BeerAdvocate benchmark, Beer-Aroma (Table 3). And our N2R still beats all the baselines. Since CR did not conduct experiments on the other two Hotel benchmark datasets, and due to the lack of comparative baselines, we did not perform experiments on these two datasets.

While using BERT, we found that some baseline methods were very sensitive to hyperparameters. Since previous baselines tend to design additional regularization terms, this adds complexity to the hyperparameter search process. Due to the limitations of our computational resources, we could not perform sufficient hyperparameter searches for these methods (as noted in L457-460). To avoid unfair comparisons arising from hyperparameter issues, we primarily compare against other baselines that have already been reproduced by the authors of CR as supplementary to the main experiments, verifying the generalizability of our model with different encoders. Since experiments involving BERT may be affected by unknown factors such as hyperparameter tuning, making it unsuitable for directly validating our claim, we did not choose it as the main experiment.

The reason we conducted experiments using BERT on only one dataset is that the BERT experiment serves merely as a supplementary experiment to the main study (L457-459) and is intended to verify the generalizability of our model when different encoders are used. Therefore, we selected the most commonly used Beer-Appearance dataset (which has been used by nearly all methods in the field) as a representative. Our main experiments (Tables 1 and 2) follow the settings of A2I and Inter_RAT.

Weakness 2 (It would be better to study what the improvement in the numbers actually means). Thank you for your suggestion, we have now add a case study in Appendix A.9.

Here is the explanation for your intuition "I personally prefer taking only R2 as the rationale as it is quite enough to make the prediction". For your convenience, we copy the mentioned toy example here: "⋯⋯ The food is very delicious (=R1), and I like it very much. (=R2) ⋯⋯".

We agree R2 is enough to make the prediction, and this is exactly why MMI-based methods cannot identify R1+R2. But R2 lacks comprehensiveness. Although R2 itself can indicate the sentiment label, it does not tell where the sentiment tendency comes from. For example, R2 cannot tell me why I ike the food. Is it because the food tastes good or smells good, or just has an attractive packaging?

The comprehensiveness is important in some high-risk decisions. For example, in AI-assisted diagnosis, if a person is infected with both Virus A and Virus B, regardless of whether Virus A or Virus B is chosen as the explanation, the model can successfully classify the person as sick. However, providing only one virus as the explanation could mislead the model user (a doctor) in their subsequent decision-making.

Question1 (BERT is expected to be a stronger model).

There are several possible reasons. First, Table 3 follows the CR setting by setting the rationale sparsity to 10%, compared to 15% in Table 1, which might be a more challenging setup. Second, as mentioned in L457-459, using BERT is more sensitive to hyperparameter choices, which may not be perfect (thus we mainly compare with baselines already implemented with BERT, and do not treat these comparisons as the main experiment, but only as supplementary to the main experiment).

Third, BERT's capabilities might be too strong, enabling it to classify based on very subtle clues. For MMI-based methods, MMI can only find a very small part of the correct rationale. Our method is not affected by this factor (as it does not seek rationales by maximizing prediction accuracy), hence our method's relative improvement is even more pronounced under settings using BERT.