Thank you Reviewer nq6A for a careful analysis of our paper! We are glad to know that you find our proposed framework useful and experiments well-executed. We address the concerns raised in your review below:

Are the four components really independent? Does transformation and constraint depend on each other?

We intentionally designed the independence of the four components from an implementation perspective by defining them separately in the code. This intentional design choice enables researchers and practitioners to define new linguistic capabilities tailored to their specific tasks in a modular fashion. We do agree that while defining a linguistic-capability one may implicitly weigh the interdependence of these components. For example, practitioners may choose not to impose specific constraints with particular transformations, or vice versa.

Puzzled with results of BERT-Tiny. Does this mean that the predictions of both models are quite different (hence poor agreement), but within the individual model the predictions do not change when it is given original and perturbed samples?

Precisely! We find that while larger target models (e.g., BERT-Base) disagreed with BERT-Tiny (as reference model) on individual samples (lower agreement), they still shared invariances i.e., the larger target models didn’t have substantial differences in their predictions on original and perturbed samples.

Any interesting findings on the model agreement on original input vs. perturbed input?

We didn’t observe any atypical trends on the model agreement on original input (IID-Agreement) and perturbed input (OOD-Agreement). We found them to mostly follow similar trends across many different settings. Along similar lines, a recent work in computer vision has also observed that IID-agreement and OOD-agreement are highly correlated with each other. [1]

Suggestions for presentation improvements

Mention the tasks being used in the experiments at the beginning of the paper: Thanks for the suggestion, we added a pointer to the tasks being used in experiments in the concluding paragraph of the Introduction.

Unclear how inefficient is perturbation generation: The major computation costs for SCoPE can be attributed to the process of generating invariant perturbations w.r.t the reference model. Importantly, this is a one-time process (per reference model) and we can reuse the perturbations generated for a particular reference model for comparison with different target models. This cost varies depending on the linguistic capability. For instance, generating perturbations for Synonym-Invariance takes $\approx 1.2$ seconds per sample, for BERT-Base as a reference model. We have added a few results on the compute costs for perturbation generation in the supplementary material Sec. K. Please let us know If there are other specific details regarding perturbed input generation we can further clarify.

results for fairness capability is quite interesting. If there is space, might consider to put it in the main text: We are glad to note that you find the fairness capability results quite interesting. Unfortunately, space constraints prevent us from providing a detailed discussion on the fairness capability results in the main paper. However, should our work be accepted, we plan to incorporate these findings into the main paper, as we would have access to an additional page.

[1] Baek, Christina, et al. "Agreement-on-the-line: Predicting the performance of neural networks under distribution shift." Advances in Neural Information Processing Systems 35 (2022)

Thank you Reviewer xMdc for your thoughtful comments. We address the concerns raised in your review below:

Why do shared invariances tell us about performance in the wild?; Motivation of shared-invariances beyond OOD agreement

Being invariant to perturbations in data that are not intrinsically relevant to the task labels is helpful for models to generalize beyond their training distribution [1, 2]. Therefore, comparing the invariances shared between the two models can shed light on how similarly they will generalize on data that deviates from the training distribution. This is particularly interesting when considering models with distinct design choices (e.g., distillation, model size), as low values of shared invariance signify that the design choice renders the model susceptible to perturbations that it was originally robust to. In this work, we investigate three distinct perturbations – synonym-based word substitution, inducing misspellings, and protected category-based word substitution (fairness), that preserve the underlying sentiment of a sentence for a sentiment classification task. Through our experiments, we uncover several insights such as – distilling BERT makes it more susceptible to perturbations along certain linguistic capabilities (Typo-Invariance) than others, larger models encode a more diverse set of invariances compared to comparably accurate smaller models, etc.

We would also like to clarify that we introduce novel measures of shared invariances based on the motivations described above. If existing metrics were able to capture the degree of shared invariance, then there would be no need to introduce new metrics. This is why we tested OOD-agreement, since it is one of the popular metrics to quantify the behavioral similarity of two models. To quantify the relationship between invariance- and agreement-based metrics, we conducted a correlation analysis between OOD-agreement and Soft-SCoPE which revealed poor correspondence in certain important cases. Therefore, if one wants to investigate the shared invariance between models, then using a metric like Soft-SCoPE is more desirable.

Why would a practitioner trust your metrics more than OOD-agreement to predict e.g. how much BERT distillation hurts the model’s robustness to typos?

OOD agreement can answer questions such as “Do BERT-Base and DistilBERT have the same behavior on a set of perturbed examples?”, while invariance-based metrics like SCoPE can answer questions such as “Does distilling BERT increase its susceptibility (or reduce invariances) to perturbations it was initially robust against?” The reason why OOD agreement on its own cannot answer the same question is that evaluating whether a target model is invariant to a perturbation requires determining whether it produces consistent outputs on both original and perturbed samples. In the special case of binary classification, the second-order agreement between IID and OOD agreement rates can be used for evaluating such invariances. We in fact explore this relationship in Sec. E.2 of the supplementary material in depth, and find that Hard-SCoPE (and consequently Soft-SCoPE) essentially measures this “agreement between the IID and OOD agreement rates” for a binary classification setup and hence evaluates the invariance of the target model directly. Additionally, it becomes complicated to only use agreement rates to quantify invariances in a multi-classification setup. Therefore, SCoPE-based metrics prove to be more apt for assessing invariances encoded by a target model.

Is a notation for the source of perturbed candidates x’ missing from Equation 1?

We assume that by “source” here, the reviewer means initialization of $x’$. As discussed in Section 3.1, the linguistic capability $C$ perturbs $x$ to obtain the $x’$. Hence, the source of the $x’$ here is $x$. We subject the optimization procedure to disregard trivial solutions i.e., $x = x’$. We have updated Equation-1 in the paper to reflect the same more clearly.

A table showing examples of (x, x’) pairs is a must for a paper like this.

Thanks for the suggestion. We have added the same in the supplementary materials Sec. L.

[1] Deng, Weijian, et al. "On the strong correlation between model invariance and generalization." Advances in Neural Information Processing Systems 35 (2022)

[2] Izmailov, Pavel, et al. "On feature learning in the presence of spurious correlations." Advances in Neural Information Processing Systems 35 (2022)

What is the cost of computing SCoPE?

The major computation costs for SCoPE can be attributed to the process of generating invariant perturbations with respect to the reference model. Importantly, this is a one-time process (per reference model) and we can reuse the perturbations generated for a particular reference model for comparison with different target models. This cost varies depending on the linguistic capability, and we have added a few results on the same in the supplementary material Sec. K. There is no substantial additional compute overhead on computing the SCoPE-based metrics, once perturbations are generated.

Regarding the compute-accuracy tradeoff: In Section H of the supplementary materials, we discuss the reliability of our proposed metrics as the number of base samples (X) decreases. Specifically, we report the mean metric values and the 95% confidence interval of this estimate computed over 100 trials for many values of the base-sample count. We find that our metrics provide tight confidence intervals for as low as 50 base samples. For experiments in Section-5, since the reference model was fixed as GPT-2, the costs were primarily due to running each GPT-3.5 model on 100 base-perturbed sample pairs. At the time of conducting the experiments, it cost us $\approx 55$$ to generate all the results in Section-5.

What is behavioral about the invariances?

The invariances shared between models can be evaluated either at the output level, which we refer to as the behavior of the model, or at the representation level, examining whether the model maintains very similar representations (at some intermediate layer) for perturbed and original samples. In the related works section, we discuss works [6] that compute representational shared-invariances between two computer vision models, and elaborate why their setups may not be directly applicable to natural language processing.

[1] Ribeiro, Marco Tulio et al. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,

[2] La Malfa, Emanuele, and Marta Kwiatkowska. The king is naked: on the notion of robustness for natural language processing. Proceedings of the AAAI Conference on Artificial Intelligence.

[3] Lee, H. and Ng, A. Y. Spam deobfuscation using a hidden markov model. In CEAS, 2005.

[4] Pruthi, D. et al., Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

[5] Ren, S. et al., Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

[6] Nanda, Vedant, et al. "Measuring Representational Robustness of Neural Networks Through Shared Invariances." International Conference on Machine Learning. PMLR, 2022.

Thank you for reviewing our paper, Reviewer cey2. We address all the concerns you raised in your review below.

Issues with notations

Notation delta_m1 is never formally defined: \Delta \vec{m}^{\} is the change in a model – $m$’s, output probability distribution in the presence of a perturbation ($x \rightarrow x’$), i.e., \Delta \vec{m}^{\} = m(x')-m(x). Thus, Soft-SCoPE between two models Soft-SCoPE ($m_2$ | $m_1$) is high if the change (or effect of the perturbation) in two models (\Delta \vec{m_1}^{\},\Delta \vec{m_2}^{\}) is similar (in both direction and magnitude) and low otherwise. We've improved the description for the same on Page-5/6.

Notation suggestions for perturbations and metrics: Thanks for the suggestion regarding the alternative way of denoting notations. We have incorporated those suggestions and modified some of the notation definitions accordingly. In Hard-SCoPE and Soft-SCoPE the expectation is now over $x \in X$ and we denote the corresponding perturbed samples $x’$ in each equation by $C(x; m)$, where $m$ is the reference model. Additionally, while generating $x’$, we disregard the trivial solution i.e., $x = x’$, and optimize for a perturbation that minimizes the objective function $\mathcal{L}(m(x),m(x'))$.

Hard-SCoPE of two models can be either 0, or 1: Thank you for the pointer. Your understanding is correct, we meant that the Hard-SCoPE of two models for a particular base-perturbed pair can either be 0 or 1. We have added clarifications on Page-6.

Experiments lack depth; correlations are either expected

We perform extensive experiments demonstrating the utility of the proposed measure across many different settings – different tasks (text classification and language modeling), datasets (SST2 and AG-News), linguistic capabilities (Synonym-Invariance, Typo-Invariance, Fairness), models (BERT-, InstructGPT-family), etc. Please let us know if you have are other specific suggestions for detailed experiments, and we would happy to follow up with them during the discussion period.

Choice of "capabilities"

Relationship with perturbation: As discussed in the Introduction and Related Works, we follow prior works [1, 2] in defining linguistic capabilities. A linguistic capability evaluates a model’s competence on a particular aspect of knowledge and understanding required to solve an NLP task by validating its input-output behavior under the corresponding scenario. Thus, these linguistic capabilities (such as invariance to synonyms, or variance to negation) are required by NLP models to perform a particular task (sentiment classification) well. One way to evaluate a model’s linguistic capabilities is via perturbations i.e., perturbing the input along the concept encoded by a linguistic capability (e.g., replacing words with synonyms, negating sentences) and evaluating whether the model’s prediction on the perturbed input aligns with human judgment. Thus, the perturbation generation rules are closely defined by the linguistic capability of interest.

Specific capabilities: We focus on the particular linguistic capabilities i.e., Synonym-Invariance, Typo-Invariance, and Fairness as there is rich literature studying them in NLP, albeit from an adversarial robustness perspective as they concern the reliability of many real-world systems, such as spam detection, toxicity classification, and fake news detection [3, 4, 5]. We would like to emphasize that the primary goal of this work is to a) introduce a general framework for comparing shared invariances along a linguistic capability and b) proposing novel measures of behavioral shared invariance. The list of linguistic capabilities can be easily expanded by researchers and practitioners in the future based on the specifications of their research questions and tasks as they would only need to define the specific transformation and constraints.

Confusion regarding Figure 5:

In Figure 5 (left), we aim to explore the effect of the size of different black-box GPT-3 models when compared with GPT-2 as the reference model. The x-axis represents different metrics, while each different color corresponds to a distinct GPT-3 target model. We keep the styling template similar to the one in Figure 4, where we also compare multiple target models across multiple metrics. Please let us know if there’s anything specific that’s unclear, we would be happy to clarify further.

Additional / minor issues

Thank you for these suggestions, we have incorporated them in the paper.

Thank you for your review Reviewer yXpb.

We have added descriptions for the previously undefined acronyms, namely, IID (Independent and identically distributed) and OOD (Out-of-Distribution) data in the paper. We have also included additional experiment details related to different NLP tasks in the concluding paragraph of the Introduction. Please let us know if there are any other specific details that we can further clarify.