Ensemble Distillation for Unsupervised Constituency Parsing

Thanks for the review.

Weakness 1: Somewhat narrow scope (ensembling constituency trees)

Unsupervised constituency parsing, in fact, fits ICLR themes well, as it concerns learning the representations of human language. In the literature, a number of impactful papers were published in previous editions of ICLR, such as Shen et al., 2018, 2019.

In addition, our study has general machine learning implications, especially for multi-teacher distillation. Our study shows that a straightforward union distillation from multiple teachers (as was done in previous work) may not yield improved performance, whereas our ensemble-then-distill approach is able to alleviate the over-smoothing issue in traditional methods. We’re happy to explore multi-teacher distillation for different data types (such as sequences and graphs), mentioned in Future Work.

Weakness 2: Some details missing (see the questions)

Questions: Would "Our ensemble (X teacher across runs)" in Table 2 use candidate trees from different models, or do they all end up from the same model (e.g., ConTest for X="best")? It's good to know the answer to this question because one of the main claims of the paper is that it's important to exploit the large qualitative differences between different methods (Table 1). The fact that we get the best result by just using outputs from the same model seems to refute that claim (i.e., it's all variance reduction).

Thanks for the detailed question. By “Best/Worst teacher across runs”, we mean that we must still choose exactly one run for a model, but which run may not correspond among models. A simple example:

Model1: Run1=85, Run2=90, Run3=95

Model2: Run1=65, Run2=60, Run3=55

Then, the best teachers across runs will be Model1-Run3 and Model2-Run1. The worst teachers across runs will be Model1-Run1 and Model2-Run3.

We’ve clarified this in the revision (third paragraph in Sec 3.3).

We thank the reviewer for the support, especially for recognizing the scientific rigor of our study by mentioning “the statements and conclusions in this paper are remarkably honest. Most of the claims are very well supported by related work and experimental results.”

Weakness 1 (MBR-style or consistency-based literature review)

Thanks for sharing the literature. We have included the suggested references, as well as some of their references, in the revision.

Weakness 2 (Motivation of core NLP)

We’d clarify that, by saying “a core task,” we mean that parsing has been traditionally referred to as a core NLP task. In the revision, we call parsing “a well-established task.”

Nevertheless, we would like to point out the significance of unsupervised parsing as a curious task of language structure discovery. It verifies linguistic theory, showing that linguistically defined constituents can naturally emerge in an unsupervised way. In our related work, we also discussed how unsupervised parsing may inspire the structure discovery of motion-sensor data (Peng et al., 2011).

Minor points on weaknesses 1 (conventional linguistic literature references)

We’re grateful for the suggestions and have included them in the revision.

Minor points on weaknesses 2 (Related work about the low correlation discovery)

Thanks for the suggestion. The mentioned work is discussed and cited in the revision. Williams et al., 2018 showed low correlations among early latent-tree models that were claimed to induce task-specific tree structures, whereas our paper shows low correlation among unsupervised parsing models that are aimed at discovering linguistically plausible tree structures. Further, our paper shows that such low correlation suggests different expertise of unsupervised parsers, which can be utilized by model ensembles for performance improvement.

Minor points on weaknesses 3: Detailed linguistic analysis of what patterns are fixed

In the performance-by-type analysis, we’ve shown that our approach achieves consistently high performance across all constituency types. We’re further presenting a case study in the appendix, suggesting that our ensemble indeed performs voting for local structures and fixes teachers’ predictions.

Question 1: Which split of PTB did you use to generate the statistics in Table 1?

We used the standard split, namely, Section 23 of PTB for tests.

Question 2: why are the +RNNG/+URNNG oracle performance different from the basic one (83.3)?

In Table 2, the +RNNG column for Oracle row (Row 15) is, in fact, a supervised RNNG trained on the binarized PTB-training set (binarized ground-truth). +URNNG is further tuning the RNNG in an unsupervised manner. The performance is lower than Oracle, which is the performance upper bound.

We thank the reviewer for recognizing a number of our key contributions, including a new notion of tree averaging, the ensemble-then-distill approach, the efficient inference, and an intriguing phenomenon.

Weakness 1. Lack of clarification regarding the methodology design

• (F1 score) We chose the F1 score because F1 is the standard measure for constituency parsing, explicitly mentioned in the second paragraph of Sec 2.2. Therefore, our similarity score is well motivated. We would be grateful if the reviewer can recommend alternatives that are more justified.

• (Choice of teacher models) We chose the seven teachers because they are classic or state-of-the-art unsupervised parsers (mentioned in Sec. 3.2, first line) with publicly available code. Specifically, we chose ContextDistort because it is a very recent study (ACL2023) and largely differs from previous approaches in its methodology. Note that in the analysis of the number of teachers (Fig. 2), we have experimented with various combinations of the teachers, where the results are consistent.

Weakness 2: “I wonder if it is more appropriate to choose the model with highest performance in the former setting so that it can be further validated there exists additional boost due to different expertise (e.g., Neural PCFG for Group 2).”

In Figure 1, four of the seven groups (namely, Groups 1, 4, 6, and 7) meet the criterion that the reviewer mentioned. Hence, it's possible to draw the conclusion that the reviewer is seeking from a subset of our experimentation. Moreover, we do not believe it is essential to use the best-performing model of the latter setting for multiple runs in the former, control setting. As we can see from the figure, the aforementioned four groups are not different from the rest.

Weakness 3 (two typos in a paper)

Thanks for catching two typos in our paper! We’ve fixed them in the revision.

Question 1: The ensemble method demonstrates its effectiveness on PTB. However, in 2020, the F1 score of CRF parser on PTB variants had already been above 90 (Zhang et al., 2020). There is indeed a performance boost in comparison to oracle score (the highest possible F1 score of binarize groundtruth trees). But it is not appropriate to claim that “largly bridging the gap between supervised and unsupervised constituency parsing” on Page 6.

Thanks for pointing this out. We have revised our claim as “largely bridging the gap between supervised and unsupervised binary constituency parsing”.

Question 2: In Results on SUSANNE, the authors claim that “This is a realistic experiment to examine the models’ performance in an unseen low-resource domain.” However, SUSANNE is an English dataset. In CoNLL Shared task, there are tree-banks on low-resource languages (Zeman et al., 2017). It’s better that the authors can demonstrate the effectiveness of the approach on some of these low-resource language datasets.

Here, our claim about the SUSANNE experiment is domain shift, instead of low-resource language. As shown in Table 3, models trained on PTB underperform in other domains in the same language, showing that unsupervised parsing remains challenging in low-resource domains, even for English.

As mentioned in the response to Reviewer yYrJ, a key difficulty of a non-English ensemble is to find teacher unsupervised parsers, which are less addressed in the literature. We’re happy to address this direction as future work.

Question 3: In Table 3 on Page 7, PTB-supervised model is used for comparison. Which PTB-supervised model is used?

The PTB-supervised models are the RNNG and URNNG, shown by the column headers.

---

Overall, the reviewer identified a number of important contributions in our work, but gave an outrageous score of 3 by missing the key rationales of our model design and key experimental results that are already mentioned in the paper, as well as overly emphasizing two typos in our paper as a weakness of the writing. On the contrary, our contributions, thorough experimentation, and clear writing are well recognized by all the other reviewers.

In our author response, we have answered all the questions raised by this reviewer. We urge the reviewer to revisit our paper and adjust the score accordingly.

Thank you for recognizing the contributions of our work and strongly supporting our paper.

Weakness 1. not carrying distillation boost under domain shift

Thanks for the insight! We acknowledge that distillation does not help our ensemble, or even the supervised model, in the SUSANNE experiment. To be honest, this is also a little bit unexpected to us as well, but we have honestly stated that in the paper (left to the caption of Table 3). We’re happy to explore domain adaptation of unsupervised parsing more in future work.

Weakness 2: There is a lack of qualitative analysis of the types of behaviors that different model types exhibit, and how ensembling actually combines those. Some of this is done in the Appendix but it would be nice to see specific examples in the main paper, especially since that analysis is wrt constituency labels which the model isn’t actually being evaluated on.

Thanks for the suggestion. We have included a case study in the revision (which inevitably overflows to the appendix due to the volume and substance of our paper).

Question 1: How does regular RNNG perform, and why not use it as a teacher?

RNNG does not get trained well from scratch in an unsupervised manner, which is also reported in previous work [Kim et al., 2019a; Cao et al., 2020] and mentioned in Footnote 4 on Page 6. As a result, we did not use RNNG/URNNG as a teacher.

Question 2: Regarding the experiment in Figure 1, do you see similar results if you measure the gains from the distilled ensemble? That would be useful to see alongside

Thanks for the suggestion. Distillation analysis is expensive, as it requires performing inference for all teachers on the training data, training RNNG, and then refining with URNNG. For Figure 1, it requires 21 teachers' inference over the training set and training 14 RNNG/URNNG models, which may be unaffordable to us.

Nevertheless, we expect the phenomenon to hold for RNNG/URNNG, because during our development, we observed similar patterns among RNNG, URNNG, and their ensemble teacher. The F1 scores are also similar, with RNNG slightly lower and URNNG slightly higher, consistent with all PTB experiments.

We thank the reviewer for the great insight and strong support.

Weakness: It is comparing only for English, and it would be better to compare the model with other languages, e.g., Chinese, for further strengthening this submission.

Thanks for pointing out that our approach currently works for the English language. A key difficulty of non-English ensemble is to find teacher unsupervised parsers, which are less addressed in the literature. Nevertheless, our ensemble approach brings new opportunities for unsupervised multilingual and non-English parsing (such as transferring and building ensembles with the structural knowledge of different languages). We’re happy to explore this direction in the future, as has been mentioned in our future work section.

Question: I'd like to know the impact of length of inputs, e.g., whether the proposed method is better in lengthy input or not.

Thanks for the suggestion! We actually had the length analysis in our development but didn’t include it in the submission because our paper had already overflowed to the appendix. The finding is similar to the analysis of constituency labels: our ensemble maintains high performance across different lengths, compared with teacher models.

We now show the results in Appendix B.2.