We thank the reviewer for their valuable feedback. The reviewer appreciated the simplicity and superior performance of our approach. The reviewer also acknowledged the interestingness of our weakly supervised training method.

Response to the weakness mentioned by the reviewer:

I’m wondering if the comparison is fare for other systems…:

1. Concern about using the external data such as XNLI and MT-Prism: One of our baseline T5Score uses the same machine translation dataset MT-Prism to train their model. Other than SentBLEU all our baselines are based on pretrained-encoders. It is fair to compare our systems against these systems since our system does not require costly human annotations of machine translation quality scores from humans. Regarding comparing reference-included and reference-free systems, it is better to have a reference-free systems since in practice references are not always available. The fact that our system does not require the references during inference is a strength of our system.
2. Concern about using the same amount of data as the baselines: Regarding comparing score-based vs comparison-based systems on same training resources, we have argued in our paper that it is much easier to gather synthetic data for comparison based approach, which means comparison based approach has the advantage of having larger synthetic training data. Since it is the advantage of our approach, constraining our approach to have the same amount of training data would be unfair to our systems.
3. Concern about using different pretrained model from baselines: The reviewer also mentioned having the same base model as our baselines. Our best performing baseline T5-Score uses the mT5 model which is the same model we have used. Our second-best performing baseline used the XLM-Roberta-Large model to train their systems. To make sure our performance is not just due to using mT5 model, we have now trained a XLM-Roberta-Large based model using our method. The performance for this model which we call Comparator-XLMR on the DA20 benchmark is shown below. Comparator-XLMR achieves superior performance compared to baselines and similar performance to our Comparator-Large and Comparator-XXL models.

There should be more analysis on the proposed methods…:

We have performed further ablation studies of our different training stages. We trained ablated versions of our Comparator-Large model by removing different training stages. We show the performance of our ablated models on the DA20 dataset in the following table. Here sub-stage 3.1 refers to using the BERTScore based weak supervision and sub-stage 3.2 refers to the perturbation based weak supervision in our stage 3. The results show the specific contribution of metric and perturbation methods in Stage 3. The BERTScore based weak supervision has more contribution to the final performance. However, all the stages contribute positively to our final performance.

We thank the reviewer for their valuable feedback. The reviewer appreciated the simplicity and superior performance of our approach as well as the fact that our system requires much less costly data. The reviewer also appreciated the analyses we performed on our system.

Response to the weakness mentioned by the reviewer:

We thank the reviewer for bringing to our attention the related works that we missed. We note the following differences between our approach and the previous literature.

1. The previous systems rely on feature engineering and on external parser to generate some of the features. In contrast our approach does not rely on feature engineering and can work directly on the source sentence and the translation pair.
2. Our models show generalization to unseen language pairs during inference. In contrast, the previous systems only work on a single language pair.
3. Our system does not rely on costly human-annotated quality scores of machine translations and achieves superior performance with weakly supervised data. The previous systems mentioned by the reviewer were trained on human annotations of quality judgments.

Response to reviewer’s questions:

1. Direct assessment quality scores of machine translations collected from human evaluation is very noisy. It requires at least 15 annotations per machine translation to get reproducible scores. Since it is costly to get so many annotations per translation especially for low resource languages, in practice much less annotations may be available, sometimes as few as one. However, in this scenario suppose translation A gets one annotation of direct Assessment score 80 and translation B gets one annotation of Direct Assessment score 30, the difference in score being 50, we can still conclude that translation A is better than translation B. In previous literature a threshold of 25 score difference has been used to make similar conclusions.
2. We think it is important to have the figure since our approach is different from the usual MT evaluation. It is important to convey the difference clearly to readers.
3. mT5 was pretrained on the mC4 corpus. The latest data in mC4 corpus is from Common Crawl dump 2020-34 which includes web scrapes upto August 15th 2020. The WMT20 dataset was released on August 23rd 2020. Other benchmark datasets we used were released much later. Thus we are confident that the mT5 models have not seen the benchmark test data.
4. The references are required in stage 2 and 3 to generate the synthetic data. If we ablate the references completely from our system then we are left with only stage 1. The ablation results in our paper and in the table later in this response shows that without stage 2 and 3 the performance of the system is very poor. Thus access to the reference translations is required for training of our system. Our focus in this work is two-fold, i) to remove the requirement of having references during inference and ii) to remove the reliance on costly human annotations of translation quality scores for training machine translation evaluation models. Collecting these annotations are much costlier than collecting reference translations which can even be performed automatically by bi-text mining. Thus we believe reliance on reference translations during training is not a drawback of our system.
5. We performed the PERM-BOTH test following previous literature.[1] The results show that our Comparator-Large and Comparator-XXL model significantly outperforms all baselines on the DA20 benchmark dataset with a p-value < 0.05. On the MQM20 and MQM22 benchmark dataset our Comparator-XXL model significantly outperforms all baselines on en-de and zh-en language pairs with p-value <0.5. Even though our system does not significantly outperform other baselines in other settings, our system does not use any supervised data during training in contrast to the baselines.
6. We have performed further ablation experiments. We show the results in the following table. The results of our ablation show the performance of the system drops by one point to 35.5 when we remove stage 1 and 2 and just perform stage 3.

[1] A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (Deutsch et al., TACL 2021)

We thank the reviewer for their valuable feedback. The reviewer appreciated that our paper is well written and explains our motivation well. The reviewer also appreciated the extensive experiment results reported in our paper and the use of synthetic data to achieve good performance.

Response to weaknesses mentioned by the reviewer:

1. In practice we do not have that many systems to compare. Usually it is less than a dozen. In this scenario, our system does not add much overhead. Given the simplicity of our system, not relying on human annotations during training, not requiring access to references and generalization to unseen languages, it is an attractive alternative to existing score-based approach.
2. We can provide our models with more than two translations. On the pooling layer we can construct the pooled embedding for all pairs simultaneously. This can be done in parallel with full matrix operations. These pooled embeddings can be sent to classifier head in parallel to get rankings for all the translations. We are currently pursuing this approach to create a more efficient system.

Response to reviewer’s questions:

1. The reviewer is correct. We will fix the issue in our revised paper.
2. We report the results for the three languages of DA20 in the following table.

Our systems strongly beat the baseline systems on these languages even though our systems were not trained on human annotations. We unintentionally filtered out these languages in our evaluation. Additionally, for the iu-en and en-iu language pairs we noticed that we filtered out these language pairs also from our training. The strong performance on these two pairs also show the generalizability of our system to unseen language pairs during training.

We thank the reviewer for their valuable feedback. The reviewer acknowledged the soundness of our approach and the clear presentation of our paper.

Response to the weakness mentioned by the reviewer:

1. A straightforward approach to applying our system to perform system-level evaluation is to compare all machine translation pairs of pairs of MT systems. Since our model predicts a probability of which translation is better we can aggregate these probabilities across all the MT pairs to give a probability score for which MT system is better for each pair of. Then for each MT system we can average these probability scores to get a score for the system. These scores indicate how likely a system is to beat another system. For example, suppose we have three systems A, B and C under consideration. We apply our system to each pair and get the following scores. Each cell indicates the probability that the system on the row beats the system on the column. Thus, system A beats system B with a probability of 0.7. We take the average across rows to get the score for each system. Based on the scores we can conclude the ordering C>A>B.

We apply this approach to system-level evaluation on the DA20 dataset and achieve competitive results with baseline models. In the following table we show the results. The reported values are Pearson correlation with the system level gold human evaluation scores.

2. Machine translation evaluation is an important field in natural language processing. We have proposed a new framework for MT evaluation that beats state-of-art supervised baselines without using any human-annotated comparison data. Given the simplicity of our approach and the large reduction in human annotation cost, we believe our work is a significant contribution that should be of interest to the wider machine learning community. We also note that in previous ICLR conferences papers on Dialogue system evaluation has been publishe[1]That being said, we are currently exploring using our approach to other NLG evaluation tasks as future work.

3. We note that the xCOMET paper was released after we submitted our paper to ICLR and the SLIDE paper was released only 12 days before our submission. We agree that fine-grained error span evaluations are an important direction for MT evaluation systems. We plan to explore this direction in our future work.

4. DABertScore proposes a modification to the BertScore algorithm based on token-level difficulty weighting. Like BertScore, it is a reference-based metric. We have not found any conceptual overlap between DABertScore and our approach. Our system is a reference-free pairwise MT evaluation method that has been trained on synthetic data without using any human annotated translation quality scores. If the reviewer could clarify the conceptual overlap between our method and DABertScore we would be happy to include those comparisons in our paper.

[1] Amplayo, R.K., Liu, P.J., Zhao, Y., & Narayan, S. (2022). SMART: Sentences as Basic Units for Text Evaluation. ArXiv, abs/2208.01030.

Thank you for your further questions.

1. We chose the WMT20-COMET-QE-DA model for evaluation on DA20 and MQM20 datasets. We chose WMT21-COMET-QE-MQM for evaluation on MQM21 and MQM22 datasets. Both of these models are available from the COMET library. Models of different scales (ie base,large,XXL) are not available for COMET-QE.
2. We have updated the tables.
3. We will revise our paper to include discussion of system-level evaluation.
4. COMET-QE is supervised, It was trained on human annotations of machine translation quality scores (ie Direct Assessment and MQM scores). In contrast our models were trained on weakly supervised synthetic dataset. Our pairwise-comparison based approach enables us to create this large synthetic dataset. Thus, even though system-level evaluation with our comparison-based approach is more complicated than with score-based approach (ie COMET-QE), we believe the fact that our approach does not require costly human annotations for training makes up for it.
5. In the following table we show the system-level Pearson correlation of our system and the baselines on the MQM22 dataset.

6. We again note that the xCOMET paper was released after we submitted our paper to ICLR and the SLIDE paper was released only 12 days before our submission. Thus including insights from these two publications is not relevant to our current submission to ICLR.
7. We are still confused as we do not find any inter-system comparison that is similar to our approach in the DABertscore paper. Similar to Bertscore, DABertscore is a reference based approach, where machine translations are compared with the reference sentence to generate token similarity scores. Machine translations from two systems are not compared by DABertScore.
8. Thanks a lot for your valuable feedback.