We thank Reviewer Pupg for the detailed comments. We address some general concerns in the general response and response to your specific comments below:

(1) The main weakness is the limited compared method in the experiments. In the Tables in the paper, the state-of-the-art meta-learning methods are missing and the latested human annnotation simulators are also disregarded. Please consider add more SOTA methods for comparsion.

Human annotation simulation (HAS) is a new task proposed in our paper, so there is not, to our knowledge, latest human annotation simulator available - all baselines in our paper are adapted from suitable existing methods from other domains. We could not find suitable existing meta-learning techniques to adapt because our method is the first method that aims to meta-learn density estimators, whereas existing meta-learning methods focus on meta-learning (single-output) predictors. We’d be grateful if the reviewer could give citations to suitable prior meta-learning methods for HAS.

(2) The other weakness is that the costing time is not reported in the paper. Since the authors claimed that the proposed method is effective, the training and test time should be list compared to other SOTA in the experiments. Please consider to add more details here.

The computational time cost of all of the methods that have been compared for the four tasks studied is shown in Table (i) - (iv) in general response (2). Please refer to general response for more details and analysis.

We hope our response resolves your concerns.

We are thankful for Reviewer 2YK4’s feedback. We address some general concerns in the general response and response to your specific comments below:

(1) The proposed method is evaluated on only three human evaluation tasks, which may not be sufficient to generalize the effectiveness of the proposed method to other domains.

The three tasks used to evaluate the proposed method in our paper are real-world tasks from three very representative domains (emotion class annotation, toxic speech detection, and speech quality assessment) that concern human evaluations. In addition, we’ve added an extra emotion attribute prediction task in Appendix K of the revised manuscript, which, instead of estimating discrete emotion labels such as happy, sad, neutral, predicts emotion attributes (i.e. valence – whether the speaker is positive or negative, arousal – whether the speaker is excited or calm, dominance – whether the speaker is weak or dominant). Please also refer to general response for more detail.

(2) The paper does not compare the proposed method with the latest state-of-the-art methods for HAS...... The most recent method A-CNF was proposed two years ago.

Human annotation simulation (HAS) is a new task setting proposed in our paper, for which no directly applicable prior method is available. All baselines considered in our paper are adapted from suitable existing methods from other domains. In addition, although we cite (Malinin & Gales, 2018) and (Amini et al., 2020) for DPN and EDL in the main paper, they are actually adapted from variants proposed in recent papers (Wu et al., 2022b) and (Wu et al., 2023) which are cited in Appendix E.

The paper does not provide a detailed analysis of the robustness of the proposed method.

The proposed method is evaluated for four different real-world tasks (three in the paper plus one newly added). Results reported in Table 1-3 are the mean and standard error from three independent runs with different random seeds, which shows the robustness of the proposed method.

Ensemble achieves the best performance in Table 1 and Table 2. The proposed method is much more efficient than the ensemble method. Could you please provide a detailed complexity comparison?

The complexity of all methods for the three tasks studied in the paper and one newly added task is shown in Table (i)-(iv) in the general response (2). Please refer to general response for more detail and analysis.

In addition, we’d like to clarify that achieving the highest accuracy in Table 1 and Table 2 doesn’t indicate that the ensemble has the best performance. As explained in Section 2.1, due to subjectivity in human interpretation, deriving a deterministic “ground truth” in subjective tasks is not feasible. Therefore, we advocate for modelling the label distribution to account for human perception variability instead of only predicting the majority opinion. Various metrics are adopted in the paper to evaluate the performance of distribution matching and human variability simulation, as described in Section 4. The proposed method outperforms all baselines by yielding the smallest $\text{NLL}^\text{all}$ (indicating its superior distribution matching capability) and the smallest $\text{RMSE}^s$, $\mathcal{E}(\bar{s})$, $\mathcal{E}(\hat{\kappa})$ (indicating its superior simulation of inter-annotator variability).

We hope our response resolves your concerns.

We appreciate Reviewer js5b’s positive feedback and insightful comments. We address some general concerns in the general response and response to your specific comments below:

(1) In the experiments, test performances are reported, however, considering the problem as "simulating subjective human annotations", I would expect to see (i) a training to learn the latent model in a labeled training subset, (ii) generate simulated labels on a held-out training subset and (iii) training a classifier only with these simulated labels and evaluating on test set. Instead, the current evaluation is more like supervised evaluation.

The purpose of the HAS system is to simulate annotations that match the distribution of human annotations. Since the underlying annotation distribution is unknown, it’s not easy to directly assess the distribution matching performance by a supervised-learning downstream task. Therefore, we adopted various metrics in the paper to evaluate the distribution estimation performance, as described in Section 4.

As a concrete example, evaluating the performance of HAS with a classifier, which usually requires the training data to have a single ground truth (i.e. majority vote), is not suitable for our modelling purpose, because there might not exist a single ground-truth label in human evaluation tasks.

(2) I found the problem address highly similar to the following paper [1,2] that aims to model the label space in each step iteratively using Gibbs sampling (or previous like of work that used MCMC). This work may require annotation process in a more dynamic setting, still very relevant to subjective labelling task and I think, they are needed to discussed.

Thanks for pointing out these related works. We agree that delving into the differences between these tasks and our method is enlightening.

Paper [2] developed a MCMC method for sampling from a particular subjective probability distribution which allows a person to act as an element of an MCMC algorithm by choosing whether to accept or reject a proposed change to an object. Paper [1] generalises paper [2] to a continuous-sampling paradigm. Despite both modelling subjective distributions, our method differs from these methods in the following two aspects.

Firstly, these methods require an online annotation process where the annotators need to label data points based on the current state of the Markov chain, whereas in our setting the annotations can be obtained offline.

Secondly, in those traditional density estimation frameworks, distributions are estimated based on observations while our meta-learning method does not require observations to be provided. Extending the notation in the paper, we denote an event as $d_i$, which consists of a descriptor (i.e. an utterance) $x_i$ and $M_i$ associated observations (i.e. human annotations) $D_i=\\{\eta_i^{(m)}\\}_{m=1}^{M_i}$. For a test event $d^*$, the test descriptor and observations are denoted as $x^*$ and $D^*$. The target to estimate is the distribution of $D^*$, denoted as $p^*$.

Given an event of interest $d^*$, MCMC-based methods present the descriptor $x^*$ to human participants who are asked to provide sequence of decisions $D^*$ following a Markov Chain Monte Carlo acceptance rule. The distribution $p^*$ is then estimated based on $D^*$. In other words, observations $D^*$ are necessary in order to estimate each subjective probability distribution. Each Markov chain only targets a specific subjective probability distribution and there is no obvious way to transfer information between different Markov chains. Therefore, these methods cannot be applied to simulate annotation distribution for unlabelled data.

An advantage of our method is that only the descriptor $x^*$ is needed to simulate the distribution of event $d^*$ and no $D^*$ is needed which are often unavailable in real-world settings. As described at the beginning of Section 3.1, the proposed approach meta-learns a conditional density estimator across $D_{meta}=\\{D_i\\}_{i=1}^N$.

In other words, given $\\{x_i, D_i\\}_{i=1}^N$, the model is trained to learn how to learn the underlying distribution of $D_i$ given $x_i$. During testing, given the test descriptor $x^*$, the model directly estimates $p^*$ in a zero-shot fashion.

We’ve added this discussion in Appendix M in the revised manuscript.

The rest of your comments will be addressed in Response (2).

We appreciate Reviewer szvo's thoughtful comments. We address some general concerns in the general response and response to your specific comments below:

First, we’d like to clarify that the motivation of the paper is to model label distribution rather than correcting the labelling bias. Human perception and interpretation is subjective. There may not always exist a single “ground truth” for tasks that involve human evaluation (i.e., how expressive the synthesised speech is? What’s the correct score for an ICLR paper review?). In those tasks, multiple human annotators are commonly employed to label each sample (e.g., each ICLR submission is assigned at least three reviewers). The provided labels can be different. The difference (“bias”) reflects different human perceptions of the same event. No particular perception is correct nor wrong and the difference is valuable. As stated in Section 2.1, we propose modelling annotators’ subjective interpretations rather than seeking to reduce the variability in annotations by enforcing a single correct answer.

(1) The whole motivation of paper is that it should be possible to capture the data bias (when majority of labelled samples are wrong), and correct for it in some way. However, in all of the tables, the ensemble (tab 1,2)/gaussian process(tab 3) seem to be performing better than the proposed cnf method. While it is qualitatively visible that CNF is learning a better density distribution, the quantitative numbers dont justify the intuition. Perhaps, the authors would be well served by explicitly identifying the samples which possess such labelling bias, correct for it (by forcing the distribution to skew), and show better performance than all other methods.

As mentioned in the clarification above, there may not exist a single correct answer for such subjective tasks. We thus frame the task as label distribution estimation. Therefore, accuracy/RMSE is no longer the proper metric for the human annotation simulation (HAS) task as these measures can only measure the majority/mean prediction. In order to better assess the model’s ability to model label distribution, we adopted multiple metrics as discussed in Section 4. The quantitative numbers in Table 1-3 indeed justify that the proposed method outperforms all baselines in label distribution estimation. It produces the smallest $\text{NLL}^\text{all}$, indicating better distribution matching, and the smallest $\text{RMSE}^s$, $\mathcal{E}(\bar{s})$, $\mathcal{E}(\hat{\kappa})$, indicating a superior simulation of inter-annotator variability. Besides, we have added an additional metric ICC following Reviewer js5b’s suggestion, for which our method also outperforms all baselines.

(2) What is the motivation behind latent-diffusion model. in my understanding, the marginal p(y|z) is already encoding information on different annotators z1,z2....z_m sampled over z. what does additional variable v inject into the model conceptually (apart from additional representational capacity).

Regarding variable $v$. This is because the flow-based model is an invertible transformation, as mentioned in the paragraph above Eqn. (2), which cannot directly transform the continuous latent variable $z$ into discrete output $y$. Therefore, $z$ is first transformed into $v$ (continuous) with a flow and then discretized into $y$ (discrete).

(3) What is the reason for separate analysis of ordinal and categorical variables? I understand that ordinal categories are ordered, and that probability estimation then reduces to summing over the continuous space of latent variable v. However, I haven't seen standard classification setups explicitly enforce such ordering constraints. Also are there any cases where annotations are continuous (for eg, annotation by a speech etc.), which could be used to explore continuous cases? That could act as an interesting toy experiment.

Regarding separate analysis of ordinal and categorical variables. You are correct that the standard classification setups cannot explicitly enforce ordering constraints. That’s why we conduct separate analysis of ordinal and categorical variables.

Ordinal rating schemes (e.g., the 5-point Likert scale) are commonly employed by human evaluation tasks (e.g., the speech quality assessment task in the paper, ICLR review) . Directly applying standard classification setups to ordinal annotations would result in poor performance due to model mis-specification. Therefore, we propose I-CNF to separately handle such cases.

Continuous annotations can be handled by the identity transformation $p(y|v)=\delta(y-v)$ as discussed at the bottom of page 3. In addition, we’d like to point out that continuous annotations are less common in human evaluation tasks (e.g., ICLR reviewers would not rate papers with a score of 6.12).

The rest of your comments will be addressed in Response (2).

Tables for computational time cost

Due to training complexity, the number of annotations to be simulated $M$ is set to 10 for ensemble and 100 for all other methods in the following tables.

Table (i) Emotion category annotation

Table (ii) Toxic speech detection

Table (iii) Speech quality assessment

Table (iv) Emotion attribute annotation