Measuring Diversity in Synthetic Datasets

We appreciate the reviewer's recognition of our work. We respond to the reviewer’s question as follows. Limited by the space, we present our additional experiments in an anonymous URL (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).

---

Q1: It would be better to add more investigation of failure cases and limitations

R1: Thank you for your suggestive comment. One of the primary limitations of DCScore is its inapplicability to multimodal data. This limitation arises from the challenges associated with feature extraction and alignment across different modalities, which can affect the calculation of the classification probability matrix in DCScore. We will include a detailed discussion of this limitation in future versions of our paper and explore potential solutions in our future research.

Q2: It would be better to add more discussion of the hyperparameter sensitivity of the classification temperature τ.

R2: We would like to clarify that we have conducted sensitivity experiments on the classification temperature $\tau$ in Section 5.4 of our paper. These experiments explore how different values of $\tau$ influence the classification resolution. Specifically, a lower $\tau$ enhances the discriminative ability of DCScore to different samples.

Q3: It would be better to add diversity evaluations on more real-world synthetic datasets beyond those augmented by LLMs.

R3: Thank you for your suggestion. We have conducted several experiments across different existing text and image datasets. Specifically, in Figure 4 of the anonymous URL, we present diversity scores for the AGNews/SST/Yelp_A.P. datasets augmented by AttrPrompt[1]. In Figure 2 of the anonymous URL, we show results for image data. Overall, DCScore shows strong correlation with baseline methods.

Reference: [1] Large language model as attributed training data generator: A tale of diversity and bias, Neurips 2023.

Q4: From my experience, it would be better to add discussion of connections to other classification-based metrics in machine learning beyond just diversity measurement.

R4: Thank you for your thoughtful review comment. We will include a discussion on classification-based metrics in machine learning in future versions of our paper. To the best of our knowledge, we have identified classification-based methods primarily within the domain of metric learning. To further enhance our work, we would greatly appreciate any additional guidance or references within this topic.

Q5: The paper should discuss recent work on classification-based evaluation metrics, particularly "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" (Ribeiro et al., 2020) which provides relevant insights about classification-based evaluation.

R5: Thank you for your suggestion. This paper (Beyond Accuracy: Behavioral Testing of NLP Models with CheckList) primarily focuses on guiding users in conducting Behavioral Testing of NLP Models, but it appears to not mention classification-based evaluation metrics. However, we recognize the relevance of this work to our paper and will include a discussion of its insights in the related work section of the future version.

Q6: The main weakness is limited discussion of the method's applicability to non-text modalities.

R6: Thank you for your valuable review. We conduct experiments on the image modality (colored mnist dataset), please refer to Figure 2 of the anonymous URL. We follow the setting of [1] and observe that DCScore presents higher correlation with the label number compared to VendiScore.

Reference: [1] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024.

Q7: 1.How does the method perform on multi-modal synthetic data where text is just one component? 2.What is the sensitivity of the method to the choice of classifier architecture beyond the examined options?

R7: For 1, our method has limitations in evaluating multi-modal synthetic data, which is an area we plan to explore further. Specifically, the effectiveness of DCScore in accurately evaluating multi-modal data depends on the extraction of multi-modal representations and the alignment of different data modalities. We believe this is a very worthwhile research direction and appreciate your guidance on this matter.

For 2, in addition to the factors we have already explored, we believe the sensitivity of the classifier architecture lies in its ability to effectively distinguish differences between samples. The fundamental requirement for our method to function correctly is that the classifier can accurately identify distinct samples.

We thank the reviewer for reading our paper. We respond to the reviewer’s question as follows. Limited by the space, we present our additional experiments in an anonymous URL (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).

---

Q1: The claim about DCScore's correlation with human judgment would be strengthened with more details on the human evaluation protocol and inter-annotator agreement.

R1: Thank you for your valuable comment. We will update the appendix in the subsequent version of our paper to include details of the human evaluation protocol. Human evaluation data were generated at six temperatures, with five results per context (prompt) in each subset. During the evaluation, annotators were asked to select the more diverse subset from pairs of subsets. Across six temperatures, this resulted in 15 comparisons, and with three annotators, a total of 45 judgments were made. Subsets were ranked by the frequency of being chosen as more diverse. This process was repeated five times with different contexts to derive the final human diversity ranking.

Q2: The claim about correlation between DCScore and downstream task performance could benefit from more extensive experimentation. Additionally, it seems like the number of epochs was not fixed for this experiment.

R2: We** have included some downstream task experiments in Appendix E.2. In Table 7**, except for the scenario with $\tau_{g}=1.2$(360 epochs), all other results were obtained with 120 epochs. The higher dataset diversity, e.g. datasets with $\tau_{g}=1.2$, required more epochs for model convergence, with details in Appendix E.2. Figure 8 also presents results for $\tau_{g}=1.2$ at 240 and 120 epochs.

Q3: A comparison with reference-free diversity metrics from other domains (e.g., ecology) could have further contextualized DCScore's advantages.

R3: Our method is primarily concerned with evaluating the diversity of synthetic datasets. We believe that comparisons with methods from more closely related domains might carry more conviction. The axiomatic requirements for diversity evaluation can vary significantly across different fields, and using metrics from ecology to assess text or image datasets might result in an unfair comparison. Additionally, the main idea behind the Vendi score method originates from ecology, which can be considered an indirect comparison with metrics from other fields.

Q4: Limited discussion of potential failure cases or situations where DCScore might not accurately capture diversity.

R4: The main limitation of our method is that it is not applicable to multimodal data. Due to space constraints, please refer to R1 of the Response to Reviewer QL32 for more details.

Q5: The batch evaluation protocol is appropriate but could introduce bias if contexts vary significantly in diversity potential.

R5: The batch evaluation protocol was chosen to ensure that the diversity pseudo-truths are accurately reflected when generating multiple samples from the same context. This approach helps maintain consistency in evaluating the diversity of generated datasets. We adopted this evaluation strategy to conduct correlation experiments, and under this setting, bias is not introduced.

Q6: The paper appropriately tests on both self-generated and publicly available datasets, but the dataset sizes (100 samples per sub-dataset) are somewhat limited for diversity evaluation.

R6: As shown in Q8, we conducted experiments on datasets with sizes reaching up to 64k samples. The smaller datasets, consisting of 100 samples per sub-dataset, were specifically designed for batch evaluation protocols (Please refer to R5 for more details).

Q7: Limited experimental evaluation across diverse domains beyond text classification and story completion.

R7: Thank you for your suggestion. We provide experimental results on image data (colored mnist dataset), please refer to Figure 2 of anonymous URL. DCScore presents a higher correlation with the label number compared to VendiScore.

Q8: Computational results focus on small to medium datasets (up to 64k); scalability for extremely large datasets remains unproven.

R8: We provide experimental results on extremely large datasets (up to 120k) in Figure 6 of the anonymous URL. Notably, DCScore exhibits similar changing trends with the baseline methods across these large-scale datasets.

Q9: Have you explored whether DCScore can be used for diversity-aware sampling from generative models, beyond just evaluation of already-generated datasets?

R9: Our method is similar to approaches like the Vendi score, thus it can be applied to diversity-aware sampling from generative models. However, we are more focused on scenarios of synthetic data diversity evaluation. We also provide a detailed discussion of potential application scenarios for DCScore in Appendix A.2.

We thank the reviewer for reading our paper. We respond to the reviewer’s question as follows. Limited by the space, we present our additional experiments in an anonymous URL (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).

---

Q1: it does not sufficiently justify why treating each sample as a separate category is a robust measure of diversity. the paper should provide that classification-based diversity correlates well with entropy-based or clustering-based diversity measures.

R1: Thank you for your insightful comments. We have indeed provided both experimental and theoretical validation to support DCScore as a robust measure of diversity. We offer empirical validation demonstrating a strong correlation with the entropy-based method, as shown in Figure 5 at the anonymous URL.

Theoretically, we have proven that DCScore satisfies several axioms that an ideal diversity evaluation method should meet, as detailed in Section 4.2 and Appendix B of our paper.

Empirically, we have demonstrated that DCScore has a high correlation with multiple diversity pseudo-truths, such as generation temperature $\tau_{g}$, human evaluation, and LLM evaluation, as shown in Section 5.2. We believe these pseudo-truths more accurately reflect the true diversity of datasets.

Q2: VendiScore can be optimized to O(d 2n)using low-rank approximations, which the authors do not address.

R2: As shown in Table 2 of our paper, VendiScore can achieve $\mathcal{O}(d^{2}n)$ complexity when using a linear kernel, such as Inner Product. We have provided a detailed analysis of this in Section 4.3. However, in practical evaluation scenarios, the diversity of synthetic datasets often requires more complex kernels beyond linear ones.

Q3: 1.Classification probability correctly represents diversity, but this assumption is not tested against other diversity definitions (e.g., entropy-based measures). 2.No test on datasets with strong semantic similarity.

R3: For 1, DCScore and entropy-based or clustering-based measures operate on fundamentally different principles, making it infeasible to validate one method's assumptions within the framework of another. Limited by the space, please refer to R4 for more details. For 2, as shown in Figure 1 of anonymous URL, we conduct evaluation experiments on datasets with strong semantic similarity and observe that DCScore performs well in this scenario.

Reference: [1] Ospanov et al., Towards a Scalable Reference-Free Evaluation of Generative Models, NeurIPS 2024.

Q4: If classification probability does not fully represent diversity, then the theoretical guarantees may be incomplete. The author should add mathematical proofs for why classification probability is a good diversity metric

R4: Thank you for your suggestion. For diversity metrics, there is no ground truth or golden rule, only characterization through axioms. Therefore, in Section 4.2, we demonstrate that DCScore satisfies these axioms, indicating that it is a good diversity evaluation metric. From the theoretical perspective, Diversity evaluation aims to capture dataset richness by identifying sample relationships, which aligns with the classification perspective of DCScore. Thus, a sample-level classification probability can fully represent diversity.

Q5: There is no ablation study on how different embedding functions affect DCScore, the author should analyze how different embeddings (e.g., SBERT, CLIP) affect DCScore

R5: We would like to clarify that we have conducted experiments on the impact of embedding functions, and the results are presented in Appendix E.3 and Table 11. DCScore demonstrates strong performance across various embedding functions.

Q6: The supplementary material includes additional experimental details but lacks deeper theoretical analysis.

R6: Thank you for your valuable feedback. In Appendix B of our paper, we provide a theoretical analysis of the properties satisfied by DCScore, demonstrating that it adheres to the axioms expected of an ideal diversity evaluation method. Additionally, in Section 4.3, we offer a theoretical analysis of the computational complexity of DCScore.

Q7: The paper does not mention entropy-based diversity measures, which are commonly used in active learning and generative models.

R7: Thank you for your suggestion. We believe the omission may stem from our different taxonomy of existing methods. According to our classification, entropy-based diversity measures fall under Transformation-based Methods. We will clarify this distinction in future versions of our paper and include a more detailed discussion of entropy-based methods.

Q8: Weaknesses: Highly sensitive to how categories are defined

R8: We would like to clarify that DCScore is not sensitive to category definitions. It doesn't require predefined categories and the category count is the dataset sample size.

We thank the reviewers for providing detailed review on our submission. We respond to the reviewers’ concerns and questions one by one. Limited by the space, we present our additional experiments in an anonymous URL (https://anonymous.4open.science/r/ICMLRebuttal_DCScore).

---

Q1: Relying solely on the trace of P may obscure important structural details. I expect a more in-depth analysis between Vendi Score and DCScore.

R1: A more in-depth analysis is as follows:

• Computational Efficiency: As shown in Table 2, with a general kernel, DCScore has a complexity of $\mathcal{O}(n^2)$ during summarization, while VendiScore has $\mathcal{O}(n^3)$, making DCScore more efficient.
• Optimization Stability: As an optimization target, the trace of P (DCScore) provides simpler gradients, enabling more stable optimization. In contrast, Vendi Score can have gradient issues due to identical eigenvalues.
• Suitable for highly diverse scenarios: DCScore highlights distinct samples (Please refer to R5), while Vendi Score may underestimate diversity. Thus, our approach aligns with the trend of generating diverse data.

For more advantages of our method, please refer to R4.

Q2: There are several related works on reference-based methods that seem to be missing such as [1, 2]. There are additional works [3, 4] addressing complexity challenges that are not cited.

R2: Thank you for your suggestion. We will include the discussion of the related works you pointed out in the revised manuscript. For paper 3, the complexity of $\mathcal{O}(n^2)$ only considers the calculation of the Frobenius norm and excludes the specific kernel calculation due to varying complexities across different kernels (e.g. RBF kernel:$\mathcal{O}(n^2d)$, Graph Kernels (random walk):$\mathcal{O}(n^3)$).

Q3: The authors need to mention the similarity of Equation 4 , as it is similar to the contrastive learning literature, such as Equation 1 of the SimCLR.

R3: Thank you for your suggestion. The core concept of contrastive learning—distinguishing samples—is similar to DCScore's classification perspective, reflecting a common idea across fields. We will highlight this in the revised manuscript.

Q4: The paper should clarify whether the classification perspective yields benefits beyond computational efficiency. Could the authors elaborate on what new insight this perspective can bring?

R4: As noted in Papers 3 and 4, entropy-based metrics like Vendi Score face high computational costs, underscoring the need for efficiency. Additionally, DCScore offers some new insights:

• A novel perspective closer to the essence of diversity evaluation (identifying sample differences).
• More stable optimization due to its simpler gradient, unlike Vendi Score's instability from eigenvalue calculations [1].
• A clearer interpretation of sample uniqueness compared to entropy-based metrics.
• Sensitivity to unique samples via Softmax and Trace operations, aiding outlier detection and reducing model impact.

Reference:[1] Eigenvalue optimization[J]. Acta numerica, 1996.

Q5: The paper needs to illustrate scenarios where Vendi Score fails to capture diversity correctly, but DCScore succeeds.

R5: Thank you for your suggestion. In datasets with distinctive samples, Vendi Score fails, but DCScore succeeds. In this regard, eigenvalue distributions of the kernel matrix can be highly uneven (for example, some eigenvalues are much larger than others), leading to a significant reduction in Shannon entropy. Consequently, the Vendi score yields a diversity estimate much smaller than the actual sample size. In contrast, DCScore accurately estimates the diversity of a dataset based on classification probabilities.

Q6: 1. Could authors provide a more detailed explanation of why such a choice is a suitable metric of diversity? 2. Could authors compare the effect of replicas on Vendi Score and DCScore?

R6: For 1, as noted in R1, R4, and R5, DCScore is more computationally efficient and serves as a more stable optimization target. As theoretically shown in Section 4.2, DCScore satisfies the axioms required for an ideal diversity evaluation method. For 2, DCScore does not fail in the presence of repeated samples. According to Section 4.2 (Effective Number), we provide an proof regarding the evaluation under repeated samples. Furthermore, we provide an evaluation of repeated samples in Figure 1 of anonymous URL.

Q7: Can the authors provide a computational complexity comparison between DCScore and the mentioned related works [3, 4] as they also provide an efficient computation of Vendi Score?

R7: We present a comparison in Figure 3 at the anonymous URL. DCScore outperforms efficient methods in terms of computation time on the AGNews and SST datasets. Notably, the efficient computation method from Paper 4 is incompatible with non-shift-invariant kernels, which don’t meet diversity evaluation requirements.