What's New in My Data? Novelty Exploration via Contrastive Generation

Thank you very much for your comments and feedback. We have revised the manuscript based on your feedback, with the changes highlighted in red color.

---

The application of this question seems limited, because for real-world cases, i.e. confidential fine-tuning datasets, we might not know which pre-train model they are using. Also we may not be able to access the probability of fine-tuned model.

We address a scenario where model developers or analysts have access to both the pre-trained and fine-tuned models but do not have direct access to the fine-tuning dataset. This situation is common in industry, especially when subcontractors develop language models (LMs) using private or confidential data (see references [1, 2], where it is good practice that subcontractors are restricted to access the dataset ). For instance, developing a customer-care chatbot may involve fine-tuning on customer-service interaction data, to which developers are restricted from direct access. In these cases, insights about the dataset must be drawn indirectly from the model’s outputs.

We would like to emphasize that our approach is easy to use and does not require any preprocessing (e.g., building an n-gram index or clustering), which is important from a practical perspective. This may make our approach a method of choice even in scenarios where access to the data is available.

[1] Garrido et al., Lessons learned: Surveying the practicality of differential privacy in the industry. Proceedings on Privacy Enhancing Technologies, 2023.
[2] Sarathy et al., Don’t look at the data! how differential privacy reconfigures the practices of data science. CHI, 2023.

---

Writing about evaluation on novel examples is not clear.

The evaluation process for different domains (i.e. Non-English and toxic) is not clear. Which domain is chosen as the ground truth for first iteration? How the accuracy is calculated based on the 100 generated texts? They are not clearly stated.

We apologize for the lack of clarity in our initial explanation. For instance, in the non-English setting, we regard 10 non-English languages (e.g., Japanese, Chinese, Persian) as positive examples, with English text considered as negative examples. The detection rate is then calculated as the percentage of positive examples among the 100 generated texts. The coverage rate reflects the proportion of unique non-English languages represented in the generated texts, relative to the total distinct languages (10). For example, if 100 generated texts include 50 English, 30 Japanese, and 20 Chinese texts, the detection rate is 0.5 [(30+20)/100], and the coverage rate is 0.2 [2/10]. The coverage metric approximates the number of novel domains an analyst can detect within a limited time frame. We clarified this in Section 4.3.

---

For the iterative version, how to make sure you fine-tune the pre-trained model on generated text can produce a new model with appropreate distribution. i.e. for the first iteration, the distribution of pre-trained model has no problem, and you identify and generate some Japanese text. However, when you fine-tune on Japanese task, it is possible Japanese text will dominent the distribution and English/Russian will be "novel character" and it is possible to generate English again.

Yes, such situations can happen, and the iterative version generates English as “novel” examples. That’s why the detection rate substantially drops for the iterative version, as shown in Figure 2 (left). However, even if such cases happen, the iterative version tends to generate a wider variety of novel examples than the static version, as demonstrated by the higher coverage rate in Figure 2 (right).

Thank you very much for your comments and feedback. We have revised the manuscript based on your feedback, with the changes highlighted in red color.

---

The writing is poor, and the story and motivation are vague and confusing. The authors keep switching between writing "novelty exploration", "novel characteristics", and "novel domains" which makes it difficult to understand what the actual task is. After much digging, this work is basically about identifying example classes that are in the fine-tuning set and not in the pretraining set. Perhaps the title could be, "identifying new classes in the finetuning set".

Thank you for the helpful suggestions. We have revised the paper to use the term “novel domain” consistently throughout the manuscript, to clarify the task. We also edited the introduction to clarify the motivation.

---

It is not clear what the goal of this work is in terms of the benefits of identifying new classes in the fine-tuning set. The introduction mentions that this capability would allow us to remove toxic values, but how would the model know the new classes are "toxic"?

The goal of this work is not to create a fully automated system but rather to equip analysts or developers with a tool that enables them to discover new domains within the data (without direct access to the data). If they detect an undesirable domains, they can take corrective actions, such as adding filters, performing model edits, or even deciding that the model poses too great a risk to deploy. We edited the introduction to emphasize this scenario.

---

There are so many trivial ways to identify whether the model knows whether a certain type of examples were in the pretraining set. What if you simply compute the uncertainty or the entropy of the model output on the examples, and identify those with the highest entropy? those are most likely examples that the model hasn't seen.

Our primary goal in this work is the discovery of new domains using a fine-tuned model, which precludes the use of simpler methods, including those you proposed. We also do not assume that the analyst have access to the data or has a preconception of what specific novel domains (e.g., type of toxic data) may be present.

However, in Section 4.2 (rather than in the main experiments in Section 4.3), we assume access to the fine-tuning dataset and perform sentence selection to showcase the scoring function's effectiveness. Additionally, we have incorporated your proposed method as a baseline in Section 4.2 (see Table 1), following a recent work that uses Shannon entropy and generalized entropy for ood detection [1]. Though they work well in detecting toxic text, they do not perform well specifically for non-English and source code datasets because the entropy of non-English text and source code is not generally high (more details are described in Section 4.2). In contrast, the contrastive score focuses on the difference in the log probability rather than their absolute values, performing robustly across various types of novelties.

[1] Liu et al., Gen: Pushing the limits of softmax-based out-of-distribution detection. CVPR, 2023.

・Results for OpenLLaMA (Table 1):

・Results for Falcon-RW (Table 1):

Thank you very much for your comments and feedback. We have revised the manuscript based on your feedback, with the changes highlighted in red color.

---

The baseline methods are kind of old and weak. It would be better to compare methods proposed in 2023 or 2024, but the baselines used in this submission are proposed before 2021.

The primary focus of our paper is to propose a method that can be used when the fine-tuning dataset is unavailable, and examples need to be generated rather than selected from a dataset (Section 4.3, Table 2). For this setting, there is no prior work, hence we can only use simple ablated versions of the model. We believe that this assumption matches practical setups and is important; introducing this setting is also a contribution of this paper. Moreover, our approach is very simple to use and does not require any time-consuming processing. Thus, it may be preferable even if the access is available, but the extra time necessary to build an index or process the dataset is undesirable.

In Section 4.2, we show that the scoring function underlying our generative approach is sufficiently strong. The goal is to just show that it is robust across various settings, and performs better or on par than simple alternatives. Still, to address your reservation, we added several recent baseline methods, Entropy[1] and GEN [1] in Table 1 (Section 4.2). They employ Shannon entropy and generalized entropy for detecting OOD examples. Though they outperform other methods in their work, they do not perform well in our case, specifically for non-English and source code datasets. This is because the entropy of non-English text and source code is not generally high (details are described in Section 4.2). In contrast, the contrastive score focuses on the difference in the log probability rather than their absolute values, performing robustly across various types of novelties.

[1] Liu et al., Gen: Pushing the limits of softmax-based out-of-distribution detection. CVPR, 2023.

・Results for OpenLLaMA (Table 1):

・Results for Falcon-RW (Table 1):

---

To compare with DP-based methods, the stardard setting is to compare over different \epsilon instead of the noise multiplier. \epsilon indicates the strength of the privacy protection but noise multiplier is not so clear to show the privacy protection performance since the scale of noises varies in different models.

We have added the corresponding \epsilon values for each noise multiplier in Figure 3. The \epsilon values are the same for both OpenLLaMA and Falcon, as we used the same size of the datasets for fine-tuning these two models.

---

The source code should be public accessible.

We made the code publicly available at the following link: https://anonymous.4open.science/r/cge/

---

It is better to bold the best results in Table1. From the experimental results shown in Tables 1, the proposed model cannot always outperform the baselines' results. More detailed and insightful explanations are required.

Thank you for your feedback, we’ve bolded the best results as requested. The primary focus of our paper is to propose a method that can be used when the fine-tuning dataset is unavailable, and examples need to be generated rather than extracted. Table 1 presents preliminary experimentation that the scoring function underlying our contrastive generation method is accurate. In Section 4.3, the scoring function consistently shows strong performance across different models and datasets (AUROC is above 0.9 for all cases, and FPR95 is around 0.1 in 3 out of 6 cases). More importantly, the rest of the methods in Section 4.2 are not applicable to generation, i.e. the main focus of the paper (Section 4.3). We rephrased the introduction to Section 4.2 to clarify this.

Dear Reviewer J5Ry,

Following up on our previous reminder, we wanted to let you know that we have made substantial revisions to the paper, including new experiments aimed at addressing your concerns (e.g., additional baselines and code release as detailed in our previous messages). We would appreciate your feedback on whether these updates address your critiques.

Thank you very much for your comments and feedback. We have revised the manuscript based on your feedback, with the changes highlighted in red color.

---

Missing an honest baseline like n-gram: This paper misses basic baselines such as n-gram distribution change, or top-k nearest neighbour distance in the embedding space.
・This could quite naturally be done in the RPJ experiments.
・Can also be adapted in the DP-SGD experiments by returning privatized n-gram stats.
・You might like the infini-gram paper as a tool to enable such analysis

Both these baseline methods (n-gram distribution changes and top-k nearest neighbor distance) assume access to the fine-tuning and pre-training datasets. These are assumptions that we did not make in our method.

If direct access to the pre-training data (or its index) were available, we agree that this can be advantageous. For most models, however, this access is unavailable. To see what happens when both pre-training and fine-tuning datasets are available, we ran the top-k nearest-neighbor distance measures [1] and showed its performance in Appendix A.4 (Table 5). K-NN performs exceptionally well for non-English text and source code, though it performs worse on the toxic text dataset for Falcon-RW. When both novel and in-distribution examples are available, distinguishing between them becomes trivial since typical English text and novel-domain texts (e.g., non-English text, programming languages, and toxic text) differ significantly, and the latent representations effectively capture these differences.

However, we relax the assumption of having no access to the fine-tuning dataset in Section 4.2. This serves two purposes: it allows us to evaluate the underlying scoring function - the difference in the log probability - in a simpler setup (before using it in generation in Section 4.3), and also to show that our method is competitive even in this setting. Now, we added recent novelty detection methods, Entropy [2] and GEN [2], to these experiments. These methods employ the Shannon entropy and generalized entropy of the pre-trained model’s predictions and do not assume access to the pre-training dataset. As shown in Table 1, they do not perform well in our case, specifically for non-English and source code datasets, because the entropy of non-English texts and source code is generally low (details are described in Section 4.2). In contrast, the contrastive score focuses on the difference in the log probability rather than their absolute values, performing robustly across various types of novelties.

We would like to emphasize that our approach is easy to use and does not require any preprocessing (e.g., building an n-gram index or clustering), which is important from a practical perspective. This may make our approach a method of choice even in scenarios where access to the data is available.

[1] Sun et al., Out-of-Distribution Detection with Deep Nearest Neighbors. ICML, 2022.
[2] Liu et al., Gen: Pushing the limits of softmax-based out-of-distribution detection. CVPR, 2023.

・Results for OpenLLaMA (Table 1):

・Results for Falcon-RW (Table 1):

---

Constraint: The paper writes that one of the constraints for the method to work effectively is that the number of novel examples substantially smaller than the remaining examples in the fine-tuning data. This is a very hard-to-match constraint in my opinion

It was not our assumption, and we rephrased the text for clarity (see Section 2). Our model does demonstrate strong performance when the proportion of novel examples is high (see Table 4 in Appendix A.3). Our objective in experiments was to show that our method remains effective even in the challenging scenario where certain novel domains are ‘hidden’ (e.g., cannot be at all surfaced through random sampling from fine-tuned models). The ability to reveal such hidden domains is crucial, as harmful data - such as toxic content or misinformation - may present in training in tiny amounts yet still compromise model safety.

・Performance of CGE when varying the proportion of novel examples in the fine-tuning dataset (Table 4):

---

No [y-]axis labels on graphs 2,3

The titles shown about the graphs are also the y-axis labels. To prevent any confusion, we have annotated y-axis labels explicitly.

---

No samples shown.

We have included examples generated by our method in Appendix A.6 (Figure 4 and 5). We also made the code publicly available at the following link: https://anonymous.4open.science/r/cge/

---

Can you think of adapting this metric to measure the distance between two distributions?

This is an interesting question. Intuitively, using importance sampling with samples from the renormalized distribution q = renorm(p_ft / p_pt) could serve as an effective proposal for estimating KL(p_ft || p_pt) (or any f-divergence more generally), as it emphasizes examples that contribute significantly to the KL divergence. However, the connection may be less straightforward, as our decoding method does not directly produce samples from q.

Dear Reviewer B4zm,

As we approach the end of this discussion period, we want to thank you for your thoughtful feedback, which has significantly improved our paper. We believe we’ve addressed most of the concerns raised by you and the other reviewers. Specifically, we have revised our paper as follows:

• We have clarified our problem setting, which focuses on the scenario where the pre-training and fine-tuning datasets are not available, and previous novelty detection methods (e.g., kNN and n-gram distribution changes) are not applicable to this scenario (Section 1 & 2).
• We demonstrated that our method robustly performs even in the setting where we have access to the fine-tuning dataset. We have added recent novelty detection methods (Entropy and Gen) to the experiments, where they do not perform well on some datasets (Table 1 in Section 4.2).
• We clarified that our method does not require the assumption that the number of novel examples is smaller than that of in-distribution examples in the fine-tuning dataset. We showed our method performs well even when the majority of the dataset consists of novel examples (Table 4 in Appendix A.3).
• We have added y-axis labels to the graphs and included examples generated by our method. (Figures 4 and 5 in Appendix A.6)

We would appreciate it if you could let us know whether our responses properly address your concerns.

Dear Reviewer B4zm,

Thank you so much for your feedback and for raising your score. We appreciate your recognition of the improvements we’ve made and your further suggestions. We’ll do our best to incorporate them in the next revision.

Thanks again for your time.
the authors.