Topic Modeling as Multi-Objective Contrastive Optimization

Thank you for your helpful reviews! We are happy that you appreciate our novel, effective, and interesting method. We sincerely hope that our response and the rebuttal version of our paper can address your concerns and improve your ratings.

Q1: Do you perform both ChatGPT-based and embedding-based augmentations for comparisons?

We use Word2Vec to generate augmentation examples in Appendix H. In our main experiments in section 5.2 - 5.3, we only use ChatGPT generated examples. We have moved the Word2vec detail to Appendix H to make our work clearer.

Q2: Reason why both MinPool and MaxPool are used.

In Algorithm 1, we use max pooling for negative samples because we want to focus on pushing away strongest features of negative samples and suppress their noise. We use min pooling for positive samples because we want to attract even the weakest features of positive samples.

In section 5.5, we experiment with the pooling function for negative samples. We have compared pooling function choices for positive samples:

We can see that our min pooling provides better aggregated mutual information for positive samples in neural topic modeling. We have added these details in Section 5.5 and Table 10 in Appendix F of the rebuttal revision.

Q3: The average and standard deviation of $\alpha$.

We observe that the values of $\alpha$ vary from the starting epoch to the final epoch. In particular, in the beginning, the mean and standard deviation of alpha are about 0.001. This means that in the beginning, it is better to focus on optimizing the ELBO objective of the neural topic model.

Gradually the value of $\alpha$ increases and converges to approximately 0.3 in the last epoch. This validates the usefulness of our setwise contrastive learning that gives beneficial mutual information signal to topic modeling.

We specify the values of $\alpha$ after epochs for 20NG and IMDb datasets in the following table. We have added $\alpha$ values for all other datasets to Appendix J in our rebuttal revision.

Q4: The first sentence of the Hard Negative Sampling paragraph

Sincerely thank you for your note. We have provided two modifications for Algorithm 1 in our rebuttal revision to more accurately describe our method:

• Include the iteration $p$ that loops over the permutation matrix: We diversify our document sets by permuting the document index within a batch. We have added the index $p$ to the formula using MaxPool and MinPool operations to reflect this (line 10-13 in Algorithm 1).

• Modify the contrastive loss: In our implementation, we consider a negative pair as a document set $i$ with all other document sets $j$ which are generated based on the permutation matrix $M$ (except for the set which includes corresponding augmented documents of set $i$). We have corrected the denominator of our contrastive loss to reflect this (line 16 in Algorithm 1).

Therefore, as in the first sentences of our hard negative sampling paragraph, we put a document into multiple sets and compare among them. This would create hard negative samples and benefit contrastive learning.

Q5: In Algorithm 1, $\text{exp}(s_{i}^{min}, s_{i}^{+})$ should be $\text{exp}(f(s_{i}^{min}, s_{i}^{+}))$

Thank you so much for your note. We have edited this detail in Algorithm 1 (line 16) of our rebuttal revision.

Q6: The value of contrastive weight $\beta$.

We compare between the adaptive scheduling in NTM+CL (Nguyen et al., 2021) and fixing it to 1. However, since the difference is not significant, for simplicity we directly set $\beta$ to 1 (line 16 in Algorithm 1).

Thank you for your insightful reviews! We are delighted that you appreciate our clear, reasonable method, and our rigorous experiments. We sincerely hope our response and the rebuttal version of our paper can address your questions and improve your ratings.

Q1: Details about topic diversity

We clarify that to measure the topic diversity, TSCTM [1] and WeTe [2] extract 25 top words in each topic, while we only use 10 top words, which would make the evaluation harder. If we follow strictly follow their protocol, most of the topic diversity values will be ~0.999, which cannot discriminatively evaluate the topic models.

Q2: Comparison with (Wu et al., 2022 [1]) work

We clarify that the main difference between our work and TSCTM (Wu et al., 2022 [1]) lies in that their work is restricted to short texts, e.g. TagMyNews dataset (mean length: 8 words, max length: 17 words), and GoogleNews (mean length: 5.38 words, max length: 12 words). In contrast, we work with texts of multiple lengths, e.g. AGNews (mean length: 20.19 words, max length: 76 words), 20NG (mean length: 152.21 words, max length: 12419 words), and Wiki (mean length: 130.14 words, max length: 640 words).

We also compare our model with TSCTM in the settings of $T = 50$ topics:

We see that our model outperforms TSCTM on both topic diversity and topic coherence. We believe the reason is that our setwise contrastive learning captures more mutual information than TSCTM in documents of various lengths. Thank you for your comment. We’ve added these to Table 4 and Table 5 of our rebuttal revision.

References:

[1] Wu et al., Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning, arXiv 2022.

[2] Wang et al., Representing mixtures of word embeddings with mixtures of topic embeddings, arXiv 2022.

Dear Reviewer 3szg,

We wish to express our sincere gratitude once again to you for the valuable contributions and considerate feedback. We would like to gently bring to your attention that the discussion phase between authors and reviewers is nearing completion (within 6 hours).

Given the inclusion of the comparison results with related works and our clarification towards topic diversity, we kindly hope our responses can address your concerns and improve your rating. Should you have any further insights to share, we are more than willing to sustain our discussion until the deadline.

Thank you for your helpful reviews! We are happy that you appreciate our well-motivated and novel method. We sincerely hope our response and the rebuttal version of our paper can address your concerns and improve your ratings.

Q1: p-values should be reported.

Thank you for your suggestion. We have conducted a t-test between our method and the second-best method in Table 2-6, i.e. NTM+CL, Linear-$\alpha=0.5$, NTM+CL, NTM+CL, and GAP, respectively. We obtain $p$-values as follows:

• t-test for Table 2:

• t-test for Table 3:

• t-test for Table 4:

• t-test for Table 5:

• t-test for Table 6:

We see that all of the results are statistically significant ($p$-value < 0.05). We believe the reason is that our method better refines the topic representation through setwise contrastive signal and our effective Pareto solution for the NTM. Thank you again for your advice. We have added these to Appendix I in our rebuttal revision.

Q2: Human evaluation is needed.

Thank you for your comment. We have followed previous works [1, 2, 3] to utilize two automatic metrics, i.e. topic diversity and NPMI, to measure topic quality. However, we acknowledge that human evaluation will provide a more rigorous evaluation. We have conducted human rating and word intrusion tasks for 20NG and IMDb on Amazon Mechanical Turk (AMT) between our topics and those of NTM+CL, and obtain the result in Figure 5 in Appendix K of our rebuttal revision.

We see that human judgement also prefers our generated topics than the baseline ones. Our inter-annotator agreement is 0.72, which indicates high agreement. Thank you for your interesting suggestion, we have added these to Appendix K in our rebuttal revision.

References:

[1] Thong Nguyen and Anh Tuan Luu, Contrastive learning for neural topic model, NeurIPS 2021.

[2] Xu et al., Hyperminer: Topic taxonomy mining with hyperbolic embedding, NeurIPS 2022.

[3] Wang et al., Representing mixtures of word embeddings with mixtures of topic embeddings, arXiv 2022.

Dear Reviewer ioHf,

We wish to express our sincere gratitude once again to you for the valuable contributions and considerate feedback. We would like to gently bring to your attention that the discussion phase between authors and reviewers is nearing completion, which is within 1 hour.

Given the inclusion of the results for statistical significance tests and human evaluation, we kindly hope our responses can address your concerns and improve your rating. Should you have any further insights to share, we are more than happy to sustain our discussion until the deadline.