Multi-environment Topic Models

In Equation 1, should σbe revised to α?

Yes, thank you for pointing this out!

In lines 286-287, the authors mention that they test all models on three held-out datasets, but I can not find the statistics of these held-out datasets. Can you provide the testing set size in Table 10?

The test set is 10% of the training set size. We will incorporate this information in the paper.

Is there a typo in “The style dataset” at the first sentence of Appendix C.4? Should it be revised to “The channel dataset” when referring to Appendix D.1 or “The political advertisements dataset” when referring to Table 10? I’m confused about distinguishing the last two datasets in Table 10.

Yes, thank you for pointing this out!

In lines 306-309, the authors mention that they have test data from a distribution that is unseen during training, which makes them not have access to environment-specific γs. Besides, they cannot use γ when calculating perplexity for out-of-distribution test data. Since I cannot find the corresponding steps in Algorithm 1, can you provide the details regarding the above strategy?

We calculate the MTM performance with γ in none of the experiments, as stated in section 5.4. We only use beta when calculating perplexity. For MTM+γ, we use the combination of both.

I wonder how the proposed method scales with a larger number of environments.

Thank you for this suggestion. We focus on these types of datasets that are frequently used in the social science literature. Please let us know if you have any dataset suggestions.

In Table 6, the most typical word "health" appears in all the Global, Republican, and Democrat. Does this redundancy suggest that the inference is sub-optimal?

It is possible that the inference is sub-optimal in some situations. In section E in the appendix we describe an experiment where we evaluate that a given word w that is highly probable in a certain environment ei and a specific topic k occurs more frequently in documents discussing topic k in environment ei than in documents discussing the same topic in a different environment ej. We find that the MTM does well at this.

Does the environment-specific variation include "negative" variation to counteract the global "top" words in some environments?

Can you elaborate on what you mean? If our reading of this point is correct, we think that our model already does this. In our model, if a word does not appear in an environment, it is less likely to be considered in the global topic beta words.

However, I wonder if the top words in the global topics are still supposed to be shared across all environments because they are in the "global" topics.

Yes they should be. Our point was to emphasize that it is completely possible for a baseline model that captures the environment specific variation in the top words for a topic to have a better NPMI score.

In Section 3, E is not formally defined, and both |E|and e are used for the number of environments. In Section 3, π (softmax?) is used without its definition In Equation (1). σc must be αc. At the end of Page 6, Table 15 must be Table 5.

Thank you for this valuable feedback. We will update the paper accordingly.

It would be beneficial to include a more thorough analysis of sparsity enforcement and the selection of the ARD prior versus other priors. Providing ablation studies or theoretical comparisons with alternatives (e.g., the Horseshoe prior) would strengthen the rationale behind this choice and clarify its role in enhancing MTM’s performance, as suggested by the evaluation metrics.

Section E in the appendix provides a detailed comparison of the MTM, horseshoe MTM, and nMTM. We prefer the MTM because it is better than the baselines at identifying terms that are unique to a particular while also being more sparse.

Although known techniques are employed, incorporating a detailed derivation of the ELBO alongside the algorithm would be beneficial. Topic models are widely used outside of machine learning, and a clearer explanation could make the paper accessible to a broader audience.

Thanks for the suggestion. We will add this derivation to the appendix.

Expanding the experiments beyond political and media datasets could illustrate MTM’s versatility and applicability. Datasets from scientific literature or product reviews, for example, could demonstrate the model’s adaptability and provide insights into its generalization capabilities across domains.

That’s an interesting idea! What is the use-case you envision? Our focus is on political datasets because they are commonly used by social scientists to estimate causal effects.

The presented generative process and inference approach is novel. However, the family of topic models is extensive, and alternative baselines might provide a more comprehensive comparison. For instance, MixEHR [1,2] uses CVB and could treat each specialist type as an environment, could serve as an interesting baseline for MTM, and highlight the model’s strengths.

These are interesting papers. MixEHR has a different goal from the MTM. It focuses on modeling the heterogeneity of each patient, while the MTM aims to learn a global representation while capturing the environment specific effects on the global topic representation.

Differential Topic Models

We are interested in learning a global topic distribution; differential topic models are good at finding topics that vary across different data collections but do not contribute to our goal of learning a unified topic distribution for each topic that can be used in downstream causal experiments. DTMs share some similarities, but are relatively limited compared to other metadata models such as the structural topic model or SCHOLAR, which we include. Using DTMs instead of MTM would lead to unbiased causal estimates in our setting, which is motivated by real social science use-cases.

ContraVis: Contrastive and Visual Topic Modeling for Comparing Document Collections

ContraVis classifies entire topics into local or global categories, allowing for contrastive analysis of differences and similarities between document collections. In contrast, the MTM learns a unified, global version of each topic, with a parameter representing the environment-specific adjustments on that global distribution. This global topic distribution in MTM supports causal inference by enabling consistent topic proportions for documents across different environments, rather than each document from different environments having different topics assigned to it.

It is not surprising to see the perplexity improvement, given that the proposed model makes use of extra supervised information, i.e., the environment indicator variable.

More information can lead to perplexity to improvements; however, sparsity plays a more important role in improving model performance. The nMTM includes a non-sparse prior and consistently performs worse than the MTM, and even worse than the VTM on some experiments despite the VTM containing no environment specific information.

There is a lack of comparison with those models that are capable of modelling different corpora, like these two models mentioned above.

The SCHOLAR is an example of a model that is widely used by social scientists that models environment metadata.

Questions

Thank you for your questions. π() represents the softmax and σc should be αc in eq.1.