Leveraging semantic similarity for experimentation with AI-generated treatments

Thank you very much for your careful engagement with our paper and the great questions.

Re weakness 1:

• The novelty of the estimator: the key part of our estimator is that we introduce the kernel-based factorization approach to learn the CATE function. This estimator enjoys several advantages: (i) it allows us to learn the representations of treatment and user covariates contributing to the treatment effect. Treatment embeddings are taken in by the treatment kernel to learn a low-dimensional representation. (ii) it is computationally efficient comparing with some deep learning baselines. This naturally extends to an efficient adaptive algorithm that assigns treatments to users in an online manner. (iii) backed with strong theoretical guarantees, including convergence rates for treatment effect estimation and a sublinear regret bounds of an algorithm for the adaptive experimentation setting.
• Below is a table that compares several estimators :

The table demonstrates the advantages of our proposal. We will include the comparison in the updated draft.

Re weakness 2: While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging. To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings.

• First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates.
• Second, we relax the outcome generation process to allow more complex user-treatment interaction.
• Third, we test out a broader set of baselines methods for a more comprehensive understanding.

Specifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\Gamma x + epsilon$, where $\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\Gamma = U \Lambda V^\top$, where $\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and kernel regression. The methods are combined with different estimation strategies: "Z" means only incorporating treatment information, "X" means only include covariate information, and "ZX" means include a (Z,X) vector. The following two tables compare the training and testing RMSEs:

Training MSE

Testing MSE

From the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features.

Re weakness 3: Thank you for the suggestion. We will take the advice and switch to pdf figures.

Thank you very much for the constructive points. We sincerely hope these explanations address your concerns!

We sincerely thank you for your careful review of our paper and insightful questions.

Re weakness 1: Thank you for this suggestion. We have carefully considered the positioning of our paper, and will try to take your advice to reframe the paper. Since we cannot revise the manuscript during the rebuttal period, we hope to propose a new abstract following your suggestion:

The growing popularity of Large Language Models (LLMs) introduces exciting opportunities for digital experimentation, as marketers increasingly combine human-generated and model-generated content to design treatments. A central challenge in this setting lies in representing these complex, high-dimensional treatments in a way that preserves their semantic meaning and enables scalable analysis. In this paper, our primary focus is to learn low-dimensional representations of complex treatments that captures this semantic structure. These representations can then serve as a foundation for downstream applications such as guiding generative models to produce meaningful treatment variants and enabling adaptive treatment assignment in online experiments. To this end, we propose a method called double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an efficient alternating-minimization algorithm to learn these representations from data. Additionally, we propose an adaptive design strategy for online experimentation as an application. We provide convergence guarantees under a low-rank factor model and demonstrate the effectiveness of our approach through numerical experiments.

We will incorporate your suggestions in the next iteration of our paper.

Re weakness 2: We apologize for not clearly stating the assumptions. We listed several assumptions for estimation in Theorem 4.2 and 5.1, including the noise distribution and the low-rank interaction structure. Moreover, as we are studying an experimental setup with no interference, we need SUTVA, unconfoundedness, and a weaker non overlapping assumptions ($e_z(x)\in(0,1)$). We will clarify these in the updated draft.

Re weakness 3, part 1: While we want to demonstrate our proposal on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging. To address this, we consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings.

• First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates.
• Second, we relax the outcome generation process to allow more complex user-treatment interaction.
• Third, we test out a broader set of baselines methods for a more comprehensive understanding.

Specifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of item recommendation are unknown. We consider a semi-synthetic setup: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended. All these embeddings are taken from the knowledge graph embeddings that are included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\Gamma x + epsilon$, where $\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\Gamma = U \Lambda V^\top$, where $\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and kernel regression. The methods are combined with different estimation strategies: "Z" means only incorporating treatment information, "X" means only include covariate information, and "ZX" means include a (Z,X) vector. The following two tables compare the training and testing RMSEs:

Training MSE

Testing MSE

From the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It achieves a comparable performance that is close to XG boost with combined features.

Re weakness 2, part 2: To your second point of using the proposal to guide LLMs to generate treatments, our online experimentation discussion in Section 5 can serve as an example. We use Explore-then-commit strategy to demonstrate how the proposed method can guide treatment optimization: in the exploration stage, our method can be used to learn a good user-system interaction model, and in exploitation, this model can be used to assign the best personalized treatment among a pool of LLM-generated treatments to maximize final rewards. Furthermore, we believe the downstream applications is extensive, for example, to finetune a language model to generate personalized treatment based on the learned representation model. These are saved as future work.

Re Q1: yes, $d = d_1 + d_2$ is the sum of dimensions for user covariates and treatment embeddings. We apologize for the missing notation.

** Re Q2**: for theory, we need the additive noise model and low-rank assumption. Also, for identification, we need SUTVA, weak overlapping and unconfoundedness. We apologize that we did not formally state these, and will add these in the updated draft.

Re Q3: generating treatments by LLMs is used in two aspects: (i) To learn how semantic information and user covariates interact and contribute to the outcome in A/B testings, we can build models in experiments with LLM generated treatments; (ii) it can guide optimizing personalized treatment generation with the learned model. So it's not just a downstream task; instead it is a full pipeline to borrow semantic information in LLM embeddings to understand and optimize treatment generation for maximizing rewards.

Thank you very much for the constructive points. We truly hope these explanations mitigate your concerns.

We sincerely thank you for your careful review of our paper and insightful questions.

Re weakness 1: Thank you very much for the critical comments. While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging. To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings.

• First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates.
• Second, we relax the outcome generation process to allow more complex user-treatment interaction.
• Third, we test out a broader set of baselines methods for a more comprehensive understanding.

Specifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\Gamma x + epsilon$, where $\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\Gamma = U \Lambda V^\top$, where $\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and a kernel regression. The methods are combined with different estimation strategies: "Z" means only incorporating treatment information, "X" means only include covariate information, and "ZX" means include a concatenated (Z,X) vector. The following two tables compare the training and testing RMSEs:

Training MSE

Testing MSE

From the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the interaction matrix is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It achieves a comparable performance that is close to XG boost with combined features.

Re weakness 2: We appreciate the reviewers questions regarding the low-rank assumption for user-treatment interactions.

Our low rank assumption states:

$\text{rank}{(\Gamma^\star)} \le r.$

First, we clarify that we can relax this assumption from a strict low rank to a so-called approximate low-rank structure by characterizing the spectral decay (i.e., the sorted eigenvalue distribution) of the matrix $\Gamma^\star$. This characterization is using the following inequality, where we bound the $l_q$ norm of the eigenvalues.

$\sum \lambda_i^q \le R_q, ~\text{for some } 0 \le q < 1.$

This is a much weaker version of strict low rank assumption, in the literature and has been well studied by references:

• Negahban, Sahand, and Martin J. Wainwright. "Restricted strong convexity and weighted matrix completion: Optimal bounds with noise." The Journal of Machine Learning Research 13.1 (2012): 1665-1697.
• Rohde, Angelika, and Alexandre B. Tsybakov. "ESTIMATION OF HIGH-DIMENSIONAL LOW-RANK MATRICES1." The Annals of Statistics 39.2 (2011): 887-930.

Under this assumption, we can establish the following guarantee in our setting:

$\frac{1}{d_1d_2}||\hat{\Gamma} - \Gamma^\star||_F^2 \le C R_q \lambda_N^{2 - q}$

Strict low-rank assumption is a special case for this when q = 0; which then aligns with our theoretical results in the paper. More generally, it allows a continuously decaying signals, and there is a trade-off between the signal decaying speed and the estimation accuracy: the closer q is to 1, the harder it is to separate the signals and the wider the ERROR bound. Thus, our results continue to hold under this relaxed restriction.

From a numerical perspective, we also added additional experiments to evaluate the performance of the method under the approximate low-rank setting with the MIND experiment in response to your point 1. As we can see from there, when there is low-rank struture, DKRL exploits the strict low-rank structure most effectively, yielding the best generalization. When the interation matrix is high-rank (for example, $q = 0$ which is full rank), DKRL can still effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features.

Re weakness 3: Thank you for this suggestion. In response to your point 1, we have performed a comparison with multiple baselines and estimation strategies, under different rank specifications. DKRL proves to have the advantage of adapting the low-dimensional information in both the embeddings and the interaction matrix.

Thank you very much for the constructive points. We truly hope that these explanations will mitigate your concerns regarding our work.

We sincerely thank you for your careful review of our paper and insightful questions.

Re concern 1: we agree with you that our setup shares many similarities with relevant problems, such as those in recommender systems. The reason we formulate the problem this way is that we are seeing an increasing practice in digital experimentation of incorporating LLM-generated treatments in A/B testing problems such as generating advertisements or email campaigns. The differences from traditional methods are two-fold:

• Traditional recommendation methods typically choose among a fixed catalog of a few thousand items. Yet here, the "treatment" might be any of millions of possible sentences, with various styles or tones the LLM can produce. Designing an experiment to explore that space is qualitatively different from merely scoring existing items. Moreover, LLMs can be prompted to generate new variants on the fly (e.g. refine a headline after mid-experiment analysis), which turns the pipeline into an adaptive experiment rather than a static recommendation.
• Here we are focusing on experimental design settings, where the treatments are "Interventions", not just features. This allows us to ask causal questions such as “What causal impact does wording X have on click-through, comprehension, or satisfaction?” rather than merely correlation questions like “What patterns correlate with clicks in historical logs?” This is guaranteed by unconfoundedness via randomization. By randomizing which LLM‐generated treatment each user sees, we break any link between unobserved user traits and the chosen treatment. That means we can estimate unbiased treatment effects (CATEs), whereas in a pure recommender system, we need to correct for selection bias and confounding issues on observational data.

Re your concern 2: While one could indeed parameterize $g$ and $h$ as neural nets or fine-tune the original LLM (e.g. via LoRA or contrastive learning), our kernel-based formulation brings three key advantages:

1. Nonparametric flexibility, avoiding the need to choose and tune a specific network architecture.

2. Theoretical guarantees on convergence and generalization.

3. Low-dimensional feature extraction, which enhances interpretability of the learned treatment effects.

We agree that a LoRA-based baseline would be a valuable comparison—this is a natural next step, and we intend to explore it in future work. In the present paper, we focus on CATE estimation with a fixed embedder and a downstream online experimentation strategy. We will leave treatment optimization and generation via fine-tuning as exciting directions for follow-up studies.

Regarding concern 3: While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging. To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings.

• First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates.
• Second, we relax the outcome generation process to allow more complex user-treatment interaction.
• Third, we test out a broader set of baselines methods for a more comprehensive understanding.

Specifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\Gamma x + epsilon$, where $\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\Gamma = U \Lambda V^\top$, where $\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and a kernel regression. The methods are combined with different estimation strategies: "Z" means only incorporating treatment information, "X" means only include covariate information, and "ZX" means include a concatenated (Z,X) vector. The following two tables compare the training and testing RMSEs:

Training MSE

Testing MSE

From the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features.

Thank you very much for the constructive points. We hope these explanations mitigate your concerns regarding our work.