An Online Learning Approach to Prompt-based Selection of Generative Models

We thank Reviewer WovK for his/her time and thoughtful feedback on our work. Here is our response to the reviewers' comments and questions.

1. Performance of Lin-UCB compared to general Kernel-UCB

If we correctly understood the reviewer's comment, Reviewer WovK suggests using a linear map to learn the relationship between the CLIP embedding $c_T$ of text $T$ and the CLIPSCore of the text-to-image model for the input prompt $T$. We believe this proposal will not generally address the task, and the choice of non-linear kernels is not just due to the max operator in the definition of CLIPScore.

To understand the necessity of considering a non-linear model, note that the CLIPScore of a text-to-image model $G$ for text input $T$ with CLIP embedding $c_T$ is \text{CLIPScore}_G(T)= \max\bigl\\{ 0 , 100\cdot\langle c_T, G(T)\rangle\bigr\\} In this definition, the term $\langle c_T, G(T)\rangle$ is not linear in the CLIP embedding $c_T$, because the output of the text-to-image generator $G(T)$ depends on the text embedding $T$ in a potentially nonlinear way. In fact, due to the complexity of the text-to-image model $G$, one can expect the $\text{CLIPScore}_G(T)$ to be highly non-linear as a function of the CLIP embedding $c_T$.

Consistent with the above intuition, we observed that the linear model in Lin-UCB could perform considerably worse than the non-linear model of a general Kernel-UCB method. Please note that the baseline named SCK-UCB-lin in our previously submitted draft indeed refers to the Lin-UCB. Our numerical results in Figures 2-7 on the comparison between Lin-UCB and SCK-UCB with nonlinear kernel functions all suggest that non-linear kernel-UCB can perform better than Lin-UCB.

2. SCK-UCB-poly3 and comparison to Lin-UCB

We would like to clarify that the SCK-UCB-lin algorithm (i.e., SCK-UCB with linear kernel given by $k_\text{lin}(x,y)=x^\top y$) in the experiments is the same as the Lin-UCB method. Additionally, the SCK-UCB-poly algorithm is SCK-UCB with the standard polynomial kernel of degree 3, i.e, $k_\text{poly3}(x,y)=(1+\gamma\cdot x^\top y)^3$. We have clarified the kernel definitions in the revision.

3. Typos in writing

We thank the reviewer for pointing out the typos in the paper.

• Variable $s$ in Section 3.2: We note that the variable $s\in\mathbb{R}$ is the target for kernel ridge regression, which can be CLIPScore (or another performance measure) in the setting of prompt-based model selection.
• Remark 1: We have revised Remark 1 in the updated draft.
• O2B metric: To address the reviewer's comment, we have clarified in the revised text that O2B refers to the outscore with respect to the best single model.

4. Findings from the experiments

The main finding of our numerical experiments is the improvement of the proposed contextual bandit SCK-UCB algorithm over the one-arm oracle baseline that always chooses the model with the maximum averaged CLIPScore. This result means that the online learning algorithm can outperform a user with side-knowledge of the single best-performing model, which is made possible by a prompt-based selection of the model. This finding supports the application of contextual bandit algorithms in the selection of text-based generative models.

Moreover, our numerical results indicate that the proposed SCK-UCB algorithm can perform better with a non-linear kernel function. Finally, in our experiments, the proposed RFF-UCB variant could reduce the computational costs of the general SCK-UCB algorithm. We have explained these findings more clearly in the revised text.

5. Computation time

To address the reviewer's question, we have compared the running time of RFF-UCB and SCK-UCB using the RBF kernel in Part 3 of Appendix A. The results show that RFF-UCB can achieve 5x acceleration compared to SCK-UCB with Gaussian RBF kernel.

We thank Reviewer 7U2V for his/her time and thoughtful feedback on our work. Here is our response to the reviewers' comments and questions.

1. Scalability of RFF-UCB

We believe that our proposed online learning approach can meet the scalability requirements of a typical generative AI user. In practice, users often have specific interests. For example, one user may mainly focus on sports and generate sports-related content. Our online learning approach is designed to efficiently adapt to these user-specific interests, by identifying the best generative model for prompts relevant to the user.

For example, in the case of a user interested in sports, the algorithm adaptively learns the model assignment rules for sports-related prompts. This adaptation reduces computational and statistical costs by avoiding the need to find model assignments for all possible prompts, many of which would be irrelevant to the user. Therefore, by focusing on the user’s specific interests, our online learning method provides an efficient solution to the prompt-based model selection task.

2. Selection of hyperparameters

As requested by the reviewer, we have included the ablation study in Part 2 of Appendix A. We test 1) RFF-UCB with different selections on the hyperparameter in the RBF kernel function and the number of the random features, and 2) SCK-UCB-poly3 with different selections on the hyperparameter in the polynomial kernel and the regularization parameter in KRR. Our numerical results show that the proposed algorithms achieve consistent performance across different selections.

3. Adaptation to new models and prompt types

As requested by the reviewer, we included two numerical experiments in Section 6, where new prompt types and generators are introduced after initial deployment. The results show that the proposed SCK-UCB-poly3 algorithm works well in these scenarios.

We thank Reviewer T1Qe for his/her time and thoughtful feedback on our work. Here is our response to the reviewers' comments and questions.

1. Discussion on related work

We thank the reviewer for pointing out the related works. We note that the mentioned references concern the online training of generative models, which is a different task from ours on the online selection among a group of already-trained generation models to reach the best performance in response to an input prompt. We have discussed these references in the revised draft.

2. Additional baselines

As requested by the reviewer, we included two additional baselines in our testing of the proposed kernel UCB algorithm, which are included in the plots of our revised draft. The baselines include: 1) the reviewer's suggested pure random selection of the model, and 2) the one-arm oracle baseline where the user has side-knowledge of the averaged CLIPScore of each generative model and always picks the model with the maximum averaged CLIPScore. Our numerical results in the revised draft indicate that the proposed SCK-UCB can perform better than both these baselines.

3. Novelty in the technical contribution

First, we would like to clarify that our proposed SCK-UCB algorithm is not a special case of the kernelized UCB approach in [1]. Note that the kernelized UCB [1] assumes a shared weight for the model's score function to an input and the context variable is different across arms. In contrast, in our setting, the score functions are different between arms, as the generative models can perform differently in response to the same prompt. Also, the context variable (text prompt) is shared between models in our formulation, which is also different from the kernelized UCB [1].

Since the problem formulation in our case and the existing kernelized UCB literature are different, none of our theoretical regret bounds will follow from existing results in the literature. In other words, the formulated problem setting, proposed algorithms SCK-UCB and RFF-UCB, and the regret bounds are all novel results which, to the best of our knowledge, have not been discussed in any existing work.

4. Novelty in the Proofs

To highlight the novelties in the paper's proofs,

• The existing regret analyses in the literature do not apply to random Fourier features for approximating a shift-invariant kernel in the kernel-UCB setting. Our regret analysis in Theorem 2 addresses this task.
• The existing results cannot be applied to derive finite-sample error bounds on both the estimated mean score $\widetilde{\mu}_g$ and the uncertain quantifier $\widetilde{\sigma}_g$ (lines 8 and 9 in Algorithm 3). We establish the error bounds in Lemmas 5 and 6 in Appendix C.1.

[1] Michal Valko, Nathaniel Korda, Remi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time analysis of kernelised contextual bandits, 2013.