Retrospective Learning from Interactions

We thank the reviewer for their helpful comments. We address the concerns below, and following this discussion, we will upload the updated version (the changes are not yet in the PDF).

Response to Raised Weaknesses

Although I appreciate the rationale behind using tangrams, I wonder whether the authors could have tested this approach in more realistic reference game that are well-known in the community such as 20Q game [1], GuessWhat?! [2] or Photobook [3].

MultiRef has several key attributes of language interaction that are not well exposed by the other scenarios you list (although we do consider them interesting and related). MultiRef has a combinatorial solution space. This elicits the multi-turn interaction, with gradual instruction by the speaker and gradual solution construction by the listener. The impact of listener's actions is immediately seen by the speaker. The gradual construction and immediate visibility are essential to elicit the natural implicit feedback we observe. These are properties that are common in natural interactions.

20Q and GuessWhat both are focused on binary questions, in contrast to our instructions that lead to more complex action space. They also both feature a much smaller solution space, which is guessed in one goal after the data is shared, not allowing for the immediate visibility of action impact after each turn. PhotoBook is more related to our scenario, but it also doesn’t provide real-time action visibility, and actions are only visible after both sides submit. More important: PhotoBook is a collaborative scenario where two equal partners negotiate a common ground. MultiRef is an instructional scenario, where the speaker instructs the listener.

Therefore, while the scenarios you list are relevant, their focus is different, and they don’t expose the kind of real-time dynamic behavior we observe in real-time multi-turn interactions – and this is what MultiRef is about.

We will add a discussion of the related scenarios.

It's not clear to me what is the rationale behind using KTO compared to DPO which is more established (e.g., used by Meta for Llama 3.2 tuning). Considering that the authors have access to positive and negative examples, I wonder whether they should try DPO instead considering that has been tested more for VLMs

DPO requires paired preferences, that’s two potential outputs for exactly the same input, one marked as winner and the other as loser. We don’t have this kind of data, so DPO is not applicable. This is why we chose KTO.

It's not clear to me how the authors complete their fine-tuning considering that most of the training regimes are not designed for dialogue data specifically. See Question 1 as well for details.

We create an example for each utterance. Each such example includes the set of images, their selection status, the history of the interaction up to the current utterance, and the utterance. The model output is the action generated by the model. The learning algorithm sees each such “example” as separate. This data formulation is identical to what the model “sees” during deployment.

I think the authors are missing a simpler baseline that is fine-tuned using the final reward (i.e., whether you win or not the game) as a reward as was done in previous work [4].

We computed this (below). However, it’s very important to note that this is not a baseline, but an alternative that uses privileged information that our approach does not have access to, because our approach has no information about task success, but only the human utterances that the model was exposed to. This is a really important design decision, which is critical for applicability to real-life scenarios.

We simulate Interaction-Oracle learning on B-SUP datasets. We report the exact match rate on an offline validation set of 344 turns from human-human games (not exactly in the distribution of human-bot games, but this is informative).

Providing clean and granular progress rewards achieves higher validation rates except for the first round (even though B-SUP has no idea about task success). The model picks up intermediate mistakes in the process by filtering solely based on the final task success. We also overestimate the performance of Interaction-Oracle with offline simulations here, because they were trained on interactions collected from a better model (potential model distillation) – so please consider these numbers an overestimate.

To be continued in Part 2.

Thanks for the clarifications; these are really useful indeed. However, I would like to touch upon a point that does not convince me entirely:

We computed this (below). However, it’s very important to note that this is not a baseline, but an alternative that uses privileged information that our approach does not have access to, because our approach has no information about task success, but only the human utterances that the model was exposed to. This is a really important design decision, which is critical for applicability to real-life scenarios.

I wouldn't consider this as privileged information. If you're playing a game, you always know at the end if you won or lost. So I consider your design choice as an artificial constraint that your model doesn't need to have. So technically using the task success as an additional reward is a totally legit decision which is not accessing privileged information.

Happy to reconsider my score once this is clarified further. I also thank the authors for adding the additional experiment.

We thank the reviewer for their helpful comments. We address the concerns below, and following this discussion, we will upload the edits (the changes are not yet in the PDF).

Response to Raised Weaknesses

The experiments are confined to the MULTIREF scenario with abstract tangram shapes.

How well do you expect RESPECT to generalize to other domains or tasks beyond MultiRef? Have you tested it in any other scenarios? (Question 1)

It’s important to consider the research trade-offs and implications of asking for more scenarios. We deployed multiple variants of our approach to interact with humans over thousands of interactions. This is the only way to study such methods. However, it’s also very expensive and time consuming. Yes, there may be insights to be gained by more scenarios, but would this kind of research be feasible in an academic setting? Where budgets are limited and teams are tiny. The implications of raising the costs even higher is pushing academic research out. The problem we study is important. As a research community, we must strike the right balance to make progress. Our research contributes insights and methods to drive this important direction forward.

As we discuss in the related work, Don-Yehiya et al. showed similar signals exist in scale (unlike us, they didn’t experiment with continual learning and a model bootstrapping itself). We believe their conclusion complements our work, showing that general scenarios elicit similar signals.

There's a risk that the model might overfit to specific patterns of implicit feedback rather than truly improving at the task.

This is an empirical question. Our experiments empirically show consistent improvement in task performance over time, showing the model is improving at the task. We also extensively evaluate the feedback decoding. It’s not perfect, but very effective. There are few false negatives classified by the LLM feedback decoder (see the confusion matrix – Figure 6). False negatives are higher though, suggesting there is room to improve further (only making the approach more effective). The task success rate improved significantly (by 51%) though, even with this imperfect reward decoder.

The paper does not compare RESPECT to other established methods for learning from implicit feedback or continual learning.

We are not certain what other implicit feedback methods the reviewer is referring to. We are happy to review any specific approach.

It’s important to note that continual learning is at times used to describe scenarios where models are adapted to new tasks. We use it in the sense of improving a model on its original task over time. So methods for domain adaptation are not relevant.

With regard to RLHF-ish methods: we experimented with KTO (a utility maximization method), and presented extensive deployment results with it. General RLHF methods and DPO-like methods are not applicable to our domain because they rely on paired preference data, but we have feedback on single outputs. This is why we used KTO, which allows using single-example feedback.

The feedback decoder relies on the model's ability to interpret implicit signals correctly. However, there's no guarantee that the model's interpretation aligns with the human's intended feedback. The paper would benefit from a more thorough analysis of cases where feedback may be misinterpreted and how this affects learning.

We conduct extensive human evaluation to quantitatively assess how well the interpreted signals align with human intended feedback and their impact on learning in Section 6. To summarize:

• The LLM feedback decoder yields negligible false positives (<2%), and consistently reaches >60% accuracy considering all classes (diagonals in confusion matrix). This also suggests the training data for supervised systems are fairly clean (because they rely on positive signals only).
• The false negative rates are higher than false positives (around 15% for binary decoders, and 20% for ternary decoders throughout rounds). As a result, systems using ternary decoders recovered fewer data points as we discard predicted neutrals.
• To assess the impact of the quality of feedback decoders on learning, we adapt both binary decoders and ternary decoders for each optimization strategy. The deployment results (Figure 4) reveal that binary decoders have a slight edge over ternary decoders with B-SUP v.s. T-SUP, suggesting that more data is perhaps more important than cleaner data for supervised learning at least. The benefit is negligible for learning with RL systems.
• The human’s intended feedback is fairly unambiguous, supported by 93% agreement rate among three annotators.

To be continued in Part 2.

Thank you for your feedback. We have modified our manuscript to incorporate it (uploading soon following this discussion). Regarding the concerns you raised:

Response to Raised Weaknesses

Excessive use of training data: The proposed method relies heavily on data. The model is fine-tuned at each step with all the interaction data acquired from past steps. Now, although the authors mention that they are taking measures to avoid overfitting (lines 246-248), this much repeated data usage would eventually result in overfitting.

We train from scratch in every round (6 rounds in total – not every interaction step) for most systems. Training from scratch means: at every round, we fine-tune the initial IDEFICS2-8B parameters. We are not using the data to optimize again and again the same parameters. The measures we took to avoid overfitting are standard, just like any fine-tuning process, and the risk of overfitting is not different from any supervised learning scenario. Our evaluation is on unseen games (every round brings completely new games), showing strong generalization and improvement.

Lack of metric evaluation: Although the authors showcases various observations and results through plots and confusion matrix, they lack the tables for comparing different metics. Having those results will significantly boost the paper quality.

We will add 8 tables in Appendix E accompanying Figure 4,5,6. Figures 4,5,6 illustrate the trends. We included specific numbers in the Results section with analysis. The confusion matrix is already labeled with numbers so we will highlight the key observations in the caption as well.

Lack of generalizability ... across different LLMs: The proposed framework is only tested on one LLM (IDEFICS2-8B). Now, although this framework can be applied over other LLMs, it is unclear whether it will boost their performance or not. One reason authors might have seen such improvement is because the tested LLM is bad at that particular task. If we have a very good LLM then this framework might not help much as we will have less interaction data to fine-tune model on.

Our goal is to show implicit conversational signals can drive learning. These signals exist beyond our setup (as we discuss in related work). The poor initial performance is a feature of the experimental design – it allows us to show an effect. Other LLMs might be good at the task, but they will be bad at other tasks, and our approach will allow them to improve over time. Our research question is: Can implicit signals from interactions be leveraged and sustain continuous improvements in deployment? To show its effectiveness, we carefully selected the multi-turn reference scenario from CogSci and tangrams that are unfamiliar to modern pretrained vision LMs (shown in Ji et al.), such that (a) the scenario can be deployed in an academic lab and (b) progress is measurable with a reasonable amount of data. This methodology leads to our evaluation with one model and one task.

Of course, it is always useful to try different models, but it is also important to consider the costs of such experiments – with humans in the loop. What are the implications of demanding many LLMs and many tasks with such studies? It will simply be impossible to run such research in academic labs. Even industry labs are not likely to just put millions of dollars into a speculative new approach without a proof of concept. This is what our research is.

Lack of generalizability ... across different tasks: Another interesting extension of the proposed method can be over different tasks. Currently authors have only tested over a particular task but it would be interesting to see if it can be extended over other tasks like summarization (authors have highlighted this as future work in discussion).

We agree with the reviewer on experimenting with different scenarios. As we discussed in our related work, a concurrent work by Don-Yehiya et al. showed these signals exist in lmsys-chat-1m. So this tells us that these signals are general. We show how they can be used for an LLM to improve its own behavior. Similar to trying different models, more tasks have extreme engineering and experimental implications – this simply makes this kind of work impossible, even though it’s an important and impactful research avenue.

To be continued in Part 2.

Thank you for your response and addressing most of my queries. I would like the authors to further clarify on the framework and its usability with respect to their response.

We train from scratch in every round (6 rounds in total – not every interaction step) for most systems. Training from scratch means: at every round, we fine-tune the initial IDEFICS2-8B parameters. We are not using the data to optimize again and again the same parameters. The measures we took to avoid overfitting are standard, just like any fine-tuning process, and the risk of overfitting is not different from any supervised learning scenario. Our evaluation is on unseen games (every round brings completely new games), showing strong generalization and improvement.

I am a bit confused. My initial understanding was the framework finetunes the test LLM model (and not the base LLM model) based on all the data collected from past interactions. This flow has been depicted in figure 1 (on the very first page) of the paper. It clearly shows that $\theta_{p}$ is updated to $\theta_{p+1}$ based on all the aggregated data $D_{p}$. But now the authors claim that this is not the case—at each round, they fine-tune the initial IDEFICS2-8B parameters. This would mean that $\theta_{0}$ is updated to $\theta_{p+1}$. This brings up the question, how is it any different than fine-tuning from scratch, all that is changing is the dataset. The model is not improving on the learned information, all that is happening is model is learning new information. And the reason we see improvement over rounds is because the dataset size is being increased compared to the last round. Maybe I am missing something here and would like authors to clarify my misunderstanding.

Have you tested the framework on other LLMs or tasks to confirm generalizability? How might the framework’s usability vary with stronger LLMs that provides less interaction data?

Please see the discussion above regarding the costs and feasibility within academic constraints. Furthermore, IDEFICS2-8B was by far the best open-weight multimodal LLM available when we conducted the experiments. Of course, now there are better models (it’s a fast paced space), with which we just expect faster and even more efficient learning.

I understand the author's point on cost within academic constraints. My question was more stemmed on the usability of the framework with better multimodal models. As we have seen this trend with LLMs, we can expect better multimodal modals in few years. With those models we might not have enough interaction data for a particular task to fine-tune. Now, the authors says that with better models we can expect faster and efficient learning, but it is based on the fact that model is not performing well (the only way to get interaction data). If it performing well already then we can't expect expect faster and efficient learning. I would like to know authors take on this.

We thank the reviewer for their helpful comments. We address the concerns below, and following this discussion we will edit the paper to make sure the continual learning aspect comes through clearly (the changes are not yet in the PDF).

Response to Raised Weaknesses

We define continual learning as the model improving over time on its task through interaction with human users. Our data collection and evaluation are both integral parts of our deployment. At each round, we deploy the current model to interact with humans. Each system has its own set of interactions, and each system is trained on the interactions that it had with the human users, so its own performance improvement influences its continual learning. In this process, the model continually improves over time from the signals obtained from its own interactions with humans.

In this sense, whether the task has multiple turns (like in our case) or not is not a factor of the setup. You could think of the model learning within single interactions (improving from earlier in the interaction to later), but this is not within the scope of our work. Our work is also not about adaptation, because we don’t maintain user-specific models. This is possible with our approach, but not within the scope of our experiments.

The term continual learning is used broadly, for example for domain adaptation. We will add a clarification very early in the paper to how we use it. We apologize for the confusion. Please let us know if there is anything we can add to clarify the setup.

There is also no direct relation between the progression of rounds and the content length (context in the sense of the LLM prompt, see Figure 11). The context is longer in later turns in the same interaction because the history is longer (more past turns), but this is true both in early and later rounds of the continual learning setup.

Why is continual learning so important for our study of implicit conversation signals? There are complex dynamics between the signals we use and how the system evolves by learning from them. For example, it could be the case that as the model gets better at the task, our feedback decoding process becomes worse. We show this is not the case, and that the signals are present throughout the system’s lifetime (even as it improves) and our decoding approach remains effective and stable. Our setup also mimics the development-deploy cycles in practice, where the data collected in production often correspond to multiple prior model checkpoints.

Response to Raised Questions

Ln 425-426: According to Section 4.2, RL includes both positive and negative rewards. What might be the reasons that including extra rewards would 'encourage a uniform distribution'?

If we understand correctly, you are asking about why the inclusion of negative rewards “encourages a uniform distribution”. This was observed by Kojima et al. 2021. Essentially, by pushing down the probability of one output (the output with a negative reward), the model increases the probability of all other outputs. Because it’s a very large output space, this essentially pushes towards a uniform distribution. This is similar to unlearning objectives, which aim for the same (i.e., uniform distribution).

Please let us know if any questions remain, and consider raising the overall score if our responses are helpful.