StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses

Dear Reviewer wRZP,

We sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.

The training method is complex, resulting in higher training costs.

Our method outperforms baselines in both training and non-training settings and the non-training setting incurs no additional costs. The training method we introduced is intended to better adapt the model to the conv-attn sink mode. This results in a trade-off between significantly improved performance and increased training costs. However, regardless of the choice, our method consistently performs better than baselines.

Additionally, during the inference stage, our method can significantly reduce space and time complexity by compressing historical dialogues into conv-attn sinks. Below is a comparison of our method with StreamingLLM in both training and non-training settings.

(1) The comparison of different methods is unfair; (2) More base models and metrics are needed for the non-training setting.

(1) We have made every effort to ensure all comparisons are conducted fairly. There may be some misunderstandings: in the main experiment, all methods were maintained with the same training settings and were fine-tuned on specific datasets, including StreamingLLM. We will further clarify the experimental settings in the revision. Thank you for the suggestion.

(2) Thank you for the great advice! We have added tests on the Llama-3-8B-Instruct and Mistral-7B [1] under the non-training setting and included additional metrics: BLEU and ROUGE-L.

In line 127-128, the authors mention that by caching only the corresponding conv-attn sinks, the time complexity of attention computation can be reduced from $O(T^2L^2)$ to $O(T^2)$. However, have the authors considered that if each utterance contains $L$ tokens, the time complexity should be $O(T^2L)$?

After our thorough verification and calculations, the time complexity is indeed $O(T^2L)$. We sincerely appreciate your suggestion and will revise the complexity in the revision.

For the same sample, it would be best for the authors to use the same notation on line 157 and line 175 to avoid confusion.

Thank you for your suggestion. We will use the same notation in the revision for lines 157 and 175.

Once again, we appreciate your efforts and valuable suggestions to improve our paper. If you have further questions, please leave more comments in the OpenReview system. We would appreciate it very much if you could kindly raise your score if your concerns are addressed!

References
[1] Mistral 7B

Dear Reviewer 4bTd,

We sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.

More baselines are needed (Weakness 1).

Your kind advice has inspired us to conduct more comprehensive experiments by incorporating the recommended baselines: infinity former and two position interpolation-based approaches with RoPE, which are YaRN [1] and Dynamic NTK-RoPE [2]. Our method achieves favorable results compared to infinity former across all metrics and outperforms YaRN and Dynamic NTK-RoPE on some metrics. Since the lengths of the MSC and PersonaChat datasets are within the training length of Llama, it is reasonable that our method only shows advantages in some metrics relative to the full-attention RoPE position interpolation methods.

Line 111: Yes, position interpolation based approaches may not provide for infinitely long sequences, but is that required for dialogue tasks? Perhaps this could be rephrased to make the authors point.

Ideally, our objective is to develop a lifelong dialogue system capable of continuous conversation while retaining memory of all past dialogues. The term infinitely long refers to the cumulative length of each and every utterance, not the length of a single utterance. Position interpolation-based approaches lack this capability.

Line 119 makes a leap in logic that is perhaps not fully intuitive. The authors claim that higher attention on EoU tokens suggests that these aggregate information. Did the authors test whether the responses of the LLM to conversations implied that high attentions on EoU tokens could possibly capture information from the preceding utterances ?

We tested and confirmed that high attention to EoU tokens effectively captures information from previous dialogues. The answer is yes, and we illustrate this with the following case study.

Using an untrained Llama-2-7B-Chat model, we restrict it during inference to focus only on EoU tokens from previous utterances and the last complete utterance. Given the input "Did you have a caramel macchiato today?</s>Yes!</s>What kind of coffee did you have today?</s>," the model responds with "I'm glad you asked! I had a delicious caramel macchiato this morning." This shows that the EoU tokens successfully capture the key information "caramel macchiato" from the first utterance.

Line 160: Did the authors consider restricting the attention for u' tokens to just the sink token, rather than sink tokens and previous tokens? The current phrasing seems to indicate the former over the latter.

There may be some misunderstandings due to unclear expressions. To clarify, in short-memory reconstruction, $u'$ can indeed only see the conv-attn sink token of $u$ to reconstruct $u$, and the conv-attn sink of $u$ can attend to $u$ to compress $u$'s information onto itself.

The authors are requested to add notes on limitations and potential negative social impacts. The additional training methods proposed may incur computational cost, and I encourage the authors to comment on this.

Thank you very much for your valuable suggestion. StreamingDialogue significantly reduces space and time complexity during the inference stage. Additionally, we can outperform the baseline under the non-training setting without additional cost.

To optimize LLMs for the conv-attn sinks mode, we implement two learning strategies: short-memory reconstruction and long-memory reactivation. Consequently, this inevitably increases computational costs under the training setting, with the SMR and LMR phases requiring about two hours on two A100-40G GPUs. We will include additional details on computational costs in the limitations section of the revision.

Thank you for your instant feedback and valuable revision advice. The points you raised are worthy to ponder upon. We are so glad to make a further discussion on those issues. Hope our response will address your concern.

References
[1] YaRN: Efficient Context Window Extension of Large Language Models

[2] https://github.com/jquesnelle/yarn/pull/1

Dear Reviewer 7W1D,

We sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.

Table 1 does not show the results with USL-H and Dial-M. Also, in the case of MSC, StreamingDialogue performs better than StreamingLLM on the USL-H metric but not on the Dial-M metric.

Thank you very much for your valuable suggestion. The experimental results using USL-H and Dial-M for Table 1 are shown in the below table:

↑ indicates the higher score is better, while ↓ indicates the lower score is better. * indicates significance and ^ indicates insignificance.

Our method significantly outperforms all baselines across several metrics, including USL-H, PPL, BLEU, ROUGE, and Distinct. It also exceeds most baselines on Dial-M, but BigBird and StreamingLLM show only non-significant improvements over our method in Dial-M on MSC. Additionally, our method demonstrates significant improvemnet over all baselines in Dial-M when evaluated on two newly added datasets: Topical-Chat and MultiWOZ. These results confirm that our method is indeed better than baselines.

There is no information about inter-annotator agreement for the human evaluation.

We apply Fleiss' kappa [1] to measure the agreement among four annotators, yielding a result of 52.51%. This indicates that the inter-annotator agreement is moderate ($\kappa \in [0.4, 0.6]$).

Did the authors use the persona profiles to generate the responses? If yes, how was it included in the context? If not, is it fair to compare BigBird and StreamingLLM with StreamingDialogue?

All of the baselines and our method did not use persona profiles to generate the responses. Therefore, our experimental comparison is fair.

The use of the BLEU metric is not consistent. It would be better to show BLEU, BLEU-1 and BLEU-2 in all the result tables.

Thank you for your suggestion. We have reported BLEU, BLEU-1 and BLEU-2 in all the result tables as follows:

Table 1: Main results on the PersonaChat and MSC datasets.

Table 2: Ablation results on MSC with different learning strategies.

We will include these results in the revision. Thank you for the great advice!

More datasets are needed (Weakness 5)

Thank you for the constructive feedback.

We have conducted experiments on Topical-Chat and MultiWOZ. The results are shown in the table below.

Our method outperforms all strong baselines due to its ability to retain more complete historical information.

We will include these results in the revision.

Why are Layer 0 and Layer 1 shown in Fig. 1a, whereas Layer 28 is shown in Fig. 1b?

We have added the Layer 0 and Layer 1 attention maps for Fig. 1b, as detailed in Figs. (a)-(d) of the PDF in global response.

Did the authors analyze the conv-attn sinks for the example in Fig. 7?

In global response, Figs (e)-(f) of the PDF illustrate the model's attention to conv-attn sinks for the example shown in Fig. 7. During the inference stage, only the key-values of the conv-attn sinks are retained. In the case study of Fig. 7, the generated response requires the critical information "California" from the 12th and 14th utterances. We observed that more attention is allocated to the conv-attn sinks of the 12th and 14th utterances, indicating that during inference, the model focuses more on the conv-attn sinks that are useful for the inference.

Thank you once again for your valuable time and constructive feedback to improve our paper. We are eager to address any additional questions or concerns that you may have.

If your concerns have been addressed, we would be very grateful if you could raise the score.

References
[1] Measuring nominal scale agreement among many raters (Fleiss, J. L., Psychological Bulletin 1971)

Thank you for the detailed response. I appreciate the effort in sharing the results on Topical-Chat and MultiWOZ datasets. I have updated my scores. However, I still have the following questions and concerns.

1. For Topical-Chat, did you use the grounding knowledge to generate the response?
2. For MultiWOZ, did you use the belief states to generate the response?
3. In the Persona-Chat dataset, the users pick one or more persona from their assigned profile to generate the responses. Now, if the generation of the response is not conditioned on the persona, then the model tends to produce responses that reduce perplexity. As a result, even though the response is not persona-grounded, it may achieve better BLEU scores. So, in my opinion, grounding knowledge should be included in the dialogue context for any kind of knowledge-grounded response generation task. Otherwise, it does not provide the complete picture. This is why I am still skeptical about the soundness of the result.