Can Decoding by Contrasting Layers Really Improve Factuality in Large Language Models?

1. While the paper addresses important challenges in LLMs, the proposed method may not represent a groundbreaking innovation within the field, although it effectively builds upon existing strategies. The paper does not adequately distinguish its proposed method from existing contrastive decoding techniques, leaving readers without a clear understanding of the unique contributions and advantages of the new approach.
2. The paper's coverage of related work appears insufficient, failing to provide a comprehensive overview of the state-of-the-art methods and how the proposed approach fits within and advances the current research landscape.
3. The paper's presentation suffers from formatting problems, such as missing formula numbers, which can detract from the overall clarity and professionalism of the work.

1. The motivation behind the proposed adaptive decoding strategy could be more clearly explained, as its rationale and effectiveness might seem questionable. This strategy opts for contrastive decoding only when LM shows uncertainty in predicting the next token, which assumes that contrastive decoding will yield a more accurate prediction in these cases. However, this raises the question: if the authors believe contrastive decoding is generally more effective, why not apply it consistently instead of implementing an adaptive approach? The results of varying the hyperparameter $\tau$ confirm that such an adaptive strategy is not necessary, as always using contrastive decoding reaches the best performance.
2. Considering that the adaptive strategy is not that effective, the novelty of this paper seems limited. It seems primarily to apply AutoCompressor to compress long contexts for LMs, rather than presenting a fundamentally new approach (correct me if I’m wrong).

• Limited contribution. The contribution of this work lies in showing that compression with contrastive decoding enhances performances and that adaptive decoding mechanisms may be suboptimal. However, simply combining compression techniques with DoLa lacks sufficient novelty, making the approach more incremental.

• Insufficient analysis. While the study finds adaptive decoding ineffective, it doesn't thoroughly explore why this is the case. Additionally, it lacks analysis of what information context compression actually preserves or whether it introduces unintended side effects(e.g. lossing , . Importantly, it overlooks scenarios where external knowledge might be inaccurate or irrelevant, limiting insights into robustness of the proposed methods under real applications.

• Lack of comparison with other compression methods. The paper does not compare its compression approach with alternative techniques. A broader comparison would clarify the advantages and limitations of this method within the wider context of existing research.

In think the major weakness of the paper is that the title, experiment setting, and the conclusion lacks an alignment.

• The title asks such question: "CAN DECODING BY CONTRASTING LAYERS REALLY IMPROVE FACTUALITY IN LARGE LANGUAGE MODELS?"
• The experiments show that simply fine-tuning a llama 32k context model achieves a top performance.
• The conclusion mentioned that the proposed method improved Dola, while Dola under the proposed setting does not outperform simple fine-tuning.

As a result, I believe the reported experiment results neither answered the title question, nor supported the conclusion. However, it might because of the confusing presentation in Table 1 so I failed to get the point. I'll consider adjust the review if authors help me with my confusions in the following question section.

• The title is very misleading: "Can Decoding by Contrasting Layers Really Improve Factuality in Large Language Models?" --> It seems that you are trying to challenge the idea of DoLa? But in fact, the content of this paper is not like this at all. This paper is extending DoLa to the long-context setting finding DoLa's limitation in long-context, and trying to add some new methods based on / relying on DoLa. In this title, I don't even see the phrase related to "long-context" at all but it's exactly the main topic of this paper. The title is very inappropriate and misleading. It seems just for eye-catching purposes. I cannot accept this paper unless the title is changed. A good title can be something like "Exploring the Limits of Decoding by Contrasting Layers for Long-Context Factuality in Large Language Models"

• Adding a long-context encoder seems to have nothing to do with DoLa at all: DoLa focuses on short-length QA tasks from the beginning. Not sure how this would be a good motivation.

• Figure 1 is also misleading. The DoLa method doesn’t include any context encoder, but the figure shows that the context encoder is part of DoLa, which is not true. You should add a title to the figure such as “Context Encoder + DoLa”

• According to section 3.3, your method will dynamically choose 1) Context Encoder + DoLa 2) Context Encoder + Normal Decoding. If it’s true, then the figure 1 is even more misleading. The caption says the left subfigure is DoLa and the right figure is the proposed method, but the real proposed method actually includes both left and right?

• Some of the math equations or definitions seem redundant, for example, in line 150, the authors defined $V_m$ as the middle layer output but never used it.

• Section 3.2 is PEFT specifically for context compression encoder, the title of the section should better be “Parameter-Efficient Fine-tuning for Context Encoder”

• In Section 3.2, you define the symbol $Q$ as the query vector. However, in Section 3.3, the $Q$ here seems to be the output distribution of early exits, e.g. $Q_L$, $Q_M$, $Q_j$ from $L$, $M$, $j$-th layers, as the query vector cannot compute any JS-divergence. Please define $Q$ properly.

• In Table 1, the row names seem too arbitrary: some include LLaMA-2-7B while some do not; the proposed method seems to have 128K ctx size but the name shows 4k, e.g. 4k Ours (non-adaptive). I am pretty confused about what these row names mean here.