RE-Adapt: Reverse Engineered Adaptation of Large Language Models

1. Insufficient empirical validation of the proposed approach.
   • The paper conducts experiments on a single task (question answering), with two adaptation datasets (StreamingQA and RetrievalQA) and one general QA dataset (Natural Questions). Since the main goal of the proposed approach is to build a model capable of following instructions, a validation of the final model on a broader set of tasks is required, e.g. including open-ended instruction following, summarization, or multiple-choice QA. This concerns both instruction following capabilities on new domains and initial model capabilities, i.e. validation solely on NQ is not enough to demonstrate the preservation of initial instructed LLM capabilities. It is possible to create multi-task evaluation beds e.g. for a biomedical domain with Pubmed as domain data [1-4].
   • I am not sure if RetrievalQA is a good candidate domain for the paper’s experiments, since performance on this dataset without RAG is very low (and improvements by the proposed method are also small, i.e. 1-2 % exact match), and because evidence passages from RetrievalQA do not necessarily form a well-defined domain.
   • Tables 2 and 3 miss a baseline $\Phi + 0.5 \Delta$, to disambiguate the influence of downscaling the Re-adapter (shown to be effective in Figure 6) and of learning the domain knowledge in the domain adapter. For example, in Table 3 the domain knowledge comes from using RAG, so this baseline can potentially perform as good as Re-Adapt / LoRE-Adapt.
   • It would be also an advantage if the proposed approach was validated for more than one adaptation approach, e.g. simple LoRA, as it is widely used in practice.
2. The partial adaptation approach is not novel as the same technique of applying a scalar factor, tuned on the validation set, was used e.g. in (Inarco, 2023).
3. The experiments are conducted with a fixed learning rate, which weakens a simple baseline of finetuning an instructed LLM on the domain data. As shown in [5, 6], simply reducing an LR may help to avoid catastrophic forgetting when tuning LLMs. According to Table 4 in Appendix, the paper uses an LR of 2e-4, which is too high to preserve initial model capabilities according to [6].
4. Deeper details on the experimental setup could be provided, such as the used decoding strategy at inference, metric details in case some preprocessing was used (e.g. lower casing candidate and ground truth answer), preprocessing details for the datastore in RAG (e.g. how passages for retrieval are constructed), or which parameters are used in bm25.

[1] Anastasios Nentidis et al. Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. CLEF 2023. (Factoid / list / yes-no questions and summarization on the biomedical domain.)

[2] Xinlu Zhang et al. AlpaCare: Instruction-tuned Large Language Models for Medical Application. arxiv 2023. (This paper collects a MedInstruct-test dataset which can be used for evaluating open-ended generation on the biomedical domain.)

[3] Qiao Jin et al. PubMedQA: A Dataset for Biomedical Research Question Answering. EMNLP 2019

[4] Hongjian Zhou et al. A Survey of Large Language Models in Medicine: Progress, Application, and Challenge. arxiv 2023

[5] Nadezhda Chirkova et al. Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks. NAACL 2024

[6] Nadezhda Chirkova et al. Zero-shot cross-lingual transfer in instruction tuning of large language models. INLG 2024

• Lack of Novelty: The methods proposed by the authors, namely RE Adapt and LoRE-Adapt, are based on well-established concepts such as adding independent adapters to the model, utilizing Singular Value De composition (SVD) to extract key information from matrices, and com bining adapters through weighted averaging. These methods were widely accepted and recognized by the academic community as early as 2021 with the introduction of LoRA. In the paper, the authors do not present any novel insights or groundbreaking approaches to solving the problem, which makes the paper seem more like an exploration of existing methods on certain datasets rather than a demonstration of a new way to address challenges.
• Lacking Comparison Experiments: The experimental section lacks critical comparisons. The issue raised by the authors is the format mis match between downstream task data and the instruction model’s for mat. In fact, the authors could have easily constructed versions of exist ing datasets that match the instruction format through a text pipeline, and then fine-tuned the instruction model directly through various meth ods. This approach is more straightforward compared to the proposed method. The authors did not consider comparing their method with the restructured datasets, which I believe fails to demonstrate the practical significance of their method.
• Lacking Discussion on Hyperparameters: As an important variant of the proposed method, LoRE-Adapt lacks experimental characterization of the impact of choosing different downsampling ratios 𝜏 on model per formance and inference cost.
• The paper lacks theoretical insights. While the experiments demonstrate a significant performance improvement with RE-ADAPT, there is insufficient theoretical support for the reasons behind this enhancement.
• Equation (4) introduces α and β as scaling factors for adjusting the strengths of the respective adapters, but there is a lack of experimental analysis discussing the balance between the knowledge adapter Ψ and the RE-Adapter Δ.
• LORE-ADAPT is a very interesting component, and many of its properties are quite intriguing. However, in terms of addressing the problem, both the principles and experimental results seem to be not that different from the RE-ADAPT method.  Others
• The authors’ estimation of model parameters seems to be incorrect. In Figure 3, when 𝜏 = 1, the parameter size reaches 100%. In fact, due to the structural characteristics of LoRA, when 𝜏 = 1, the parameter size should be 200%, i.e., retaining two matrices of the same size as the original matrix, representing the A and B matrices in LoRA.
• From the experiments, LORE-ADAPT, the low-rank variant of RE-ADAPT, shows similar performance while reducing parameters by 4 to 5 times, and even outperforms RE-ADAPT in certain models (e.g., Mistral-7B). However, the paper lacks explanation and discussion on this observation. It would be beneficial for the authors to provide more details regarding LORE-ADAPT.

• The experiments focus exclusively on question-answering (QA) tasks and do not assess whether the instruction-following capabilities of instruction-tuned models are maintained post-adaptation. Additionaly experiments on common benchmarks for instruction-tuned models are expected.
• A comparative analysis between the original instruction adapter obtained via reversed engineering and the instruction adapter after further domain-specific pretraining and instruction fine-tuning could help validate the authors’ hypothesis about the transferability of the instruction adapter.
• There is insufficient analysis of the chosen hyperparameters, particularly the sensitivity of the scaling factor $λ$, which could impact the generalizability of the approach.
• Experiments on different model sizes (<7B and >7B) would provide a more comprehensive understanding of whether the proposed method’s benefits hold at various scales.
• Tables 2 and 3 should report results to two decimal places. Currently, certain models appear to have identical performance (e.g., Gemma Instruct, Gemma Instruct + DORA, and Gemma Re-Adapt on the Natural Questions dataset all report a score of 26).

1. The applicability of the proposed method may be limited, as it relies on an instruction-tuned version of the original foundation model. While many open-source LLMs offer both versions, it remains unclear how RE-Adapt would perform if only the foundation model is available and we attempt to tune a weaker instruction-following model.
2. On line 201, the authors state that “any fine-tuning approach is applicable”, which is misleading. Directly employing a full-parameter fine-tuning method would prevent the combination of the tuned model's weights ($\Psi$) with the original weights ($\phi$) as described in Equation 4.
3. There is a lack of a LoRA baseline. Since LoRE-Adapt is a variant based on LoRA, it is essential to include a baseline that fine-tunes a pre-trained or instruction-tuned model using LoRA, in addition to the +DoRA baseline.
4. The RAG setups in RetrievalQA should pose more significant challenges. Currently, the BM25 retriever searches only within in-domain data, while a more realistic scenario would involve a retrieval datastore created from diverse external data.
5. The proposed method combines different adapters and weights in a linear fashion. A previous study [1] systematically investigates the arithmetic operations of parameter-efficient tuning modules, and including this work in the literature review would be beneficial.
6. Typos:

• L028: Parameter Efficient Fine Tuning -> Parameter-Efficient Fine-Tuning
• L114: data driven -> data-driven
• L409: (RAG) Lewis et al. (2020) -> (RAG; Lewis et al., 2020)

References

[1] Zhang, J., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36, 12589-12610.