DMA: Enhancing Retrieval-Augmented Generation with Adaptive Human Feedback

We thank Reviewer NAmN for one of the most thoughtful and constructive reviews we received. We appreciate your recognition of our core contributions—including the novelty of the problem, the structured feedback taxonomy, and the rigor of our empirical evaluation.

Summary of Contributions

To briefly reframe:

1. We introduce a multi-level feedback taxonomy (document, list, response) to capture fine-grained user signals.
2. We propose a two-stage online adaptation framework, combining listwise learning (ListNet) and PPO fine-tuning with a distillation layer.
3. We validate our system via a large-scale industrial RCT, offering strong real-world evidence and a deployable blueprint for adaptive RAG.

Group 1: Methodological Clarifications

Two-Stage Listwise Training (ListNet + PPO)

• Stage 1: Train with ListNet using global feedback (e.g., thumbs-up/down on final response), treating the associated document list as a ranked unit.
• Stage 2: Fine-tune using PPO with rewards from a pairwise-trained reward model (RM). Preferences between response pairs are projected back to document lists; PPO adjusts the reranker accordingly. The full pipeline is shown in Figure 2.

Q1. Implicit Feedback

Implicit feedback refers to user sentiment inferred from replies. We classify it using a prompted LLM with Cohen’s Kappa = 0.962 (vs. human annotations), applying a confidence threshold of 0.8.

Action: We will add a dedicated subsection describing this pipeline.

Q2. ListNet Loss and Ground Truth

Ground-truth scores are heuristically derived from final response-level ratings (e.g., thumbs-up). We apply ListNet loss over the associated document list.

Action: We will formalize this derivation and present the loss as block-level math.

Q3. Reranker Architecture

• Retriever: BGE-m3.
• Baselines: Online baseline is static BGE-reranker; public dataset comparisons use known rerankers.

Action: We will clarify this in Implementation Details.

Q4. PPO MDP Definition

• State: User query + retrieved document list
• Action: Reranked document list (permutation)
• Reward: RM score for the reranked list

Action: We will add this MDP formulation to the methodology.

GBDT Distillation + Hyperparameters

Action: We will include a new appendix with full details on the distillation pipeline (Flink-based), feature set, and all key hyperparameters (e.g., k, PPO settings).

Group 2: Notation and Presentation

We agree with the issues raised and will make the following corrections:

• Symbol Disambiguation: Clarify overloaded symbols (R, y, etc.).
• Terminology Fixes: Replace “Point Level” with “Document-level” throughout for consistency.
• Equation Formatting: Convert all major equations to block-level math for clarity.
• Figures: Revise all terminology and add definitions to figure captions.

Group 3: Strengthening Analysis

Ablation and Case Studies

While the original manuscript included ablations, the presentation lacked quantitative clarity.

Action: We will expand the analysis to include numerical metrics showing the impact of each feedback level. We will also add more concrete case studies (before/after examples) in the appendix to illustrate DMA's adaptive behavior.

We believe the technical foundation of our work is strong, and your comments have provided a clear roadmap to improve clarity and completeness. We are committed to incorporating all revisions and are confident they will result in a significantly improved manuscript.

Thank you for your detailed response addressing my concerns. I acknowledge that the paper proposes a novel and potentially valuable approach to addressing limitations in existing RAG systems. However, the original manuscript lacked critical content and suffered from unclear descriptions, which significantly detracted from its overall quality. While your rebuttal outlines concrete steps to resolve these issues, the scope of the necessary revisions remains substantial. At this stage, it is difficult to definitively evaluate the quality of the revised manuscript. In light of these considerations, I have adjusted my score.

Response to Reviewer DVFp

We thank Reviewer DVFp for the thoughtful and insightful review. We greatly appreciate your accurate understanding of the DMA framework, as well as your recognition of its key contributions—including hierarchical feedback modeling, PPO-based alignment, GBDT distillation, and large-scale deployment. Your questions address central aspects of our system, and we are pleased to clarify them below.

Summary of Contributions (for clarity)

To briefly reframe, our work proposes a deployable framework for real-world retrieval-augmented generation (RAG) via:

1. Multi-level Human Feedback Taxonomy A unified schema for organizing document-, list-, and response-level feedback, enabling fine-grained supervision and preference modeling in dynamic online settings.

2. Online Adaptation Pipeline A scalable learning loop combining reward modeling, listwise PPO alignment, and GBDT distillation, triggered continuously by live user signals.

3. Industrial-Scale Validation A months-long randomized controlled trial in a high-traffic GenAI system, offering rare real-world evidence of DMA’s impact under production constraints.

Q1. Why does DMA underperform on schema-bound QA tasks?

Clarification: This is an expected outcome based on task characteristics. DMA is designed for open-ended, user-facing QA tasks—such as TriviaQA and HotpotQA—where user preference signals (e.g., document utility, list quality, response satisfaction) are rich and nuanced.

In contrast, schema-bound QA tasks (e.g., NQ, WebQSP) have singular factual answers and little tolerance for subjective variation. As such, the multi-granular preference modeling central to DMA contributes less value in these settings.

Action: We will explicitly discuss this design trade-off and task alignment rationale in the Analysis (§5.3) and Limitations (§6) sections.

Q2. What are the GBDT input features and fusion methods?

Clarification: DMA employs a two-stage hybrid retraining strategy designed to balance long-term stability with short-term adaptability:

• (1) Weekly Full Retraining: A high-confidence corpus of implicit feedback (satisfaction score ≥ 0.8, derived via Qwen-72B with few-shot calibration) is accumulated over time. Once per week, a fresh base model is trained on this full corpus to absorb slow distributional drift and correct long-term biases.

• (2) Nearline Incremental Updates: On top of the weekly base, the system continuously accumulates ~500 new feedback instances via a Flink-based streaming pipeline. Upon reaching the threshold, the following update cycle is triggered:

  • Train teacher models using document-, list-, and response-level feedback.
  • Fuse supervision via PPO into an aligned reranker.
  • Distill into a 10K-tree GBDT for low-latency deployment (<10 ms).
  • Deploy within ~10 minutes (on 8×A800 GPUs), ensuring rapid adaptation.

Action: We will add a dedicated subsection in §5.1 (Experimental Setup) to describe this hybrid retraining architecture, including batch triggers, feedback routing, latency breakdown, and deployment intervals.

Q3. Can DMA be validated without proprietary infrastructure?

Yes. To ensure full reproducibility and support further research, we will release:

1. Full Codebase: Including reward models, PPO alignment, GBDT distillation, and pipeline orchestration.
2. Pretrained Models & Scripts: For TriviaQA, HotpotQA, and other public benchmarks.
3. Synthetic Feedback Simulator: A tool for generating document-, list-, and response-level feedback to support offline experimentation with DMA.

These assets allow researchers to study DMA’s behavior and capabilities independently of proprietary data or infrastructure.

Q4. How does DMA mitigate feedback-induced bias?

DMA incorporates several safeguards to control both feedback noise and systematic bias:

1. Confidence Filtering: Implicit signals are retained only if confidence ≥ 0.8, as estimated by the calibrator model (Qwen2-72B).
2. Controlled Training Role: High-confidence implicit feedback is used exclusively in weekly full retraining, not in nearline adaptation, preventing overreaction to transient signals.
3. Global Distribution Normalization: During training, we apply resampling and normalization strategies (e.g., batch norm, loss weighting) to balance overrepresented patterns and reduce reward hacking risk.

Action: We will expand §6 to discuss these mechanisms and add an ablation analysis in the Appendix quantifying the impact of confidence thresholding on performance stability.

Q5. Is DMA tied to proprietary models like Qwen-72B?

Clarification: No. While we use Qwen-72B in production due to its multilingual performance, DMA is fully model-agnostic. The architecture is modular and can be applied to any decoder (e.g., LLaMA, ChatGLM) and calibrator.

In fact, the central innovation of DMA lies in its feedback processing and adaptation pipeline, not in the choice of base LLM. The framework supports plug-and-play integration with arbitrary generation models.

Action: We will clarify this generality in the Discussion section, along with practical guidance for adapting DMA to alternative LLMs.

We sincerely thank you again for your thoughtful feedback and constructive suggestions. Your comments will directly improve the clarity, reproducibility, and positioning of our final manuscript.

Dear Reviewer DVFp,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.

Response to Reviewer 5ST9

We thank Reviewer 5ST9 for taking the time to review our submission. However, we respectfully believe that several of the concerns raised stem from misunderstandings of our core contributions and empirical findings. We appreciate the opportunity to clarify these points.

Clarification 1: “Online Gains Are Modest”

Clarification: A +24.57% absolute improvement in session-level user satisfaction—measured across 100,000 real-world conversations in a multi-month randomized controlled trial (RCT)—is not modest by any industrial standard. In high-traffic production systems, such improvements are exceedingly rare and directly correlate with key business metrics such as user retention, engagement, and operational efficiency.

DMA’s design specifically targets online adaptability, and these results validate the effectiveness of our real-time, feedback-driven memory alignment in dynamic, user-facing environments.

Clarification 2: “Benchmark Results Are Poor”

Correction: This statement appears to be factually inaccurate.

As reported in Table 3, DMA achieves:

• State-of-the-art performance on TriviaQA (Hit@1: 68.81%, F1: 68.90%), outperforming all recent baselines including KnowPAT, RAFT, and RRHF.
• Top performance on HotpotQA (F1: 41.88%), where DMA again leads over prior methods.
• Competitive scores on NQ and WebQSP, confirming that DMA maintains strong foundational capabilities even in structured QA tasks.

Importantly, our goal in reporting benchmark results is not leaderboard chasing, but to demonstrate that online adaptability does not come at the cost of baseline generalization. These results underscore that DMA is both practically useful and technically sound.

Reframing Our Core Contributions

To rearticulate the novelty and significance of our work:

1. Multi-level Feedback Taxonomy We propose a structured framework for organizing human feedback across document-, list-, and response-level signals, enabling fine-grained supervision beyond conventional static metrics.

2. Online Learning & Adaptation Framework DMA introduces a scalable, low-latency pipeline that fuses heterogeneous user signals into real-time retrieval policy updates, leveraging reward modeling, PPO alignment, and GBDT distillation.

3. Industrial-Scale Validation Our method is rigorously tested via a months-long industrial RCT with production constraints (latency, throughput, feedback sparsity), offering rare, high-quality empirical validation for RAG system deployment.

Response to Specific Weaknesses

Weakness 1: “Methodology Unclear”

We agree that some methodological aspects could benefit from clearer presentation.

• Feedback Source and Reliability

  • Explicit feedback (e.g., upvote/downvote) is used as ground-truth supervision.
  • Implicit feedback is inferred via a 72B instruction-tuned LLM (Qwen2-72B), calibrated using high-quality human-labeled exemplars. Inter-annotator agreement achieves a Cohen’s Kappa of 0.962, and only feedback with confidence ≥ 0.8 is retained.

  Action: We will add a dedicated paragraph in §5.1 clarifying this two-path feedback pipeline and filtering thresholds.

• Notation and Model Description

  Action: We will revise Section 4 for clarity, add a notation table in the Appendix, and include further description of the GBDT ensemble used in latency-sensitive online reranking.

Weaknesses 2 & 3: “Performance and Robustness”

As clarified above, DMA demonstrates:

• Significant online effectiveness in real-world conditions.
• Competitive and state-of-the-art static performance on widely used QA benchmarks.

These two axes—adaptability and baseline strength—are the primary goals of this work. We believe the results comprehensively support both.

Weakness 4: “Missing Ablation and Dataset Statistics”

• Ablation Study: Already included (see Table 2). It shows clear drops in satisfaction when document-, list-, or response-level feedback is removed (–15.57%, –11.21%, and –5.27%, respectively), validating the complementarity of multi-granularity feedback.

• Dataset Statistics: We agree this should be made more explicit.

  Action: We will add a new table in the Appendix summarizing:

  • Dataset sizes (e.g., number of sessions, turns, feedback events),
  • Coverage of each feedback type (explicit/implicit),
  • Sampling strategies for public benchmark evaluations.

We believe DMA offers a novel and practically meaningful contribution to the field of retrieval-augmented generation. By combining fine-grained feedback modeling with scalable online learning—and validating it in both controlled and real-world settings—we hope to offer a blueprint for deploying adaptive GenAI systems in production.

We thank the reviewer again for their feedback, and we hope these clarifications will lead to a fair reassessment of the manuscript.

Dear Reviewer 5ST9,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.

Response to Reviewer Ee9e

We sincerely thank Reviewer Ee9e for the encouraging and detailed review. We greatly appreciate your recognition of the DMA framework’s novelty—particularly its decomposition into multi-level feedback modeling, reward estimation, and online adaptation—as well as your positive assessment of our large-scale empirical validation.

Below, we address each of your comments in turn:

Summary of Contributions (for clarity)

To briefly reiterate, our key contributions are:

1. Feedback Taxonomy: We propose a structured, four-tier framework to systematically capture and organize document-, list-, response-, and session-level user feedback in production RAG settings.
2. Online Adaptation: We design a hybrid learning mechanism that fuses multi-granular signals into a unified, nearline update loop via reward modeling, PPO alignment, and GBDT distillation.
3. Industrial Deployment & Validation: We validate DMA through a large-scale, multi-month RCT in a commercial GenAI assistant, demonstrating significant improvements in session-level satisfaction (+24.57%) under real-world usage.

Weakness 1: Missing Details on Feedback Collection and Update Frequency

Clarification: DMA employs a two-stage hybrid retraining strategy designed to balance long-term stability with short-term adaptability:

• (1) Weekly Full Retraining: A high-confidence corpus of implicit feedback (satisfaction score ≥ 0.8, derived via Qwen-72B with few-shot calibration) is accumulated over time. Once per week, a fresh base model is trained on this full corpus to absorb slow distributional drift and correct long-term biases.

• (2) Nearline Incremental Updates: On top of the weekly base, the system continuously accumulates ~500 new feedback instances via a Flink-based streaming pipeline. Upon reaching the threshold, the following update cycle is triggered:

  • Train teacher models using document-, list-, and response-level feedback.
  • Fuse supervision via PPO into an aligned reranker.
  • Distill into a 10K-tree GBDT for low-latency deployment (<10 ms).
  • Deploy within ~10 minutes (on 8×A800 GPUs), ensuring rapid adaptation.

Action: We will add a dedicated subsection in §5.1 (Experimental Setup) to describe this hybrid retraining architecture, including batch triggers, feedback routing, latency breakdown, and deployment intervals.

Weaknesses 2 & 3: Reproducibility and Missing Baseline Comparisons

Reproducibility While proprietary user logs cannot be released due to privacy policies, we are fully committed to reproducibility and open research:

• Open-Source Code: The entire DMA pipeline will be released, including reward modeling, reranking, PPO alignment, and GBDT distillation components.
• Pretrained Checkpoints & Scripts: We will provide full training and evaluation scripts on public QA datasets (TriviaQA, HotpotQA, etc.).
• Synthetic Feedback Simulator: A feedback generation tool will be included to emulate multi-granularity feedback signals (document/list/response), enabling full-cycle simulation of DMA’s training loop.

Missing Baselines We appreciate the recommendations to include comparisons to recent methods:

• Adaptive-RAG (Jeong et al. [1]): We will include direct benchmark comparisons in the camera-ready version. Experiments are already in progress using their proposed query complexity segmentation.
• ReAR (Wang et al. [2]): We will expand the Related Work section to include a detailed discussion of ReAR’s relevance-aware re-ranking approach and contrast it with DMA’s feedback-driven reward modeling.

Clarifying Novelty Whereas prior works often rely on static heuristics (e.g., dialog length, query type) to modulate retrieval, DMA introduces a unified, multi-granularity supervision loop. This enables continuous alignment with user intent at three functional stages—retrieval (document), ranking (list), and generation (response)—and does so in real time with production latency constraints.

Question: Robustness to Noisy or Implicit Feedback

This is an important concern in real-world RAG systems. Our design includes multiple safeguards to ensure robustness:

1. Confidence Filtering: We only retain implicit satisfaction labels if the model-calibrated confidence score is ≥ 0.8. These scores are generated by Qwen2-72B using an instruction-tuned prompt template with few-shot exemplars (see Table 1).

2. Controlled Influence on Training: These high-confidence signals are only used for initializing the weekly retraining baseline. They are not directly involved in the fine-grained feedback loops used for online adaptation (document/list/response-level reranking). This ensures that transient or misaligned feedback does not cause premature drift.

Action: We will revise §5.1 to elaborate on the confidence gating mechanism and retraining strategy, and include an ablation in the Appendix showing the impact of different confidence thresholds on performance robustness.

We thank you again for the constructive and detailed feedback. We are confident that the planned clarifications, expanded comparisons, and reproducibility improvements will significantly strengthen the final version of our paper, and further establish DMA as a practical and adaptive architecture for real-world GenAI retrieval systems.

Dear Reviewer Ee9e,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.

Response to Reviewer 2QpC

We sincerely thank Reviewer 2QpC for the thoughtful and constructive feedback. We are especially grateful for your recognition of the core contributions of our work—namely, the multi-level feedback taxonomy, the real-time adaptation architecture, and the industrial-scale validation.

Before addressing the specific concerns, we reiterate the central contributions of this work:

• Multi-level Feedback Taxonomy: We propose a unified framework to categorize and interpret heterogeneous user feedback at the document, list, and response levels. This taxonomy enables granular supervision and robust preference modeling in dynamic RAG systems.

• Online Adaptation Framework: DMA introduces a novel architecture for integrating multi-granularity feedback into continuous retrieval policy updates. This goes beyond traditional fine-tuning by enabling live feedback loops for session-specific alignment.

• Industrial-scale Deployment & Validation: We validate DMA via a multi-month, randomized controlled trial (RCT) involving over 100,000 real-world user sessions, demonstrating statistically significant (+24.57%) improvements in user satisfaction under production constraints.

We now address your key comments in detail:

Q1. Ambiguity of “Memory”

Clarification: The term “memory” in DMA refers not to the retriever’s static corpus or naive cache, but to the evolving trace of user–system interactions: a session-specific collection of retrieved documents, model outputs, and corresponding user feedback (e.g., upvotes, regeneration, etc.). This memory trace serves as the substrate for dynamic preference modeling and guides reranking alignment.

Action: We will revise the Abstract and §1 to more precisely define “memory” as feedback-aligned session state. To reduce semantic overload, we are open to renaming the method to Dynamic Feedback Alignment (DFA) in the final version.

Q2. Clarification of Online Learning

Clarification: DMA adopts a hybrid nearline learning paradigm to balance responsiveness and computational cost:

• Feedback signals are streamed via Apache Flink and accumulated in batches of ~500.
• After each batch, we retrain pointwise/listwise rerankers and distill them into a lightweight GBDT.
• The entire update pipeline (including distillation and deployment) completes within ~10 minutes using 8×A800 GPUs, ensuring sub-15-minute model freshness.

This design enables timely adaptation while avoiding the instability of per-sample updates.

Action: We will include a dedicated figure and subsection in §4.3 to describe the update architecture, batching schedule, and latency profile.

Q3. Lack of Comparison with SOTA Rerankers

Clarification: Our goal is not to surpass every static reranker on benchmarks, but to show robustness and adaptability in online deployment scenarios. That said, DMA remains competitive on open-domain QA datasets (e.g., TriviaQA, HotpotQA), achieving state-of-the-art F1 scores on both.

Action: We will extend §5.2 to include quantitative comparisons with Self-RAG, RA-DIT, and RAFT, all of which are alignment-enhanced retrievers. These experiments are currently underway and will be included in the camera-ready version.

Q4. Limited Theoretical Innovation

Clarification: We agree that DMA is primarily a systems and architectural contribution, rather than a theoretical one. Its novelty lies in orchestrating:

• a multi-granular reward modeling suite,
• PPO-based preference injection into listwise reranking,
• distillation for sub-10ms online inference,
• and deployment via streaming feedback in high-throughput industrial systems.

In this sense, DMA bridges the gap between alignment theory and scalable GenAI infrastructure—a gap we believe is under-explored in current literature.

Q5. Reproducibility and Accessibility

Commitment: While industrial user logs cannot be released for privacy reasons, we will ensure reproducibility through:

• Open-source Code: Full implementation of the DMA pipeline.
• Pretrained Models & Inference Scripts: For standard QA datasets including TriviaQA, HotpotQA, and NQ.
• Synthetic Feedback Simulator: To emulate multi-level user signals and facilitate benchmarking in academic settings.

This will allow researchers to replicate both static and dynamic aspects of DMA under controlled conditions.

Once again, we thank you for your valuable comments. We believe that the proposed clarifications and additions will address the concerns raised and significantly enhance the clarity, rigor, and impact of the final manuscript.

Dear Reviewer 2QpC,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you for your time and consideration, the authors.