Contextual Bandits with Entropy-based Human Feedback

We thank the reviewer for their thoughtful comments and valuable feedback. Below, we address the specific concerns and clarify the contributions of our work.

---

Novelty and Contribution

We acknowledge the reviewer’s concern regarding the novelty of our approach. While the use of human feedback triggered by high uncertainty has been explored (e.g., [A]), our work introduces two key components that differentiate it:

1. Two Types of Human Feedback:

   • We propose the use of Action Recommendation (AR) and Reward Manipulation (RM) as distinct feedback mechanisms, each tailored to address different challenges in contextual bandit learning.
   • AR directly guides action selection, enhancing exploration.
   • RM refines reward estimation, improving exploitation by penalizing suboptimal actions.

2. Entropy-Driven Feedback Trigger:

   • Our method leverages an entropy threshold to decide when to request feedback, which provides a structured and efficient way to incorporate human expertise.
   • While entropy-based methods have been explored in prior work, our specific application of combining AR and RM in this framework has not been studied in contextual bandit settings.

---

Theoretical Guarantees

We acknowledge that theoretical analysis would strengthen the contribution. While our current work focuses on empirical performance, we are actively working on deriving regret bounds for the proposed method, which will offer a formal understanding of its convergence behavior. We will update the theoretical guarantees soon.

Experimental Design and Realism

1. No Human Subject Study:

   • We agree that human subject studies would provide additional insights, especially to account for feedback variability and biases [C]. While our current experiments simulate feedback, we plan to extend our work to include human-in-the-loop experiments.

2. Feedback Quality Variability:

   • We simulated feedback with consistent quality for simplicity in this initial study. However, we agree that introducing variability in feedback quality, as highlighted by [C], would provide a more realistic evaluation. This is a valuable direction for future work.

3. No Feedback Baseline:

   • As suggested, we will include a "no feedback" baseline to highlight the benefits of incorporating human feedback.

4. Feedback Efficiency:

   • To address the practical value of feedback, we plan to analyze the trade-off between the amount of feedback required and the achieved performance. This will help demonstrate how efficiently our method uses human feedback.

Presentation Issues

We appreciate the reviewer’s specific comments and have made the following improvements, the new manuscript will upload soon with the following corrections.

• Error Bars:

  • Add error bars to Figure 2 and other relevant figures to show variability across multiple runs.

• Axis Scales:

  • We will ensure consistent y-axis scales across subplots for easier comparison of results.

• Entropy Thresholds:

  • Explicitly stated the entropy thresholds used in the experiments in Section 4.4 and Figure 4.

• Table 2:

  • Fixed the formatting issue in the appendix to ensure the table is fully visible.

• Algorithm Consistency:

  • Corrected the inconsistency in Algorithm 1 regarding the application of penalties in RM.

Response to Specific Questions

1. Reward Assumption (0/1):

   • We agree that the 0/1 reward assumption is restrictive. While it simplifies experimentation, our method can be extended to support continuous rewards. We will update the text to clarify this and explore this generalization in future work.

2. Feedback Efficiency:

   • We acknowledge the need to evaluate how much feedback is required to achieve a specific level of performance. We plan to add experiments that analyze this trade-off in detail.

3. Number of Actions in AR:

   • The number of recommended actions in AR is currently fixed. We will include experiments to analyze the impact of varying this parameter on performance.

4. Feedback Quality:

   • Feedback quality is assumed to be consistent and known in our current experiments. We agree that exploring variable and latent feedback quality will provide a more robust evaluation of our method.

We appreciate the reviewer’s constructive feedback and have taken steps to address the raised concerns. We believe our approach contributes to the contextual bandit literature by introducing two distinct types of human feedback and an entropy-based trigger mechanism. We will incorporate the suggested improvements and analyses to strengthen our paper further.

Thank you for your time and consideration.

We thank the reviewer for their thoughtful comments and constructive feedback. Below, we address the concerns raised and provide clarifications to strengthen the paper.

1. Problem Formulation and Oracle Access

We acknowledge that the problem formulation in Section 3.1 could be made clearer. Our intention was to describe a general framework where the agent can query a human expert for two types of feedback:

1. Action Recommendation (AR): Providing a recommended set of actions.
2. Reward Manipulation (RM): Adjusting rewards based on the expert’s evaluation.

Clarifications:

• Oracle Access vs. Realistic Feedback:
  • We assume that human feedback approximates optimal recommendations or corrections to aid the learner. This framework allows us to simulate scenarios where human input improves learning efficiency.
  • We will revise Section 3.1 to clarify these assumptions and better differentiate between simulated oracle feedback and realistic human input.

---

2. Theoretical Guarantees

We acknowledge that theoretical analysis would strengthen the contribution. While our current work focuses on empirical performance, we are actively working on deriving regret bounds for the proposed method, which will offer a formal understanding of its convergence behavior. We will update the theoretical guarantees soon.

3. Motivation for Reward Manipulation

The reviewer raises a valid point about the role of Reward Manipulation (RM). The purpose of RM is to:

• Penalize suboptimal actions that deviate from the expert’s recommendation.
• Improve exploitation by refining the agent’s reward signal in ambiguous cases where the environment’s reward alone might not sufficiently guide learning.

We will revise the text to clarify this motivation and show how RM complements AR by reducing uncertainty in action-value estimation.

4. Novelty of Entropy-Based Feedback

We appreciate the reviewer highlighting related work [1]. While entropy-based active learning is indeed well-studied, our paper differs in its application:

• Novelty in Contextual Bandits:
  • We extend the entropy-based framework specifically for contextual bandits by incorporating human feedback dynamically.
  • Unlike prior work that focuses on label uncertainty, our method uses entropy to decide when and how to query human experts for actionable feedback.
• We will expand the related work section to provide a detailed comparison with [1] and highlight the unique aspects of our contribution.

5. Hyper-Parameter Clarity

We recognize that certain hyper-parameters, such as reward penalties and entropy thresholds, were not sufficiently detailed.

• Reward Penalty: We will include a formal description of how the penalty is computed and its impact on performance.
• Entropy Thresholds: The range and dynamics of the thresholds will be explicitly defined in the experimental setup.

6. Empirical Results and Baseline Comparisons

• Figure 2: We will improve the presentation by adding error bars to show variability and ensure that all subplots use consistent y-axis scales.
• Baseline Comparisons:
  • We acknowledge that the current comparisons may seem unfair due to differences in oracle access. To address this, we will:
    1. Include a “no feedback” baseline to isolate the contribution of human feedback.
    2. Discuss limitations and justify the comparisons with clearer explanations of each method’s assumptions.

Thank you again for your time and insights. We believe that these revisions will significantly improve the quality and clarity of our work.

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address the key concerns and clarify the contributions and methodology of our work.

1. Related Work

We appreciate the reviewer highlighting relevant prior works ([1], [2], [3], [4], [5]). While our paper positions itself within the human-in-the-loop contextual bandit literature, we recognize that additional context from these works will improve the completeness of our discussions.

Revisions:

• [1] Tang and Wiens: We will discuss how counterfactual-augmented importance sampling provides a framework for evaluating interventions, which relates to our feedback setting.
• [2] DAGGER: We will include comparisons with DAGGER, emphasizing differences in how our method handles expert feedback dynamically versus DAGGER’s fixed imitation approach.
• [3] Active Preference Optimization (APO): We acknowledge the relevance of trajectory-level preference feedback in APO and will add this to show parallels with reward manipulation.
• [4] BALD and [5] Information-Directed Sampling (IDS): We will highlight the connections between entropy-based active learning (BALD) and IDS methods and clarify how our approach adapts entropy for contextual bandit problems rather than broader active learning.

2. Justification of Entropy-Based Feedback

The reviewer points out that our current justification for entropy-based feedback lacks rigor, especially when contrasted with more refined methods like BALD or IDS.

Clarifications:

• Why Entropy Works:

  • In contextual bandit problems, high-entropy policies indicate uncertainty in action selection. By querying human feedback in these high-uncertainty scenarios, we aim to improve action selection in critical situations where the model is less confident.
  • This approach aligns with active learning principles but focuses specifically on the exploration-exploitation trade-off in bandit settings.

• Entropy Reduction vs. Raw Entropy:

  • While entropy reduction (e.g., in BALD) provides a more sophisticated approach, our use of raw entropy simplifies feedback decisions with minimal computational overhead.
  • We will discuss how incorporating entropy reduction could be a promising direction for refining the feedback strategy in future work.

3. Experimental Analysis and Takeaways

The reviewer found our experimental section well-structured but suggested clearer takeaways from each experiment.

Improvements:

• Sec 4.2 (Variations):

  • We will reorganize the results to group findings into key patterns:
    1. Impact of feedback frequency on performance.
    2. Comparative performance of action-based vs. reward-based feedback.
    3. Sensitivity to varying levels of human expertise.
  • This will provide more concrete takeaways and actionable insights.

• Sec 4.4 (Entropy Thresholds):

  • We agree that this section offers valuable analysis. To improve clarity, we will:
    1. Explicitly summarize the impact of different entropy thresholds.
    2. Highlight the practical implications of tuning thresholds for feedback efficiency.

4. Feedback Selection Criteria

The reviewer asks if we have considered entropy reduction or information gain as feedback selection criteria (e.g., BALD or IDS).

Response:

• Our current framework uses raw entropy due to its simplicity and ease of implementation. However, we acknowledge that methods like BALD or IDS could provide more targeted feedback by focusing on entropy reduction.
• We will discuss how these approaches could complement or extend our method in future iterations.

5. Minor Issues

We appreciate the reviewer pointing out minor issues:

• Algorithm 1: We will streamline the pseudocode by removing unnecessary lines.
• Equation 7: We will correct the missing brackets and thoroughly review the paper for any similar typographical errors.

---

These changes will improve both the methodological clarity and overall contribution of our paper. Thank you for your time and valuable feedback.

Related Work

Our work draws inspiration from and builds upon several important lines of research in counterfactual reasoning, imitation learning, active preference optimization, and entropy-based active learning. Below, we highlight key connections and distinctions between our approach and prior work:

Counterfactual-Augmented Importance Sampling

The framework proposed by Tang and Wiens [1] introduces counterfactual-augmented importance sampling for evaluating the impact of interventions. This is closely related to our feedback setting, where we leverage counterfactual reasoning to assess the effects of dynamic adjustments in the decision-making process. Our approach extends these ideas by incorporating real-time feedback mechanisms to optimize sequential actions under contextual uncertainty.

Imitation Learning and DAGGER

Our approach also shares conceptual ties with imitation learning methods, particularly DAGGER [2]. DAGGER utilizes a fixed imitation strategy, iteratively aggregating expert demonstrations to refine policy learning. In contrast, our method dynamically incorporates expert feedback to adaptively guide the learning process. This key distinction allows our approach to better handle the complexities of continuously evolving environments and mitigate compounding errors inherent in static imitation-based frameworks.

Active Preference Optimization

Active Preference Optimization (APO) [3] explores the use of trajectory-level preference feedback for optimizing policies. We acknowledge the relevance of this feedback paradigm and draw parallels between APO and our method's treatment of reward manipulation. By integrating preference-based adjustments at multiple stages, our approach extends the flexibility of APO to broader scenarios, particularly in settings where reward structures are complex and partially observable.

Entropy-Based Active Learning: BALD and Information-Directed Sampling

Entropy-driven strategies such as BALD [4] and Information-Directed Sampling (IDS) [5] have been pivotal in active learning and decision-making under uncertainty. BALD focuses on maximizing the mutual information between model predictions and observed data, while IDS optimizes decision-making by balancing information gain and regret minimization. Our work adapts these entropy-based principles to the contextual bandit setting, where the goal is to maximize information efficiency while directly addressing the problem of sequential exploration and exploitation.

By situating our work within these rich and diverse areas of prior research, we emphasize the novelty of our contributions in dynamically integrating feedback, contextualizing entropy-based decision-making, and extending counterfactual reasoning to real-time applications in interactive learning systems.

We thank the reviewer for their detailed feedback and thoughtful comments. Below, we address the specific points raised and clarify our contributions.

1. Theoretical Analysis

We acknowledge that theoretical guarantees, such as regret bounds, are crucial in contextual bandit settings. While our primary focus in this work is empirical evaluation, we agree that incorporating theoretical analysis would strengthen the contribution.

Response:

• We are currently working on deriving regret bounds for our framework. Preliminary results indicate that incorporating human feedback under the proposed entropy-based mechanism can achieve sub-linear regret, which aligns with standard benchmarks for contextual bandit algorithms.
• We plan to include this analysis in a follow-up version to formalize the theoretical properties of our method.

2. Incentivization for Maximizing Entropy

The reviewer raises an important concern about the potential incentivization to maximize entropy to elicit more human feedback. This feedback mechanism is designed to be query-efficient, focusing on areas of high uncertainty while avoiding unnecessary feedback requests as the model converges.

Clarifications:

• Feedback Cost: In our current setup, we do not explicitly model the cognitive cost of human feedback, but we agree that this is an essential dimension for future exploration. Methods like those in [1] address feedback cost in similar settings and could inspire future extensions of our work.
• Entropy Threshold Dynamics: We aim to minimize entropy as the model improves, reducing feedback queries over time. This prevents exploitation of the feedback mechanism and ensures efficient use of human input.

3. Human Feedback Source and Baselines

We appreciate the opportunity to clarify our experimental setup:

Feedback Source:

• Human feedback in our experiments is simulated by accessing ground truth labels. This setup provides a controlled way to evaluate the potential benefits of human feedback while varying its quality and availability.

Baselines:

• We include baseline bandit algorithms without feedback to show the performance difference when human feedback is not used. Specifically, the “no feedback” baseline allows a direct comparison of the added value provided by human input.

4. Evaluation Metrics

The reviewer correctly notes that regret is a common evaluation metric in contextual bandits. In addition to cumulative reward, we computed cumulative regret but did not emphasize it in the main paper.

• We will update the paper to include cumulative regret results prominently.
• Regret curves will demonstrate how our method improves convergence to optimal actions compared to baselines.

5. Presentation Issues

We appreciate the reviewer’s comments on improving the presentation and will address the following points:

• Algorithm 1 (Lines 14-15):

  • We agree that these lines are unnecessary and will remove them for conciseness.

• Equation 7:

  • We will correct the missing closing brackets and review the document for any other typographical issues.

• Figures:

  • We will add error bars to all relevant plots and ensure consistent axis scaling to enhance interpretability.

6. Related Work: Off-Policy Bandits

We thank the reviewer for pointing out the relevance of off-policy bandits. Human feedback in our framework can be seen as introducing off-policy data since feedback may adjust the reward or action based on a different policy (the expert’s recommendation).

• We will expand the related work section to include relevant discussions on off-policy bandit methods, particularly [2], which provides insights into leveraging off-policy data in bandit settings.
• Highlighting these connections will better position our work within the broader literature.

We appreciate the reviewer’s constructive feedback and believe that the revisions outlined will address the raised concerns. Specifically, we will:

1. Include regret analysis and theoretical discussions on sub-linear regret.
2. Improve the explanation of the feedback mechanism and baselines.
3. Revise the presentation for better clarity and precision.

Thank you for your time and insights.

References
[1] Burr Settles. "Active Learning Literature Survey." 2009.
[2] Zhang, Ruiyuan, et al. "Near-optimal off-policy evaluation for linear contextual bandits." Advances in Neural Information Processing Systems (2020).

   E[\text{Regret}(T)] \leq O(\sqrt{T |A| \log T}) + O\left(\frac{(1-p)T}{q_t}\right),
   

   where $T$ is the number of rounds, $|A|$ is the action space size, $p$ is the feedback solicitation probability, and $q_t$ is feedback accuracy.
 - This bound decomposes the regret into components representing exploration and the quality of feedback, demonstrating how selective entropy-based feedback reduces overall regret.
 - Complete proofs and derivations have been included in Appendix section F&G to enhance clarity and rigor.