The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning

Thanks for the insightful suggestions and helpful feedback of our work, and for the recognition of our work's motivation, performance, and potential academic impact.

Q1: (1) Given that the training loss is so critical to filter data, what do the authors think about the unreliability of the loss function in off-policy algorithms and (2) how that may be affecting LEAST ?

(1) To mitigate the unreliability of loss calculations, LEAST mitigates this issue by using the median of the loss values rather than relying on a single loss value per transaction, which reduces sensitivity to noise & biases. It will also be an interesting future direction to consider more robust loss calculation.

(2) While LEAST shows strong performance in the current experimental settings, an interesting direction for future work would be to investigate its behavior in tasks requiring even longer-term learning. In such scenarios, the potential instability of loss calculations might affect the episode-stopping threshold, which could slightly reduce the method’s efficiency in certain edge cases. It is an interesting future direction for further improving the robustness of loss calculations, which can further extend LEAST's applicability and improve its stability across a wider range of tasks.

Q2: How do you think performance would change if instead of storing the Q values in B_Q, the MC returns were saved?

Thank you for the suggestion! MC returns could be a potential metric. However, it can introduce two main challenges for implementing this within LEAST’s framework:

• Interaction efficiency: MC return requires repeatedly resetting to specific states to compute estimates from multiple rollouts to reduce variance in its estimation, which can be sample-inefficient.
• Suffer from biased estimation: MC return heavily depends on observing complete trajectories up to their true terminal states. When trajectories are truncated prematurely (as in LEAST), the missing components of the return are treated as zero, leading to biased and inaccurate calculations.

We hope future work will explore more advanced estimations based on MC returns to effectively address the aforementioned challenges.

Q3: Isn’t the issue of unchanging policy true for even non-LEAST methods? It's a little unclear to me why LEAST has this issue. Given this seems like a universal issue...

While this problem is indeed common in RL, it is particularly critical for LEAST due to its unique mechanism.

The motivation for proposing a noise adjustment schedule stems from specific observations during our experiments. Although the Q- & loss-based stopping threshold provides core benefits, LEAST occasionally triggers frequent resets during early training stages, which can hinder long-term exploration (crucial for extended execution tasks like humanoid walk).

Thus, we introduced a simple yet effective noise mechanism controlled by premature reset frequency, aiming to stabilize exploration, especially during early training. When integrated with TD3, the results showed minimal improvement, suggesting that noise adjustment may be less beneficial for vanilla deep RL compared to its impact on LEAST.

Table. Performance $\uparrow$ (Average of 3 runs):

Q4: Nothing to really comment but something of potential interest to authors ...

This is very interesting! We will analyze it in the related work of the new version.

W1: Fig 7 does not include information about std, variance etc... even though LEAST clearly has better sample efficiency.

Thanks for your suggestion. We used 5 random seeds, and we will update the caption and add error bars in Fig 7.

W2: The discussion on entropy aware dynamic buffer size & adjusting exploration noise seems quite ad-hoc... should have some discussion

To ensure robust episode truncation in complex environments, we implement two key mechanisms beyond the core Q-value & loss thresholds: (1) a dynamic buffer that filters outliers to improve threshold stability & accuracy when Q-values become unreliable, and (2) to prevent exploration limitations from frequent resets, we introduce an action noise mechanism that adaptively balances exploration and stopping behavior.

We will elaborate more in the revision.

W3: … While LEAST does help plasticity it is a little unclear whether this is because it fits higher quality data or .... this should be specified.

We would like to clarify that LEAST improves plasticity not by reducing the number of updates, instead, it helps the agent to focus on more informative samples which accelerates adaptation and helps it explore better policies.

We would like to thank the reviewer once again for the time and effort in reviewing our work! We are happy to provide further clarification if you have any additional questions.

Thanks for the insightful suggestions and constructive feedback on our work. Please find our responses to each of the concerns below.

Q1: What do you mean in Background by "In this paper, we investigate whether deep RL agents deployed on cumulative reward maximization tasks also suffer from sunk cost fallacy."?

We investigate whether RL agents exhibit behavior analogous to the sunk cost fallacy in reward maximization tasks, where agents inefficiently persist in executing trajectories despite diminishing returns will cause more losses, similar to how investors might hold onto losing investments. Through this analysis, we aim to identify and address fundamental inefficiencies in conventional RL architectures.

Q2: What makes the introduced heuristic special for deep RL in comparison to its importance for "non-deep" RL?

We would like to clarify that our proposed methodology is general for RL (not specific to Deep RL).

We validate our method's generality through experiments on tabular Q-learning in two MiniGrid navigation tasks [1] with default parameters. Results below measure the episode number required to converge to the optimal policy. LEAST significantly improves learning efficiency compared to standard Q-learning (5%-14% reduction in episodes required to converge), demonstrating its effectiveness in improving learning efficiency in simpler, discrete non-deep RL settings.

Why we mainly discuss deep RL: While non-deep RL provides a valuable testbed for validating the algorithms clearly, our main focus is to make deep RL algorithms learn better and use data more efficiently, since Deep RL is critical for solving complex, high-dimensional problems and has unique challenges and practical relevance in real-world applications. We aim to validate our approach in these demanding scenarios while encouraging future theoretical analysis in simpler RL paradigms.

We hope our work can inspire exploration in non-deep RL, e.g. tabular & linear methods for theoretical analysis.

[1] Open source GitHub repo: Minigrid-RL (default setting)

Q3: What do you mean by "In this section, we conduct comprehensive experiments to evaluate whether LEAST module is necessary for Deep RL."?

We will revise the sentence for clarity in the revised version. To clarify, in this section, we aim to evaluate the necessity of LEAST by investigating whether it provides significant improvements in the performance & efficiency of baselines.

Q4: ... why the proposed criteria of the LEAST are a good choice in general? ... Especially why is this better than a possible multi-criteria.

The proposed dual-criteria based on Q-values & loss offer the following major advantages, making it a robust, versatile, and general choice for our LEAST framework:

• Robustness. Q-values assess trajectory quality through expected rewards, while critic loss indicates the agent's understanding (familiarity) of transitions. These complementary metrics provide comprehensive evaluation criteria for stopping decisions.
• Versatility. Q-values & loss are core metrics in RL algorithms, making them algorithm-agnostic & environment-independent, unlike entropy (SAC-specific) or scene-specific metrics. This enables the dual-criteria to work across different tasks and algorithms with minimal changes.
• Simplicity. Q-values & loss are readily computed metrics that require no extra computation (e.g., network modules), making the criteria efficient to implement.

We explored various multi-criteria but found none that consistently outperformed the dual-criteria in efficiency and task independence. Our motivation is to provide a fundamental framework that is simple, effective, and broadly applicable across diverse tasks and algorithms, and to encourage future research into more sophisticated multi-criteria approaches.

W1: The presented background has nothing to do with "Deep" RL, it describes RL basics. It seems reasonable...

We revised this paragraph in the background section to better reflect the context of deep RL.

C1: The connection between the observed problem & the sunk-cost fallacy seems far-fetched...

Our analogy here focuses on the behavioral similarity: the implicit commitment to completing trajectories without evaluating their future benefit, although the classical sunk-cost fallacy typically involves emotional attachment. The absence of early stopping mechanisms in existing RL architectures leads to decision-making inertia, resembling the sunk-cost fallacy in its effect on efficiency.

Thank you again for reviewing our work. We will refine the typos and hope our responses address the concerns and would appreciate your consideration in the evaluation. We are happy to provide further clarification if needed.

We thank the reviewer for the insightful suggestions and useful feedback of our work. Please find our responses to each of the concerns below.

Question

Q1.1: Please verify the findings with more up-to-date algorithms such as SR-SAC, CrossQ,...,e.g., repeating the experiments on Mujoco tasks with a model-free algorithm.

Thanks for your suggestion. We have incorporated CrossQ as an additional backbone algorithm and conducted experiments on Hopper & HalfCheetah (with full results to be included in the revision due to time constraints).

The results show that CrossQ-LEAST consistently outperforms vanilla CrossQ in both performance & learning efficiency. This improvement may stem from CrossQ's more stable Q-value predictions compared to SAC, which helps calculate more reliable stop thresholds.

Table: Score$\uparrow$ (Convergence steps (M)$\downarrow$ ) (Average of 3 runs):

Q1.2: Why wasn't LEAST evaluated with TACO or A-LIX as well? The improvements should be orthogonal?

Since both methods are based on DrQv2, we used it as our primary backbone to evaluate LEAST's general improvements. We also tested LEAST with A-LIX and found consistent performance gains across image-based domains (results to be included in revision). Besides, MBRL indeed remains an interesting direction for future work.

Q2: Relevant prior work from the field of robotics seems not to be discussed.

We appreciate this suggestion and will enhance the related work section in the revision to thoroughly discuss relevant research from the robotics field.

Weakness

W1.1: The algorithm is presented as the agent "learning" to avoid the sunk cost fallacy. But no learning is happening as far as I can tell.

We appreciate your suggestion and will revise the phrasing to use "decide" instead of "learn," as it more accurately reflects how LEAST functions. We originally used the learning analogy to illustrate the behavioral patterns that LEAST promotes.

W1.2: The sunk cost fallacy seems to be only somewhat applicable here...

Our goal is to make it more intuitive for readers to understand our core contribution: existing RL architectures often lack the capability to dynamically evaluate the potential utility of continuing a trajectory, leading agents to "stubbornly" execute the full trajectory regardless of its relevance or productivity within the given interaction budget. This inefficiency bears an analogy to the sunk cost fallacy, where agents implicitly persist in a trajectory simply because it has already been initiated, without reassessing whether it remains promising or worthwhile. While the sunk cost fallacy is traditionally associated with human decision-making, we draw this analogy to highlight how LEAST's dynamic stopping mechanism addresses a similar inefficiency in RL.

W2: In other tasks with less well-shaped rewards & early reset structure, the method might limit exploration to a problematic extent?

LEAST's adaptive stopping mechanism functions independently of environment-level reinitialization, as it can detect suboptimal states before the environment triggers a reset. In locomotion tasks, e.g., LEAST identifies patterns like in the agent instability that indicate imminent failure, proactively stopping the episode before environment-level reinitialization occurs—particularly during early training stages. This proactive stopping enables LEAST to leverage its dynamic stopping mechanism to improve learning efficiency, even in environments with early reinitialization (as demonstrated in Figs 6, 7, 9).

To address the exploration limitations, we introduce a simple yet effective method, dynamic exploration noise adjustment, and find that it effectively helps agents escape suboptimal trajectories to explore more diverse behaviors. Thus, in environments with less well-shaped rewards & early reinitialization structure (e.g., PointMaze), LEAST still demonstrates robust performance, further confirming the method’s general applicability.

We hope our work can inspire future research to build upon this foundation for more challenging scenarios. Exploring further improvements to LEAST's exploration capabilities remains an exciting direction for future work.

We would like to thank the reviewer once again for the time and effort in reviewing our work! We are happy to provide further clarification if you have any additional questions. We hope that our responses adequately address your concerns, and we would greatly appreciate it if you could kindly consider reflecting on this in the evaluation.

We thank the reviewer for the insightful suggestions and useful feedback of our work. Please find our responses to each of the concerns below.

Q1: Can you demonstrate experimentally that LEAST truncates low-quality trajectories?

We conduct additional experiments on Ant to analyze trajectory length and quality, and will include it in the revision. Experiments below confirm that LEAST effectively truncates low-quality trajectories.

LEAST effectively truncates trajectories: We compare the distribution of trajectory lengths stored in the replay buffer between LEAST and vanilla agents (built upon TD3) during different stages of training. Table 1a shows that LEAST stores significantly more short trajectories than the vanilla agent, especially during the early training stages.

Table 1a. Percentage distribution of lengths in the buffer at different training stages (25%, 50%, 75%, and 100%). Trajectory length (Short (S): $l \leq 333$, Medium (M): $333<l \leq 666$, Long (L): $l > 666$):

Truncated trajectories are low-quality. We design a comparison experiment that evaluates the quality of trajectories identified for truncation by LEAST. We periodically (every 10k steps) stop the truncation to allow these "would-be truncated" trajectories to execute fully until their natural termination. The final cumulative rewards of these trajectories are then compared to the average rewards of trajectories under normal LEAST training. Results below show that "would-be truncated" trajectories achieve lower rewards across all training phases, confirming that LEAST effectively truncates suboptimal trajectories. This is further supported by suboptimal path truncation visualizations in MAZE (ANONYMOUS LINK).

Table 1b. Effectiveness of LEAST in truncating low-quality trajs:

Q2: The plasticity loss experiments would be more appropriately placed in an appendix...

We will move this to the appendix and use the space to expand the analyses in appendix.

The plasticity loss experiments were included to investigate a secondary benefit of LEAST in mitigating plasticity loss. Premature convergence caused by low-quality data (primacy bias [1]) is correlated with plasticity loss [2]. Since LEAST improves buffer data quality by truncating suboptimal trajectories (Fig. 3), we hypothesized it could mitigate plasticity loss, as analyzed in Fig. 11 to provide preliminary evidence for future work.

[1] The primacy bias in deep reinforcement learning. ICML'22.

[2] Loss of Plasticity in Continual Deep Learning. Nature'24.

W1: The additional mechanisms appear to function more as heuristic patches...

These auxiliary mechanisms ensure LEAST's robustness in complex environments.

While the basic Q-based module works well on simple tasks like Hopper, but becomes less reliable in more complex environments, e.g. Ant. We propose two complementary modules: a dynamic buffer to improve threshold calculations by filtering outliers, and noise adjustments to maintain exploration despite early stopping.

W2: The paper would benefit from exploring potential tradeoffs of early stopping and exploration...

To achieve that tradeoff, we introduce a simple yet effective noise adjustment module (which reduces the frequency of early stopping), helping to maintain exploration diversity and mitigate the risk of converging to overly short-term strategies. We hope our work can inspire research for further improvement.

W3: Clarify what "Loss" refers to specifically in Fig 3?

Loss refers to critic loss. We classify samples as High/Low Q based on their Q-values compared to the median Q-value in the Q buffer (and similarly for High/Low loss).

W4: Fig 5 only presents results for Ant and HC.

We chose these tasks as they represent different complexity levels - Ant: complex; HC: simpler - allowing us to evaluate LEAST across varying challenges. We will include full results in the revision.

W5: Typos, fragments & box plot.

We improved them in the revision. The edge of box plots indicates upper and lower bounds.

Thanks again for reviewing our work! We hope our responses address the concerns and would appreciate your consideration in the evaluation. We are happy to provide further clarification if needed.