The Burden of Interactive Alignment with Inconsistent Preferences

We thank the reviewer for their thorough and positive review of our work. We’re truly heartened that the reviewer described the paper as “great,” and appreciated its rigorous theoretical framework and guarantees under well-defined assumptions. We’re especially pleased that the reviewer was able to draw connections to real-world phenomena and raised thoughtful questions.

... it might be interesting to consider a dynamic discounting factor for a user. In reality, a person's ability to be foresighted can fluctuate based on internal factors. For example, a user who is typically strategic might enter a myopic state (doomscrolling) for some sessions. I wonder what the burden of alignment is in this situation. Would it degrade over time in this tug of war, where your myopic sessions ruin your strategic ones? Would the costly signals be able to reduce the burden? What happens when a lot of users doomscroll at the same time (sending a strong signal to the algorithm) during moments of political turmoil or catastrophes?

These are great questions! While we can offer some intuitive answers within the scope of this rebuttal, several of these points suggest promising directions for future work. They also highlight how our framework—and its departure from standard rationality assumptions in alignment—can raise and help address new, nuanced questions.

Regarding fluctuating user’s discount factor, for simplicity, first suppose $\gamma_{\mathcal{H}}$ is fixed across users but can vary over time for a given user. In this case, the critical threshold for alignment remains mostly as before; the key question becomes whether this user’s $\gamma_{\mathcal{H}}$ exceeds this threshold at any given time. Now, consider a scenario in which $\gamma_{\mathcal{H}}$ of the user decreases when the user is relaxing or is a "type 2" user in our special case. Since misalignment is minimal in such contexts, fluctuations in patience are unlikely to affect outcomes significantly. However, if the user becomes more myopic precisely when exposed to tempting but misaligned content—as in the case of "type 1" users—their effective horizon may become too short to steer the algorithm effectively.

In more complex settings where the $\gamma_{\mathcal{H}}$ values of many users change simultaneously, externalities could arise, affecting the critical effective horizon required for alignment. This is a particularly intriguing question and a compelling direction for future research.

Re. Line 167 and 302

We apologize for the typo in line 302. The sentence should read: “Given the algorithm’s best response, we now formally characterize the equilibria under algorithmic entry when users…” We will also clarify the acronyms RE and AE at this point in the text. We thank the reviewer for their careful reading and for pointing out these issues.

We thank the reviewer for their insightful review of our work. We’re glad they found the problem interesting and appreciated how it connects prior theoretical and empirical research. We’re also pleased that the reviewer recognized the value of our proposed extensions. Below, we address the reviewer’s questions and concerns.

it is shown that if a user has a low enough time horizon, then they may incur relatively heavy regret. However, in real-life settings, users may have relatively short time horizons precisely because they know they will not be using the app for very long. … I believe the authors are focusing on the case where a consumer’s time horizon is driven solely by how much effort they wish to put in, rather than any actual desire to leave the platform.

The reviewer’s interpretation is correct, and we would like to use this example as an opportunity to further clarify our model. When calculating regret, we consider a session that can last arbitrarily long, where the user has no intention of leaving the platform but is instead strategizing to maximize the value of the interaction. Our notion of a discount factor captures how patient the user is across these interactions or over what horizon they are capable of optimizing. As the reviewer pointed out, this is not the horizon that the user might intend to leave. We agree that if the user is aware of an early cutoff time within the session, they may prioritize immediate consumption over long-term optimization. However, if this cutoff is sufficiently far in the future—such as planning to delete the app a year from now—we expect our results to apply readily.

The paper is very theory-heavy - which I appreciate, as its main goal is clearly a theoretical understanding. However, I would have appreciated more explicit connection to practice or policy, even in the appendix. For example, what would the results in Section 6 imply about how algorithms should be designed?

Our goal in this theoretical paper is to provide a first-of-its-kind analysis of aligning with inconsistent preferences and the burden this places on strategic users. We have intentionally presented the model and results in abstract yet interpretable terms to highlight the generality of the framework and its potential applications. That said, we agree that additional context and stronger connections to practice would be helpful. To that end, we have included concrete examples in our response to Reviewer 7AFQ, which we hope the reviewer will also find useful.

Regarding the results in the presence of costly signaling (Sec 6), we show that giving users just one additional option—even one that simply burns effort—can significantly improve their ability to steer the algorithm. As noted by Reviewer 7AFQ, a small "show less like this" button, like those found on older platforms, can serve this purpose. Such costly actions, while not directly altering engagement with the content, send a strong signal to the platform and can ease the burden of alignment on the user.

From a policy perspective, our special case already illustrates that engagement maximization alone is not sufficient to motivate platforms to offer diverse options to users. In fact, the incentives often run in the opposite direction: platforms may reduce friction as much as possible, leading to the kind of seamless, mindless scrolling now ubiquitous in video recommenders. Of course, these are speculative insights derived from our model. Strong empirical work is needed to bridge theory and policy. As Dean et al. put it, we hope that formal mathematical models like ours can guide and inspire future empirical research by offering valuable insights—while recognizing that they do not provide final answers.

How would the results change if algorithms have different incentives beyond engagement?

This is a great question. Our theoretical framework does in fact accommodate this case. One simply needs to define $1/\alpha_\theta(s)$ as the expected value—or more generally, whatever measure of utility—the platform assigns to content $s$. The rest of our results then apply directly. We chose to define $1/\alpha_\theta(s)$ as expected engagement length in our exposition to simplify the presentation, as many platforms monetize based on engagement duration. However, our results are more general and extend beyond this specific interpretation.

We thank the reviewer for their thorough and insightful feedback. We couldn’t have asked for a more engaged and in-depth review of our work. We are especially pleased that the reviewer found the paper well-organized and our interpretations clear and intuitive. Below, we address the specific questions raised.

What recommendation setting motivates this work?

We have intentionally presented our setting in an abstract form because we believe it encompasses a wide range of applications—an observation echoed by reviewers, to our excitement. To provide a more concrete illustration and clarify the model’s assumptions and generality, we begin with a simple example of the interaction structure, followed by a discussion of how our user decision-making model builds on prior work. We apologize in advance as this response will be on the long side.

Consider the following example: a user opens a music or video recommender system with a specific intent. Upon seeing a recommendation—say, a country song—the user decides whether to engage (i.e., listen) or to skip and disengage for a while. Once the user chooses to engage, however, they may relinquish rational control over how long they remain engaged, with the actual duration influenced by the content’s emotional or addictive pull. Suppose this user is working while listening and would benefit most from calm music. However, they are a fan of a particular artist, X, whose songs may be more distracting. The platform profits from longer engagement—via ad revenue or increased interaction—and therefore has an incentive to repeatedly recommend content like X’s music. In this case, the user exhibits inconsistent preferences: they prefer calm music for productivity, but are tempted by content that undermines that goal. This type of user falls under our "type 1" users, where the reward and engagement duration are misaligned across content types (here, X’s music vs. calm music). Notably, in a different context—say, when the user is relaxing rather than working—their preferences may align with the platform’s and they are "type 2" users. This motivates our session-based modeling of interactions.

Our framework extends beyond recommender systems. For instance, a chatbot that charges per API call may operate in various "modes." In an educational mode, longer sessions may be valuable for a student, aligning incentives. But for an engineer seeking a quick answer, a shorter session is preferable. Similarly, a therapy chatbot operating in an "affirmative" mode might prolong interaction by being emotionally validating, even if it does not help the user confront deeper issues. These examples all fit our model, provided we flexibly define user types and item spaces, even approximately, and work with a tractable number of them.

Turning to our user model: there is ample empirical support for user strategizing. For example, Cem et al. observed that "[some users] ignore content they actually like to" in order to shape future recommendations. Likewise, Haupt et al. found that users "explicitly and actively curate their feeds in order to influence the platform to serve them content of a particular type in the future." These findings indicate that users engage in strategic behavior.

Our user model—where users strategically choose whether to engage with content, but then greedily decide how long to engage—is inspired by the seminal work of Kleinberg et al. As they note, human preferences are inherently inconsistent and psychologically rich, but their "two-mind" model offers a tractable abstraction. In this model, System 2 represents the long-run self with "actual" preferences and makes the engagement decision. Once the user engages, System 1—the impulsive self—takes over, and the duration of engagement becomes independent of System 2’s preferences.

This is exactly how we model inconsistent preferences. Kleinberg et al.’s framework, along with supporting references, directly informs our approach. Taken together, the empirical evidence for user strategizing and the two-mind model suggest that any strategic behavior arises at the engagement decision point (System 2), not in the engagement length (System 1).

How restrictive the special case in 3.4?

We acknowledge that the special case presented in Sec. 3.4 is restrictive in that it limits content to two "types." However, this abstraction may be more general than it appears: in many settings, item types can be interpreted broadly, and with some approximation, users may consistently benefit more from one type than from another, more tempting alternative. We believe our closed-form results likely extend beyond this case, as the underlying structure often appears as a component of more complex patterns observed in practice. That said, the theoretical results may no longer be as elegant or interpretable outside this setting.

Is a "show less like this" button seen on a few platforms currently an example of a costly signal? What would motivate a platforms to implement a costly signaling feature?

This is a great question—and in fact, the "show less like this" button was the original motivation for our research on how costly signaling can help! Regarding the burden on platforms, there is indeed a cost to them as well. As one can see in our special case, maximizing engagement alone does not incentivize the implementation of costly signals. While somewhat speculative, the shift from earlier social media platforms, which required more deliberate actions (e.g., clicks, menu options), to modern platforms featuring frictionless scrolling may reflect the platform’s disincentive to support such signaling mechanisms.

Modeling questions:

On a session-level, why would the algorithm discount future rewards?

We mainly view the possibility that the platform discounts future rewards as a theoretical curiosity that further generalizes our model. However, there are also practical scenarios where this assumption is reasonable. For instance, in time-sensitive campaigns, such as those with fixed deadlines, the timing of content consumption directly affects its value to the platform (for example, by going viral). Allowing for future-discounting algorithms also enabled us to uncover interesting insights—particularly regarding steerable sets, such as the observation in line 237: "as the algorithm becomes more foresighted, even slight disengagement from type 1 users can effectively influence its behavior."

When computing the algorithm's Q-value, why does the algorithm assume the users will continue to use the same strategy $f$ in the future?

We assume that within a fixed session, users commit to a strategy $f$ that depends only on the content. While we acknowledge that users may adopt more complex strategies—such as those that vary over time within a session—we currently have no empirical evidence supporting such behavior. We therefore focus on a simpler strategy, which is (1) more plausible given typical user behavior, and (2) more feasible for the platform to learn. That said, this is certainly an assumption, and we recognize its limitations.

Is pulling the expected duration into the reward $r_\theta$ (in line 120) really without loss of generality?

We have implicitly assumed that users discount rewards between interactions, but not during a single interaction. Under this assumption, line 120 holds without loss of generality. However, we acknowledge that if the duration of engagement is comparable to the time between interactions, it is indeed plausible that users also discount rewards within each engagement—introducing dependence to $\gamma_\mathcal{H}$. This would significantly complicate the analysis, as the distribution of engagement lengths (not just their expectation) would become relevant—an aspect our current model intentionally abstracts away for tractability—and likely only distorts the results negligibly. We thank the reviewer for noting this, and we will clarify the assumption more explicitly in the revised version.

What does "when users have consistent preferences ... the alignment problem reduces to engagement maximization" mean on page 1?

By the mentioned sentence, we simply mean that if $r_\theta(s)$ and $1/\alpha_\theta(s)$ induce the same ordering over $s$, then the user and the algorithm have aligned interests. In this case, both aim to reach the same item $s^*_\theta = \arg\max_s r_\theta(s)$, which also maximizes the expected engagement length $1/\alpha_\theta(s)$. We had some difficulty understanding the example provided by the reviewer here. If the reviewer is referring to a scenario where a content $s$ yields a reward only after $T$ steps of interaction have passed, then while this is indeed interesting, it falls outside the scope of our model. In our setting, the platform’s reward—equivalently, the engagement length—depends only on the content and a time discount factor, not on when during the session (e.g., before or after time $T$) the content is consumed. On the other hand, if the reviewer means that the reward from a content $s$ is realized only if the user remains engaged for at least length of $T$, then the platform’s reward becomes $1/\alpha_\theta(s) = \mathbb{E}_\theta[1\\{y > T\\}]$, in which case our analysis still holds as described. We hope this addresses the reviewer’s question and would be happy to further clarify if needed.

We thank the reviewer for taking the time to read and review our work. We’re glad they found our results interesting and hope they find our responses to their concerns below to be convincing.

the contributions are all theoretical … what are the practical implications for recommendation engines?

As Dean et al. put it, we believe that formal mathematical models like ours can offer valuable insights and help guide empirical research—particularly in evaluating and motivating alternative designs, such as our analysis of costly signaling. We fully agree with the reviewer that theoretical results, no matter how strong, are only part of the answer to broader policy questions.

In terms of implications, beyond formalizing the burden that alignment places on users with inconsistent preferences, our findings on how costly signaling can alleviate this burden have significant practical relevance. As our special case illustrates, platforms that maximize engagement length often have no incentive—and may even have a disincentive—to introduce friction or support user signaling. This is reflected in the widespread shift toward frictionless scrolling interfaces on video platforms.

However, our framework suggests that introducing diverse options—even those that don’t directly affect engagement but instead act as costly signals of user intent (such as a “show less like this” button, as suggested by Reviewer 7AFQ)—can help reduce the alignment burden. Our work also raises additional practical questions, such as how middleware tools (as discussed by Fukuyama et al.) might be used to extend users' effective horizons when their own patience is insufficient for alignment. We thank the reviewer for raising this important point and will be sure to expand on these connections in the updated manuscript.

Fukuyama, et al. (2020). Report of the working group on platform scale. Program on Democracy and the Internet, Stanford University

The authors motivate their example using recommendation algorithms. Yet the related work in that domain is not thoroughly discussed.

We’re sorry to hear that the reviewer found our related work section insufficient. We would like to clarify that our focus has primarily been on recent theoretical modeling efforts in this space, including works by Haupt et al., Kleinberg et al., Zhao et al., Haghtalab et al., and Hajiaghayi et al. While space constraints limited what we could include in the main text, we have done our best to expand on additional relevant work in Appendix A. We would greatly welcome any suggestions the reviewer may have for further enriching this section.

Are these results (easily) extendable? … it would be helpful to see a more general results, especially for Theorem 4.1

The abstraction in Sec. 3.4 and then Thm 4.1 may be more general than it appears: in many settings, item types can be interpreted broadly, and with reasonable approximations, users may consistently benefit more from one type over another, more tempting alternative. We believe our closed-form results here likely extend beyond this special case, as this structure often arises as a component of more complex patterns encountered in practice. That said, we currently do not have formal results beyond simulation evidence, and we expect that the theoretical results may become less elegant or interpretable outside this simplified setting.