Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning

We thank the reviewer for their positive assessment of our submission, and address their comments individually below.

---

Weaknesses

Latent learning progress wasn't formally defined until well into the middle of the paper, and there it was only operationalized in terms of the specific experimental setup rather than something more general.

We provide an intuitive explanation of LLP in the introduction, and save a formal definition for later in the paper, where all other related concepts have been introduced, to facilitate the reader’s understanding of such definition. The point about a generalized version of the LLP definition was addressed in the global response.

---

While motivated by work in AI, the paper isn't really written in a way that will have a broad appeal to an AI audience. I'm not sure how much impact this work will have at least within AI.

As noted above, NeurIPS welcomes submissions from “Neuroscience and cognitive science” – therefore, we invite the reviewer to consider our contribution to such fields. That said, we provide several connections to the artificial intelligence and machine learning literature in the Introduction, Related Work, and Discussion sections of the article, and would welcome the reviewer’s suggestions to further bridge across different fields.

---

Minor

p. 1: "Along factors" -> "Along side factors" p. 4: "in which factor" -> "in which factors"; "combination of factors" -> "combinations of factors"

Minor typos have been fixed in the camera-ready version.

---

Questions

Can the authors provide a more general formal definition of latent learning progress?

This point was addressed in the global response.

---

How can latent learning progress be defined in such a way that it's likely to be both general and useful? My concern is that there are probably many versions of this which are not useful, depending on how one defines the latent variable whose progress is being monitored.

This point was addressed in the global response.

---

Limitations

The authors have adequately described the paper's limitations. I don't see any potential negative societal impacts.

We included potential societal impacts in our initial submission, but we expanded it in response to reviewer y4se: “As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences in marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.”

We thank the reviewer for their positive comments and their extremely detailed, helpful feedback. We address their suggestions for improvement below.

---

Weaknesses

First, the experimental paradigm is very specific. Users have to pick from a limited list of goals with the explicit over-arching goal of being able to create all of the potions successfully. Conversely, people usually generate their own goals (e.g., "I'd like to read this book", "I'd like to go on cross-country trip to visit Aunt Ida") rather than picking from a specific list, and they often choose these goals without any supervised over-arching goal.

We completely agree with the reviewer! In fact, we noted “In our task, people chose goals from a predefined menu of options. Although this facilitates the study of goal setting, people often invent their own goals by combining observations and imagination”. Having a highly controlled setup in which to study goal selection was essential for a first understanding of people’s approach, and our clustering analyses and plots of individual behaviors highlight vast variability in goal selection strategies even in such a limited space. Building on our initial findings, it may be possible to start exploring how people create and pursue more free-form goals (if not as rich as the examples mentioned by the reviewer, at least more self-designed than the ones we studied here). We note that our is a standard approach to studying a novel, complex question: we start from a simplified experimental design to establish findings in well controlled setting, and plan to adapt our approach to increasingly more naturalistic, complex settings to pursue a more nuanced understanding.

---

Second, the participants were university students (mostly female) participating for course credit. Would the results generalize to different populations?

This point was addressed in the global response.

---

Third, the measure of latent learning progress (LLP) is entirely specific to this experimental task. The authors operationally define latent learning progress as 1 - (N actions sequences tested / N possible action sequences). This measure only applies in situations where there are a finite number of options to try. How can future researchers extend this notion of LLP to open-ended tasks that use high-dimensional sensory and motor spaces (where this operational definition would not work anymore)?

This point was addressed in the global response.

---

On a separate note, the statistical analyses need effect sizes. (For example, a simple Cohen's d for each Wilcoxon / t-test would be suffice). [...].

We now provide the effect size for all Wilcoxon tests as the standardized effect size r (Z/√N; Rosenthal et al., 1994; see attached PDF, “Updated statistics with effect sizes”). Note that we have replaced the symbol “Z” with “W” for non-standardized test statistics in Wilcoxon tests.

---

Also, if I'm reading the stats correctly, the results in text do not match Fig 2A: Complex hierarchical G3 mean performance is reported as .37, but the bar in Fig 2A looks like it couldn't be any higher than .3.

We are grateful to the reviewer for noticing this mismatch! The stats are correct, and we have updated the figure to match them (see attached PDF, “Updated Figure 2”). The previous figure was also correct, but aggregated the data slightly differently. The two are now consistent.

---

Minor

The in-text references are sometimes formatted a little strangely in terms of the number [...].

We thank the reviewer for noticing the citation formatting error, which we have fixed for the camera-ready version.

---

The explanation of the experimental game (section 3.1) is not that clear. The figure (Fig 1) is definitely helpful, and the description in the appendix clarifies the game really well. You might consider reformatting the paragraph describing all of the goals into a list format.

We thank the reviewer for their suggestion and for consulting the Appendix thoroughly. We had initially placed a full description of the task in the main body of the paper but had to move it to the Appendix due to space limitations. We intended to provide the reader with enough information about the task to understand the main takeaways and refer interested readers, with a more specific interest in cognitive science experiments, to details in the Appendix.

---

Figure 3 is difficult to understand as-is, but I think it might actually be a really interesting figure. Part of what makes it unclear is the y-axis: What is the "action sequence index"? I suspect there might be an alternative to a dot plot that would be easier to read (maybe even forgoing the y-axis and showing a single strip of color-coded trials for each subject perhaps?). It would also help the readers to add some descriptors to each graph so the reader knows which elements to look for (e.g., setting a few objectives rather than trying all goals, unprincipled switching between goals).

We thank the reviewer for their insightful analysis of Figure 3. “Action sequence index” refers to the unique combination of ingredients selected by the participants and their order (e.g., the first point would be [0, 1, 2, 3] – where 0 is the top-most ingredient on the screen, 1 the second, etc.). To improve clarity, we have relabeled the y-axis “Action sequence”, since the index is not defined in the text. We omitted the axis labels to avoid cluttering the figure, making it unnecessarily large, and overloading the reader with information. We used the preceding paragraph to guide the reader through specific aspects of the figure they should be focusing on, and would therefore find it redundant to repeat the information in the figure caption.

We thank the reviewer for their excellent summary of our article and their thorough feedback. We address their suggestions for improvement below.

---

Weaknesses

The definition of LLP introduced here is quite specific to a small and discrete space of action sequences. A more general definition or some preliminary discussion on how one could extend the current work to continuous or countably infinite spaces of action sequences would strengthen the work.

This point was addressed in the global response.

---

Questions

The goal selection model considers a weighted sum for combining the factors that contribute to the goal values. Did you consider or test more complex functions of the factors?

In our initial analyses, we indeed considered more complex functions of the factors. . Specifically, we tried modeling more complex interactions in our current setup (e.g., interactions between performance and other goal selection motives), but did not find this procedure to improve fit. Moreover, we note that we already observe interesting dynamics with a simple model, which is preferable for interpretability. It is possible that more complex functions of the factors might be relevant in more complex settings, and this will be an important question for future research. Here, we follow the standard “Occam’s razor” approach of the lowest sufficient complexity of the model given our data (Wilson and Collins, 2019).

---

Limitations

Limitations are discussed in the conclusion. The authors do mention that potential future ethical concerns should be addressed, but do not expand on what those concerns may be.

We thank the reviewer for the opportunity to expand on such concerns, and we now state that : “As our knowledge of human goal setting becomes more precise, however, ethical concerns regarding the use of behavioral sciences in marketing and management should be addressed, particularly in cases where highly personalized methods of influence could transform advertising into manipulation.”

We thank reviewer kdQC for examining our submission and recognizing its importance and potential impact on machine learning. In the rebuttal period, we hope the reviewer could also help clarify what would improve soundness and presentation beyond the point raised so far. Regarding contribution, we would like to stress the fact that NeurIPS welcomes submissions from “Neuroscience and cognitive science” – therefore, we invite the reviewer to consider our contribution to such fields. We address all of the reviewer’s stated concerns below.

---

Weaknesses

All the subjects of the human experiment are students, which may introduce bias.

This point was addressed in the global response.

---

The definition of V_f for each kind of model is unclear: e.g. in "performance", what does "currently active goal" mean? Where does r^t come from? What is the relationship between it and the goal?

The currently active goal is the goal a participant has chosen on a given trial. To make our statement clearer, we now say “selected goal” instead of “currently active goal”. $r^{t}$ comes from the feedback provided to participants (the target potion either fills up or remains empty), as specified in the same sentence: “1 for positive feedback, 0 for negative feedback”. Feedback is goal-contingent, such that participants only receive feedback relative to the selected goal. We clarify this further in the text: “On each trial, the utility of the selected goal with respect to performance is updated based on the goal-contingent feedback received on that trial $r^{t}$”.

---

The design of LLP, which involves selecting multiple action sequences, does not provide a complete explanation of how LLP can improve human choices or actions better than LP.

As mentioned in the paper, “unlike LP, LLP does not require observing performance improvements to provide informative signals about progress.” This would make LLP particularly useful in situations (mentioned in the paper) “where no external change is visible, yet some other form of progress toward the desired outcome is made. Imagine being tasked with identifying the correct sequence of numbers to open a combination lock, which you might attempt through trial and error. Throughout most of this scenario, repeated failures would yield no difference in performance, hence no empirical learning progress (as typically defined). Nonetheless, provided that the lock has a limited number of slots and numbers and that you can avoid repeating incorrect combinations, every attempt is a step toward the solution.”

---

Questions

Hierarchy influences goal selection through latent learning processes (LLP). Will separating the evaluation of hierarchy and LLP affect the results?

Hierarchical components are a design choice in our experimental paradigm. Removing the hierarchical feature would likely make it more difficult to tease LLP and LP apart. We find, however, that exactly how hierarchy and LLP relate to and influence each other is an interesting question. Our current findings suggest that “hierarchy impacts goal selection indirectly by enabling inferences and thus affecting LLP”.

---

Section 6 discusses individual differences. Has there been an analysis of the causes behind these differences?

The goal of the current paper was not to identify the sources of individual differences in goal selection. Noting that such differences exist is in fact a novelty of our submission. That said, we agree with the reviewer that the question of individual differences is worthy of further exploration, and we mention it would be interesting if future studies crossed findings similar to ours with participants’ data related to “demographics and cultural background, cognitive abilities, and psychopathology”.