(Part 1/3)

Thank you for the helpful suggestions and feedback! We are very glad that you see this work as “tackling an important problem”, our results to be “very interesting” and “quite persuasive”, and that you believe it has “big potential”. We respond to your specific comments below.

This paper proposes a cognitive interpretability framework (IMO, a Bayesian model selection framework similar to those used in iterated learning or rational speech model) to analyze the dynamics of LLM conducting in-context learning (ICL). The analysis can help us understand what the latent concept is (it seems hard to understand this term without reading [1*]) in ICL. 

[1*] Xie, Sang Michael, et al. "An explanation of in-context learning as implicit Bayesian inference." ICLR 2022

This approach indeed is similar to iterated learning and RSA in that we consider LLM behavior in a Bayesian model selection framework. However, our work differs from prior work such as [0] that apply iterated learning or RSA as techniques to improve LLM abilities. More similar to the evolutionary dynamics of [1], or the Bayesian concept learning dynamics of [2, 3] we study in-context learning dynamics. To our knowledge, our work is novel in being the first to systematically analyze dynamics of ICL over different input ($x$, in our notation) and output ($y$) token sequences. We have modified Section 2 to more clearly explain our framework without requiring reading [1*], and provide further discussion of Cognitive Interpretability in Appendix C.

[0] Zelikman, E., Mu, J., Goodman, N. D., & Wu, Y. T. (2022). Star: Self-taught reasoner bootstrapping reasoning with reasoning.

[1] Kirby, S., & Hurford, J. R. (2002). The emergence of linguistic structure: An overview of the iterated learning model.

[2] Ullman, T. D., Goodman, N. D., & Tenenbaum, J. B. (2012). Theory learning as stochastic search in the language of thought.

[3] Piantadosi, S. T., Tenenbaum, J. B., & Goodman, N. D. (2012). Bootstrapping in a language of thought: A formal model of numerical concept learning

I cannot give a clear-cut conclusion of what is the core contribution of this paper. It would be helpful if the authors could highlight some core concepts in a relatively simple way and explain the results more. 

Thank you for raising this question. We first summarize our core contributions as follows:

• Evaluating ICL as Bayesian model selection in a practical setting (Sec. 2). We find evidence of in-context learning operating as Bayesian model selection in LLMs in the wild, compared to prior work that trains smaller toy neural networks from scratch. This grounds past theoretical work, which used small neural networks trained from scratch, in simple phenomena observed with pre-trained blackbox LLMs, and it serves as a first step towards systematically analyzing in-context learning dynamics in LLMs and comparing text generation dynamics to simpler (cognitively inspired, in our case) models.
• Subjective Randomness as a domain for studying ICL dynamics (Sec. 3). We introduce a new domain, subjective randomness, which is rich enough to be theoretically interesting, but simple enough to enable analysis of complex behavioral patterns. Traditional few-shot learning domains are difficult to analyze because of the vast number of tokens that can be sampled at each step of ICL and token generation (i.e. Fig. 5 would be impossible to visualize), whereas subjective randomness simplifies text generation to a two-token format where learning dynamics are more apparent. 
• Sharp transitions in abilities to in-context learning dynamics (Fig. 4, 5).  We systematically analyze in-context learning dynamics in LLMs without observing or updating model weights, demonstrating sharp phase-changes in model behavior. This is a minimal, interpretable example of how the largest and most heavily fine-tuned LLMs can suddenly shift from one pattern of behavior to another during text generation, and supports theories of ICL as model selection.

 We added a new illustrative toy example with Figure 2 (also see Appendix D), to help readers understand the third point, and our claim that sharp transitions in the predictive distribution (i.e. token-level behavior) support that ICL is serving as model selection. Along with this toy example, we updated the text in Sections 2 and 3 to better explain these concepts as well as our method for evaluating formal concept learning in Generation tasks. We have also updated text accompanying each key result, summarizing our interpretation and how this result supports our claims.

(continued below)

Looking forward to seeing the code and camera-ready version.

(Part 3/3)

For Figure 2, what do x (green), LLM (yellow), and ICL (purple) mean? There is no such color in the figure. 

We apologize for any confusion here. This was an error due to a last-minute aesthetic change with Figure 1 (previously Figure 2), and the caption has been fixed now. Following the figure labels on the right-hand side, $x$ is the input (top, blue), the LLM’s latent concept space is represented by the middle box. In the middle box, latent concepts selected by ICL are red and have a shaded box behind them.

I feel confused about the notation p(y=Random|x), I guess y∈[H,T]n or y∈[0,1]n is the output sequence. Should it be p(h=Random|x)?

We note that the current notation, p(y = Random | x), is actually correct. Though we do indeed consider Random and Non-Random hypotheses, our main metric is the predictive distribution p(y | x) of output tokens $y$. For Randomness Judgment tasks, we measure the probabilities for the tokens $y$ "Random" and "Non" (-Random), as in the prompt "... The sequence was generated by a _____". In Randomness Generation tasks, $y$ instead represents token sequences as you suggest, as in the prompt: "Generate a sequence of 1000 random samples … A: [Heads, Tails, _____".

In section 4, the notation of x¯t−w…t   in  p(y|x)  is not quite clear. I guess it should be x¯t−w,…,t

Thank you for this feedback! We have updated the notation in this formula to include commas, as well as other formulas with similar notation.

Figures like Figure-5-right are quite hard to understand. What is the take-away message? Is this saying GPT performs more similar to the window average model and less similar to the Bernoulli model?

Yes! That indeed is one key takeaway message. A second message is that GPT is not simply outputting the same memorized sequences–-instead, we are seeing evidence of an interesting, algorithmic capability. Each text generation sequence has a unique trajectory, as demonstrated by the running average lines, and each path stays relatively close to the sequence mean. If flip sequences relied on simple memorization, then certain running average trajectories would be much darker than others. To improve legibility and address your concerns, we note that we have also edited the text around sections 4 and 5 to better explain this point, though due to space limitations, we cannot mention it in this figure’s caption (now Figure 6).

A third message is that, when looking at the figure, we immediately see interesting dynamics over text generation. Most ICL and few-shot learning work assumes that additional context, or shots, lead to stable and consistent improvements in accuracy. In contrast to that view, here we see sequences of tokens with interesting token-level behavioral dynamics.

In the last paragraph of section 4, what is the C in x=C|x| and yt∈C means. Where is C defined? (I can find c and C in section 2, but cannot find C).

Following commonly used notation used in Bayesian modeling to define posterior predictive distributions, $C$ is the concept pattern, for example, if $C = (011)^n$ and $y = 01101101$, then $y \in C$. To address your concerns, we have edited this part of the text to clarify this definition, as well as the related notation $p(y \in C | x)$.

Rebuttals Summary: Thank you again for your feedback! Following your suggestions, we have made changes to our submission, including trimming the introductory sections of our writing and devoting more text (e.g. at the end of section 2 and the end of section 4) as well as an introductory figure (Figure 2) to helping readers understand our Bayesian modeling framework. To your point about wondering what the broader impact of this work is, we further emphasize the point that the phase changes we observe in ICL may be present or even common in real-world few-shot learning domains as additional “shots” are provided for a given prompt (longer $|x|$, in our notation), particularly when individual data sequences $x$ are considered rather than averaging over many samples. Additionally, we have added new experiments to our Appendix, including an experiment (Appendix E) which injects “Heads, Tails, …” flip sequences into MMLU task. Though it was not the goal of this experiment, we observe compelling in-context learning curves, initial evidence that we may find complex, non-linear ICL dynamics in “realistic” few-shot learning tasks. We hope our changes and individual responses to your points address your raised concerns and hope that you will consider raising your score to support the acceptance of our work.

(Part 2/3)

The paper is very hard to follow. Section 1, 2, and 3 provides the background, preliminaries, and the system setting of the paper, but I find it hard to extract the system setting of the paper. Most of the text in this part discusses some insights and hypotheses from related works in these two fields. It would be helpful to squeeze this part and discuss the related works in another section.

Thank you for the suggestions. We have trimmed down the first two introductory sections of our paper. Instead, we place more emphasis on connecting our theories with our results, and describing the significance of both.

The results on binary sequence are good, but it is not sure whether the findings can be extended to other realistic tasks. It would be helpful if the author could verify these findings in more real datasets, e.g., math problems, question answering, etc.

We have added an additional sets of experiments which may help this address this question: (a) appending binary sequences to zero-shot learning prompts for MMLU and testing whether the LLM changes tasks to generating coin flips (see Appendix E), and (b) Randomness Judgment tasks with longer length formal concepts. In a response to Reviewer F7kD, we describe challenges with using more complex concepts. For (a), though thorough analysis of ICL dynamics was not an intended goal of the MMLU experiment, we do find interesting non-linear ICL dynamics in a domain that is closer to a real-world task. 

Additionally, we added new text, particularly Section 2, explaining the significance of our findings of ICL phase changes and model selection. Our work is the first such demonstration that we know of, and we see enormous opportunity in future research that builds on our work, analyzing in-context learning dynamics in domains typically used for few-shot learning in math problems and question-answering. The sorts of sharp transitions in ICL we observe may be common in other few-shot learning domains, meaning that with just one or two more in-context samples, model behavior might change suddenly from one pattern to another. We see this work as an initial consideration and demonstration of this widely impactful principle for ICL.

There is not so much discussion about the Bayesian model. Going deeper in this direction will help the paper a lot.

Thank you for this suggestion! We have significantly improved discussion of the Bayesian model-–please see updated Sections 2 and 4. We have also added a new figure with an illustrative example, Figure 2, and an accompanying Appendix D, to help the reader connect our claims with our empirical results. We have also improved discussion of our Bayesian methodology for analyzing the LLM posterior predictive distribution with formal concept learning in flip Generation tasks (end of Section 4).

(continued below)

(Part 4/4)

Please review all figures in the body of the paper for colorblind-accessability. I am colorblind and had a hard time deciphering Figure 8. There are plenty of guides online on how to make accessible figures, and I am happy to point you to specific ones if that is needed.

Thank you for raising this to our attention. In the next few days, we will work on updating our figures with colorblind-accessible color schemes. Though we have found guides on color blind accessibility, if you have any particular suggestions for useful guides, we would appreciate pointers to them. We want to actively engage in learning how to make inclusive figures.

What does "Probability of Concept" mean on the y axis of Figure 8? … (See also the last question in Questions section.) … The mathematical notation in the sum on page 7 was confusing (the righthand side that following "and computing"): I didn't understand what is (yd==1)(i)...

We have updated this section of the paper, at the end of Section 4, to make our methodology with formal concept learning in Generation tasks clearer to the reader. We have updated the y-axis of Figure 8 (now Figure 5) Right to match the notation used in this section, the predictive probability $p(y \in C | x)$ of a generated sequence $y$ matching concept $C$.

This figure is showing results for formal concept learning on a Generation task, as is the tree on the Left of this figure. We believe the reviewer interpreted this figure as results for a Judgment task, as in Figure 4. However, we emphasize that this figure is actually showing sharp transitions in a completely different task from Figure 4, which makes this result more surprising. Instead of answering whether the source of a sequence was y=”Random” or y=”Non-Random”, the model simply keeps generating tokens (e.g. y=”Heads, Tails, Heads, Tails, …”) to continue an input sequence of tokens $x$, and once again we see transition points, where GPT suddenly shifts from generating “random” sequences to deterministically repeating the concept in the input.

Thank you for introducing me to the area of subjective randomness. Can you confirm that my understanding of it is correct, as far as needed to understand this paper? Am I missing a key insight from this area?

We are glad that you appreciate the introduction to subjective randomness–we are really excited about this topic and believe it is fertile ground for future work! We believe there are a few points of clarification that warranted discussion in the interpretation of our claims and results, which we have discussed in our responses above. Please let us know if you still have any questions!

When talking about distinguishing ability, the correct quantity to look at is the "advantage" of the distinguisher: … Can you confirm that this is the quantity portrayed in Figure 8? … without any real gain in advantage.

We apologize, but it seems due to a lack of background on the topics mentioned in your comment, we are unable to precisely understand your comment. This makes it difficult for us to confirm whether your interpretation of Figure 8 (now Fig. 5) is correct. It is possible that the question echoes our clarification point (2), since we never presented models with multiple different strings, possibly from different sources, simultaneously. We do note that we have updated the text corresponding with Figure 8 (now Figure 5)---maybe this helps clarify some of the points raised in the question above.

Rebuttals Summary: We hope that our updated submission addresses some of your concerns. We have updated our paper following your suggestions, including a new figure (Figure 2, Appendix D) with an illustrative example to help readers connect our claims with our findings. This example reinforces a key point of our work, that we interpret sharp phase changes in ICL as indicative of model selection. We have also added new experiments, such as those in Appendix G, which may address your point about non-random concept simplicity. After considering these changes, as well as our responses to your individual points above, we hope that you will consider raising your score.

(Part 1/4)

Thank you for the helpful suggestions and feedback! We are very glad that you see this work as “important”, “creative”, “original”, “interdisciplinary”, and “well-written”. We respond to your specific comments below, beginning with some clarifications on two key points of our work that may have been misinterpreted.

Points for further clarification:

1. The transitions we observe, for example in Randomness Judgment tasks, are (a) sharp and sudden, and (b) move from a state of strongly following one response pattern (“Random”) to strongly following a very different response pattern (“Non-Random”). Both points are critical to understanding the significance of this result and we have added further clarification in Section 2 and Figure 2 to explain this point, contrasting model selection with model averaging interpretations of ICL, and how our experiments lend support for the former. To our knowledge, our work is novel in being the first example of such a dramatic phase change, or S-shaped curve, in ICL or few-shot learning with an LLM. We observe these in-context learning dynamics in our formal concept learning experiments across two separate tasks, with separate analyses: flip sequence Generation, and randomness Judgment. The implications of this phenomenon may be significant if such in-context phase changes are found in other few-shot learning domains.

2. None of our experiments involved prompting an LLM with two strings simultaneously, i.e., we never task an LLM with determining which string was generated by a given source. Some of the questions in your review suggest that there may have been a misinterpretation of our experimental protocol, though it is possible that this is merely an issue with phrasing. To clarify: our experiments tested LLM (a) inferences about the source that generated a sequence of flips (i.e. was the source “Random” or “Non-Random”?), and (b) next-token completions for a sequence of flips, e.g. “[Heads, Tails, Heads, ____”   (see Section 4 for further details).

For the distinguishing experiments, …. The authors find that each language has a sharp phase shift …  the likelihood that GPT predicts the context to be determinstically-generated increases from ~0.5 (a random guess) at n=3, to ~0.8 at n=4.

We would like to clarify some important details of our results. The specific values you list seem to be inaccurate (.5, .8) based on Figures 4, 5, which is related to our clarification point (1) above. The reason this experiment is interesting is because the likelihood sharply changes from high confidence in a random process to high confidence in a non-random algorithm. That this change is sharp, and the LLM transitions from strongly following one pattern to strongly following another, are both essential parts of why this result is significant. In addition, the phrasing in the second half of the first sentence implies that we tested GPT with uniformly random sequences $x$ in the Randomness Judgment condition. Following Reviewer xnie’s comments (a different reviewer), our updated version of the paper includes such an analysis, however, our original pdf did not include this.

Usage of the term "pseudorandomness"

Thank you for this useful input! We were not aware of this being established terminology in computer science, and we have tweaked our paper to remove “pseudorandom” to avoid this confusion. Our intended meaning is, as you inferred, “seemingly random”, or “subjectively random” as in the cognitive science literature.

(continued below)

(Part 2/2)

“ChatGPT generates higher complexity sequences than chance. …” I’m unsure what this means - can you elaborate what you mean by higher complexity in this context?

We interpreted your comment as a request for clarification about this remark, please let us know if this was not your intent and we’ll follow up. Accordingly, we have updated this line in the paper to clarify this point: “... if the LLM makes a sequence more subjectively random by avoiding repeating sub-sequences, like a human might, then it will produce sequences with higher-than-chance complexity overall.”.

What is the tokenizer doing? Given that the whole paper is based around heads/tails sequences, how are they being represented? … prompts specify generating 1000 tokens, but only ever the first 50 are used? Is there a particular reason for that mismatch?

We appreciate your questions, as these prompting details were intentional, and we have updated Appendix B to better clarify them. We chose the format “Heads, Tails, …” specifically to avoid the alternate interpretation you describe, of some patterns merely being tokenization artifacts, since flips with shorter string formats, such as 0s and 1s with no comma separating them, often get chunked into different tokens (“01”, “000”, etc.). We specify 1000 tokens to avoid issues with GPT generating too few tokens, where for example if the '50 samples' is specified, less than 50 flips will be generated in some cases. In the future, we hope to explore how different token representations influence results, similar to [0], but with black-box in-context learning behavior.

[0] Wei et al. (2023). Larger language models do in-context learning differently.

Rebuttals summary: Thank you again for your feedback! In response to your point about concept learning in generation tasks, we updated the prompt for this task and re-ran the corresponding experiment, with relatively similar results. Diving deeper into this point, we conducted a new experiment to test whether, as you suggest, GPT is merely forgetting the prompt when it begins repeating the deterministic pattern of one of our concepts. Our results suggest not; instead, GPT has a strong bias to stay on-task for an unrelated task (MMLU benchmark questions), even after many “Heads, Tails, …” tokens are appended to the prompt. We also added a Randomness Judgment baseline using sequences sampled from a Bernoulli distribution (Figure 20), following your suggestion, and found these in-context learning curves to have very different patterns than our repeating concepts. Besides these changes, we have also made smaller tweaks following some of your particular suggestions, e.g. adding tokenization details to Appendix B. We hope these updates address your concerns and hope that you will consider raising your score to support the acceptance of our work.

(Part 1/2)

Thank you for the helpful suggestions and feedback! We appreciate that you find our paper “well written” and “easy to follow”, and our experimental setting to be “well chosen” and “novel”. We are also glad that you appreciated our claim that phase changes in ICL suggest model selection, which we sincerely believe is a fertile avenue for future research!

For this reason, in Fig. 6 we show results for P (Tails) = 51%. It would be clearer to also update Figure 6 to show that it is 51%, as opposed to only mentioning it in the text.

Good point, we have updated Figure 8 (previously 6) to state this.

“In other words … it’s just dominated by the local signal. For example, if you put in any prompt, then a long enough sequence of 001001, it may ignore the prompt and start predicting 001001… as it’s “forgotten” about the prompt due to the strong local predictable signal. … One way to address mismatch could be the prompt being explicit about it being an ICL problem. For example: ‘Continue the following potentially random sequence: <...>’”

These are good questions, and we have run new experiments to address these points. 

Experiment 1 (on prompt specification): We re-ran the formal concept learning task in randomness generation with a different prompt, analogous to what you have suggested: “Generate a sequence of 1000 samples that may be from a fair coin with no correlation, or from some non-random algorithm”. We observed similar results, with stable transitions from random behavior and deterministic concept learning, and we have updated Figure 5 and accompanying text accordingly (note that this updated figure also includes 4-length concepts, which the previous version did not).

Experiment 2 (on local bias): In Appendix E , we provide a new experiment to demonstrate that sharp phase changes in ICL as a function of additional context cannot be simply attributed to local bias leading the model to copy the pattern of the last few tokens of text, regardless of the overall task. Specifically, we append a sequence of “Heads, Tails, …” (i.e. the concept (HT)+ ) to the end of a zero-shot prompt for a widely used LLM evaluation dataset, MMLU, testing 5 different subjects. In this experiment, we show that GPT shows a strong task bias, rather than merely being “forgetting” the prompt and being “dominated by the local signal”. Even after appending hundreds of distractor tokens following a repeating pattern “Heads, Tails, Heads, …” to an MMLU prompt, GPT reliably converges back to answering the MMLU question and maintains a high probability of generating answer tokens (“A”, “B”, “C”, “D”, since MMLU is multiple choice) over the course of in-context learning. We provide further discussion in Appendix E.

“Additionally, experiments for determining whether a sequence is random should include some truly random baselines. … What is the pattern where the LLM starts to fail?”

Thank you for this suggestion! To address this point, we have added random baselines for Randomness Judgment to Appendix G as well as accompanying discussion. ICL dynamics for random sequences vary widely, and do not show the characteristic sharp transitions we find for our non-random concepts, further corroborating our perspective on ICL as model selection instead of model averaging.

(continued below)

(Part 2/2)

How is the proposed "Cognitive Interpretability" different from ... For example, some line of works studied the the attention map of convolutional neural networks when fed certain inputs [0].

Thank you for this comment, which prompted us to make a standalone background section in the appendix (see Appendix C.) Therein, we further define Cognitive Interpretability and describe its significance. In brief, this is different from [0] in two important ways. First, we match cognitive models with behavioral observations of deep NNs. Our window average model, as well as our framework of ICL as model selection (or, equivalently, hypothesis search) are inspired by models and theories from cognitive science. While [0] makes anecdotal analogies between visual circuits in artificial NNs and circuits in the human visual cortex, we provide formal models and compare behavior and learning dynamics between an LLM and these models. Second, we analyze in-context learning dynamics, where behavior changes as a function of addition context data, i.e., with increasing $x$ in length. This is similar to how learning dynamics are studied and modeled in cognitive science, and LLMs present new opportunities for understanding the structure of behavior underlying ICL capabilities. Unlike [0], we do not examine hidden units in deep NNs, we analyze input-output dynamics.

Do you see a link between this paper and the emerging line of work on the equivalence of ICL to gradient descent [1]

Indeed! Our original draft cites [1] in the second paragraph and discusses this perspective. To emphasize this relationship further, we have also updated our paper to emphasize our framework of ICL as discrete hypothesis search. We hope these updates will further clarify this point, so readers immediately see the connection of our work to previous work such as [1]. A key takeaway is that, unlike most prior work viewing ICL as gradient descent or as few-shot learning with loss steadily decreasing with more samples (“shots”), we find sharp phase changes in ICL, analogous to phase changes observed when training NNs. 

Rebuttals summary: Thank you again for your feedback. Based on your suggestions, we have shortened the background sections of the paper and moved discussion of Cognitive Interpretability to Appendix C, hence improving the paper’s presentation and addressing your concern about background details. We also agree with your point that open-source LLMs are important for replicability and we are developing a pipeline to replicate our experiments with open-source LLMs. We hope to add initial randomness Generation results with LLaMa 2 by the end of the review period, but are limited by time and resource constraints currently. We promise these results will be included in the final version if we’re unable to meet the deadline right now. We also emphasize we have made further edits to the submission, including additional experiments and a new explanatory figure. In considering these and our responses to individual points you raised above, we hope that you will consider increasing your score to support the acceptance of our paper.

(Part 1/2)

Thank you for your useful feedback! We are encouraged that you found our work to be “novel”, “of great interest”, and “enjoyable to read”.

I found the paper to utilize a lot of its space to discuss ideas that seems unrelated to what I understand to be its main message. ... I don't understand the value of Figure 1 in the dissemination of the idea.

Thank you for the suggestion. Given that subjective randomness and Bayesian cognitive modeling are not well known across the ML community, our goal was to describe relevant background and highlight the novelty of this work in analyzing token-level in-context learning dynamics. However, we agree that this background can be made more succinct.   We have accordingly  trimmed sections 1 and 2, and moved what was previously figure 1 to Appendix C for further discussion of cognitive interpretability. This reduces the background content to ~1.5 pages now. We have used this space to expand the discussion of our results (see updated text in blue in the latter half of the paper).

The paper seems a little bit rushed. For example, the caption of Figure 2 seems partly unrelated or outdated to the figure. ... Figure 5 left seems to be the same figure three times.

We appreciate these useful suggestions and have made changes accordingly, though we also stress that our modeling framework and experimental design are thorough and deliberate, and were not created in a hurry. Specifically, we note that in Figure 6 right (previously Fig. 5 left), though the graph structures are the same for all three markov chains, arrow width varies between the three figures. We have amended the caption to emphasize the key takeaway of Fig 6 right - that the top plot has uniform arrows, whereas the middle and bottom figures are irregular but similar to one another. While all figures in the appendix had text accompanying them, a couple figures only had captions and not further discussion. We have updated these sections with further discussion of the results in these figures. For your comment on Figure 1 (previously Fig. 2), we concede that the error you pointed out occurred due to a last minute change in figure aesthetics. We have updated the figure caption to address this and apologize for any confusion.

The paper utilize closed products ... authors are encouraged to use open models or demonstrate that the observed behaviors cannot be attained with the currently available open-source models.

This is a very reasonable point, and we agree that this is important for future replicability. We have accordingly begun conducting Randomness Generation experiments with Llama 2 (7b and 13b) using the hugging face API. Because of limited time, we have not yet included these in the paper, though we are trying our best to add some results before the end of the review discussion period and to add further replications of our other experiments thereafter. Due to resource constraints, it is unlikely that we will be able to run Llama 2 70b during this time period, however, in the future we may be able to acquire computational resources to run the largest open-source LLMs on our tasks. 

(continued below)

I thank the authors for their careful response. While I am raising my score to a 6, I encourage the authors to conclude their experiment on an open model and report their finding whether positive or negative.