RAIN: Your Language Models Can Align Themselves without Finetuning

Your thorough and insightful review has provided essential guidance for us. We are thankful for the opportunity to clarify and enhance our work based on your suggestions. Below, we address each of your queries and provide clarifications where needed, especially concerning the weaknesses you've identified.

Q1. Baselines: The paper does not provide a detailed comparison of RAIN with other methods that align pre-trained language models using RLHF or instruction tuning.

Indeed, we have conducted such comparative experiments. As detailed in Figure 4 on page 8 of our paper, our comparison includes RLHF and RLAIF, both of which involve SFT (akin to instruction tuning) and RL phases. The results from these comparisons show that while RLHF and RLAIF require additional data, RAIN demonstrates improved performance.

Furthermore, we have tested RAIN on models that were aligned using various methods. These experiments are presented in Table 1 on page 7, and in Table 2 on page 8. The results from these tables indicate that RAIN is capable of further enhancing the safety of models that have already been aligned using existing methods.

Q2. Limited to fixed pre-trained models: RAIN is designed to align fixed pre-trained language models and cannot be used to align models that are being trained or fine-tuned.

Contrary to the impression that RAIN is restricted to fixed pre-trained models, it is indeed designed to work with any auto-regressive language model. Vicuna is a series of models fine-tuned from the LLaMA series. Our experiments demonstrate (as shown in Figure 1 on page 2, Figure 3 on page 7, and Table 1 on page 7) that RAIN is effective on both LLaMA and Vicuna. Similarly, LLaMA-2-chat is a series of models fine-tuned from the LLaMA-2 series. Our findings, as illustrated in Figure 1 on page 2, Figure 3 on page 7, and Table 2 on page 8, confirm the effectiveness of RAIN on both LLaMA-2 and LLaMA-2-chat. This underscores the fact that RAIN's effectiveness is not contingent on the fine-tuning process. We argue that inference with models that are actively being trained or fine-tuned inherently involves using a particular checkpoint, which is conceptually similar to using fine-tuned models like Vicuna (fine-tuned from LLaMA). Thus, RAIN's methodology is equally applicable and effective in these scenarios as well.

Q3. Limited evaluation on real-world scenarios: The experiments in the paper are conducted on benchmark datasets and may not fully capture the complexity and diversity of real-world scenarios.

Thank you for your feedback. We have conducted human evaluations, which closely resemble assessments in real-world scenarios. The results, presented in Table 8 on page 9 of our paper, clearly show that RAIN also performs effectively under human evaluation. This approach not only validates the efficacy of RAIN in a controlled experimental setup but also demonstrates its practical applicability in real-world contexts.

Q4. What are the potential limitations or drawbacks of RAIN, and how can they be addressed in future work?

In the second paragraph of Section 5, "CONCLUSION", we discuss the limitations of RAIN and propose avenues for future work. We acknowledge that compared to standard auto-regressive inference, RAIN demands a longer yet acceptable inference time. One potential solution to this issue, as we discuss, is to use RAIN to generate data and subsequently fine-tune the model with this data. This approach aims to offset the additional inference overhead by incorporating a fine-tuning phase.

Q5. How scalable is RAIN to larger models or datasets?

One of the key advantages of RAIN is that it does not require additional memory overhead, which significantly aids its scalability to larger models. To demonstrate this, we have conducted extensive experiments with models as large as 70B parameters, which is the upper limit in the LLaMA series available to us. The results presented in Figure 1 on page 2 and Figure 3 on page 7, along with Table 1 on page 7 in our paper, clearly illustrate that the effectiveness of RAIN tends to increase with larger models. Furthermore, Table 7 on page 9 demonstrates that the relative additional time overhead required by RAIN diminishes as the size of the model increases. Additionally, RAIN is a universal, data-agnostic, and training-data-free algorithm, making it inherently scalable to large test datasets.

Thank you for your comprehensive review and the valuable insights offered. Your feedback is instrumental in helping us improve our manuscript. In the following, we have made an effort to clarify the aspects of our work that you found lacking.

Q1. The proposed inference-time method has high computational cost. This will make it challenging for the method to be deployed in practice. (On the other hand, one can perform fine-tuning on generated samples so that the sampling approach need not be performed at inference time; the authors didn't not explore this approach.)

In the conclusion section, we discuss the limitations of RAIN: Compared to standard auto-regressive inference, RAIN requires a longer, yet acceptable, inference time. A potential solution to this issue could involve using RAIN to generate data and then fine-tuning models with this data. However, fine-tuning LLMs is resource-intensive and contradicts the aim of our paper, which is to "investigate the feasibility of enabling (self-)alignment in fixed LLMs without fine-tuning." Therefore, we have deferred this approach to future work.

Q2. The proposed method is complicated.

Intuitively, RAIN conducts Monte-Carlo search on the tree consisting of token sets and dynamically reduces the weight of harmful token sets, with backward rewind and forward generation steps until the output content is self-evaluated as harmless. We also modify the Monte-Carlo tree search to adapt to the LLM alignment tasks by introducing the token similarity check. The potentially "complicated" design is to improve the search efficiency.

Q3. The authors use an umbrella "harmless" notation (for which the self-evaluation score is based on), whereas in practice whether something counts as harm can be contextual.

We chose the umbrella term "harmless" to maximize the generalizability of our method. Our experiments demonstrate that utilizing a generalized "harmless" notation for self-evaluation allows the model to identify different types of harmful content effectively, thus enhancing RAIN’s universal applicability in assessing harmlessness. We also recognize that adopting more specific terms, such as "privacy," could potentially yield better results in assessing specific types of harm. We appreciate your comment and agree that harm must be considered in context. In fact, the self-evaluation in RAIN and its performance evaluation (whether using GPT-4 or human evaluation) take context into account. The <Generated Text> in the prompt examples on page 5 actually include context. Replacing <Generated Text> with <Contextual Generated Text> or <Query and Generated Text> would provide greater clarity. Your suggestion is highly valuable.

Thank you for your acknowledgement! Your thorough reading and insightful comments have been tremendously helpful to us. This response addresses all of your questions.

Q1. Algorithm 1 is not referenced in the main text, although it is presented in the methodology section. It would be helpful to mention Algorithm 1 and provide a brief explanation or reference to it where relevant.

Thank you for highlighting the oversight regarding Algorithm 1. In the revised version of our paper, we will ensure that Algorithm 1 is clearly referenced. Furthermore, in the methodology section, we will add correspondences between the text and the pseudocode of the algorithm to enhance clarity and understanding.

Q2. In the evaluation and searching process, there are several thresholds mentioned, such as $T_m$ and $V$. It would be beneficial to clarify whether these thresholds have common settings or if they are benchmark-specific.

Thank you for your insightful suggestion. In our methodology, we utilize thresholds $T_m$ and $V$ to effectively manage the early exit mechanism, aiming to balance time efficiency with performance. The calibration of these parameters depends heavily on the accuracy of the model's self-evaluation. Generally, the more accurate the self-evaluation, the lower we can set these thresholds.

In our experiments, we identified the truthfulness task as particularly challenging for self-evaluation, requiring different threshold settings. Specifically, we used $T_m=2$ and $V=0.7$ for all tasks except truthfulness. For the truthfulness task, the parameters were adjusted to $T_m = 16$ and $V = 0.8$ to accommodate its unique challenges. The upper limit $T$ for the inner loop was consistently set at 50 across all tasks.

Regarding response generation, earlier tokens in a sequence have a more pronounced influence on the overall arrangement. To address this, our node structure is designed to vary the token counts at different positions in the sequence. The structure follows a pattern of 1, 2, 4, and 10 tokens, with subsequent nodes containing 10 tokens each.

Q3. In the inner loop, could alternative LLMs be utilized for conducting evaluation?

Using alternative LLMs for evaluation in the inner loop is an interesting idea. It could reduce biases, especially if the evaluation LLMs differ from the generation model. It is important to note, though, that if the models are similar, the extent of bias reduction might be limited. Additionally, this approach increases memory overhead due to the need to store both generation and evaluation models.

Q4. What are your thoughts on the potential benefits of employing multi-agent debating for self-evaluation?

The idea of employing multi-agent debating for self-evaluation is indeed promising. It has the potential to facilitate more accurate assessments, potentially leading to the generation of higher-quality responses. Multi-agent debating is computationally more intensive. This increased computational demand suggests that its most effective application might be in scenarios where immediate response times are not critical, allowing a greater focus on maximizing output quality.

Q5. RAIN employs the embedding of pre-trained Sentence-BERT for calculating cosine similarity. How do you perceive the possibility of replacing it with the embedding from the LLM itself?

Thank you for your comment. This method has the advantage of not requiring an additional model or extra computation during feature extraction. It is noted that the significant size difference between Sentence-BERT and LLMs results in a minimal difference in computational cost. We conducted experiments under the settings of our ablation study for this aspect. In our experiments, we utilized the last token's feature from the last layer's hidden state of the LLM, as it encapsulates the context of the entire sentence, following the standard approach for sentence classification with LLMs. Our experimental results are as follows:

The results show that while using LLMs for embedding extraction outperforms omitting embedding-related mechanisms (Dynamic Node Addition & Similarity Update), it is less effective than utilizing Sentence-BERT. This is likely due to Sentence-BERT being specifically trained for sentence embeddings, thereby providing higher-quality embeddings compared to LLM features, which are primarily designed for next-token prediction.

We greatly appreciate your insightful comments and constructive criticism. They are invaluable in guiding us to refine our manuscript. Here are our detailed responses to your points and further explanations to address the concerns raised.

Q1. Best-of-K/rejection sampling method is not listed as baseline in this work.

In fact, we have conducted a comparison with the method you mentioned. This is detailed in Table 5 on page 9 of our paper, where the best-of-K method is referred to as "Vanilla (Repeated)." Best-of-K is a specific case of rejection sampling, and if the evaluation is effective, best-of-K should not perform worse than rejection sampling. The experimental results demonstrate that even with 500 trials (best-of-500), for the 7B model, best-of-K barely improves the model’s safety. With the 30B model, RAIN shows a significantly enhanced performance. This clearly indicates the superior efficacy of RAIN compared to the best-of-K/rejection sampling approach.

Q2. Even though other related works have not been applied to the specific task set used by the author, they still have the potential to be strong competitors.

For [1] (RLAIF), we have included comparative results in Figure 4 on page 8. The method described in [6] is applied to models trained with RLHF, which diverges from our focus on non-finetuned models. To the best of our knowledge, for the methods in [3,4,5], there is no clear evidence suggesting their applicability in the domain of LLM safety. The method in [5] necessitates interaction with the environment to gather feedback, which, in our research context, would mean requiring real user feedback. This is impractical in the field of safety and makes a comparison with RAIN unfair. The problems addressed by RAIN and those tackled by the aforementioned studies are fundamentally different. RAIN's external form is consistent with that of auto-regressive generation, which makes it adaptable to a wide range of language generation tasks. The studies in [3,4] are primarily concerned with reasoning tasks, which inherently allow for explicit planning. For example, [3] investigates tasks like the 24-point game, which involves multi-step number combinations, while [4] focuses on logical reasoning, which can be broken down into multi-step selection and deduction processes. Such explicit planning structures are absent in tasks like harmless generation, leading to different search spaces. The search space in [3,4] revolves around actions like combining numbers, whereas RAIN's search space is broader, encompassing the entire language space.

Despite various challenges, we made our utmost effort to adapt LEAP proposed in [4] for the task of harmless generation and subsequently conducted experiments on the HH dataset. Here are the experimental details and results. We replaced the actions in LEAP with the selection of token sets. LEAP uses $u=\log p_{\text {sel }}(\mathbf{s})+\alpha \Delta u$ and $\Delta u = \max_d \log p_{\text{ver}}\left(x_0 \mid y_d\right)$ to determine the score, where $p_{\text {sel}}$ is given by the selection model and $p_{\text{ver}}$ by a different verification model. As we did not train a verification model, we set $p_{\text {sel}}$ to be the probability obtained from querying LLMs and $p_{\text{ver}}$ as the normalized self-evaluation score, ensuring the meanings of both terms remain unchanged. The experiment's hyper-parameters were consistent with the original paper of [4], such as $B=5$, $\alpha=10$, etc. The results are as follows:

Our findings indicate that while LEAP improves harmlessness over vanilla auto-regressive generation, RAIN significantly outperforms both methods.

Q3. Unclear experiment details. What are the maximum number of iterations $T$ and the minimum number of iterations $T_m$ used in the experiment? How many tokens are generated for each node?

Thank you for pointing it out. In RAIN, hyper-parameters $T_m$ and $V$ control the early exit strategy, balancing time cost against performance. These parameters are tuned based on the accuracy of self-evaluation: the more precise the self-evaluation, the lower we can afford to set $T_m$ and $V$. For all tasks except truthfulness, we set $T_m=2$ and $V=0.7$, whereas for truthfulness, due to its increased complexity, we used $T_m=16$ and $V=0.8$. Across all tasks, the upper limit of the inner loop iterations, represented by $T$, was fixed at 50. When generating responses, tokens positioned earlier in the sequence have a greater influence on the entire token sequence. Therefore, we set a varying number of tokens for nodes at different positions, specifically in the sequence of 1, 2, 4, 10 (with all subsequent nodes containing 10 tokens each).

Thanks authors for clarifications. I have further questions after reading your response:

1. Regarding best-of-K sampling as a baseline:

   I see that I misunderstood your setup of making "Vanilla (Repeated)" as best-of-K sampling. I strongly suggest you have a section of baseline in your formal version explaining all baselines you plan to use for clarification, explaining what baselines you are implementing, how you set up hyperparameters for them, and how you adapt them to your experimental setting if necessary, etc. Currently, there is no clear explanation for it to be regarded as a best-of-K sampling baseline. However, it adds more confusion as to why this can be a best-of-k sampling implementation. First, there is no explanation of your sampling parameters (the temperature, the top_p and top_k) -- this is very important as we need to select from a diverse set of generations. Second, why do you think using LLM itself as an evaluator is a valid best-of-k sampling (perhaps you can mention some work to support your decision, but I do not see any such support in the current version)? Typically, the best-of-K sampling method is implemented by using another "preference model" to pick up the best one as the final answer ([1], Section 2.3). If you use the same model to select its outputs, there might be strong biases put on the maximum likelihood one, making the output not significantly different from "Vanilla" (as evidenced by LLaMA-7B results in Table 5). Considering there are many open-source preference models already released online, I think a valid best-of-K sampling baseline should be at least using one of these models (e.g., OpenAssistant Reward Model: https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large).

   In short, if you want to justify Vanilla (Repeated) as a valid best-of-K baseline, more details and perhaps some more experiments need to be conducted.

2. Regarding Comparison to Previous Works

   Thanks for the detailed explanation. I again strongly suggest you add a more detailed section on baselines and include your discussions here. I appreciate your efforts in replicating some of my mentioned previous works and it is great to see your proposed method can give such significant performance boosting.

3. Regarding Missing Experiment Details

   Thanks for the explanations. Please add these parameter specifications in your paper as well as your justification on setting them.

References:

[1] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).

I appreciate the detailed responses from the author and it does clear many of my confusions. However, the limited application range (due to computational overhead), unclear explanation of baseline settings, etc. are still my concerns. Based on these considerations, I would raise my score to 6.

Q4-Q6: Thanks for the explanation. Please re-read your current formulation and update it appropriately.

Q7: I agree the detailed running time strongly depends on your experimental setup and can vary by different computational infrastructures underlying your experiments. Thanks for providing these efficiency numbers. I think it is a good illustration of the points I explained in "limited applications": the proposed decoding method is not easy to parallelize, thus the computational overhead seems to overwhelm the performance benefit it can bring. Also, it does not seem to be easy for practitioners to incorporate the proposed method in decoding time from an engineering perspective.

Q8: Thanks for proposing a potential solution to achieve a better efficiency-performance trade-off, although it perhaps makes your method closer to best-of-K sampling/rejection sampling as proposed in [1]. As I explained in my responses (part.1), there are already many preference model/reward models available online. You do not necessarily need to collect your data to train a new reward model. Also, there are already many human preference data online and it seems straightforward to train your own reward model if you are not satisfied with the open-sourced reward model. With that said, I want to clarify that this question is more about a discussion for future extension, and does not constitute a criticism (that is why I only listed it in the "Questions" section).

Additional Notes to facilitate discussions: Except for reducing the number of iterations to achieve better efficiency, I originally thought of not doing this tree search for a proportion of tokens generated and using vanilla auto-regressive generation instead. Using the human utterance generation process as an analogy, we do not need to consider that carefully for every token when the context is clear (you also observe that not all tasks require the same (large) number of iterations, and $T=0$ is just a special case). Not sure whether the author may find it a useful and interesting idea to explore. Just write it in case the author may find it useful.

I really appreciate the author's timely follow-up experiments, explanations, and insightful discussions. Please make sure to add them to the upcoming version. My main concerns now are only on computational costs (as other reviewers also point out) and moreover, I am not fully convinced yet that we need to develop such complicated methods to verify that the model can do (self)-alignment -- the question that confused me most is that, if we really need a complicated method design to make the model to do self-alignment, is it really "self"-alignment? I am worried maybe we are risking treating some other emergent capabilities as "self"-alignment.

Also here is another conceptual question (open to discussions with the author, other reviewers, chairs, and public reviewers are welcome): Let's ignore the methodology part for a while, if we can observe the model can do self-alignment, then why do we need to do "alignment"? By definition, self-alignment already means the model should be able to align itself really well, and should not need help doing alignment. This is not a criticism, instead, it is more of a research question that is hanging in my mind, and curious to hear people's opinions.

I realized that the discussion phase is over and considering my timeline, I am no longer for further discussions. Thanks to the author for the hard work on discussing with me and providing follow-ups. I tend to keep my scores unchanged.

Thank you for your detailed review and valuable feedback, which will greatly assist us in improving our paper. Below are our responses to your queries and clarifications regarding some of the weaknesses you have pointed out.

Q1. How do naïve baselines (using a pre-trained binary classifier in place of self-evaluation, or using the current prompt without the inner loop but in the context of naive generate and evaluate) compare in term of running time and performance?

We have compared RAIN with the naïve baseline, labeled as "Vanilla (Repeated)", in Table 5 on page 9 of the original manuscript. In this experiment, the naïve baseline involves conducting 500 sampling iterations, followed by selecting the optimal generation through self-evaluation. For the 7B model, the naïve baseline shows minimal improvement in the model's safety. For the 30B model, RAIN significantly outperforms the naïve baseline in terms of both efficiency and performance. This comparison illustrates that RAIN is markedly superior in enhancing model performance. A binary classifier designed to evaluate whether a text is harmful must be trained using data related to harmlessness. This is akin to training the reward model in RLHF, leading to unfair comparisons (RAIN does not utilize additional data). To ensure a fair comparison, our experimental approach is in line with your suggestion, namely "using the current prompt without the inner loop in the context of naïve generate and evaluate."

Q2. While I understand that this may already common practice, having almost all of the production of your model evaluated by pre-existing model - which, furthermore, are not even public (nor is their training data) is something I believe to be an issue and should be at least discussed in the paper.

To address this potential limitation, we have not only utilized evaluations from pre-existing models but have also conducted human evaluations to validate our findings. The results of these human evaluations are presented in Table 8 on page 9 of our original manuscript. These results demonstrate a close alignment between evaluations conducted by GPT-4 and those by human assessors, indicating consistency in the assessment quality. Additionally, recent research [1] supports the reliability of GPT-4 evaluations, further validating our approach. In our experiments conducted on AdvBench [2], we used the same metrics as those in the original paper that introduced this benchmark. These metrics are not determined by model judgment but are based on explicit, predefined rules.

Q3. The method completely relies on a self-evaluation measure, in a form of a prompt. This dimension of the method, which to me seems like is the most capital, stays completely uninvestigated. Have other prompts been tested? With a different keyword (harmful/harmless)?

Thank you for your suggestion. We acknowledge the importance of this aspect and agree that it could enrich our ablation study. Our original prompt already incorporates both "harmful" and "harmless" as key terms, as demonstrated in the prompt example on page 5 of our paper. To further investigate the sensitivity to variations in prompts, we conducted an additional experiment using the following alternative prompt: "This is a conversation between a human and an AI assistant. The assistant's response should comply with ethical guidelines, comply with the law, and avoid providing information that could cause harm or adverse effects. Do you think the assistant's response is harmful?" The results of this experiment are as follows:

Our experimental findings indicate that the utilization of different prompts with similar thematic content does not significantly alter the results, suggesting that the system’s performance is robust to prompt variations.

References:

[1] Pan, Alexander, et al. "Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark." International Conference on Machine Learning. PMLR, 2023.

[2] Zou, Andy, et al. "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv preprint arXiv:2307.15043 (2023).

Q4. While I can see its interest, I believe the robustness experiment should be better introduced and motivated in the paper.

The significance of robustness in LLM alignment is highlighted by a study [1] introducing the Greedy Coordinate Gradient (GCG). This study reveals that even well-aligned models, including ChatGPT, are vulnerable to harmful responses under GCG attacks, underscoring the importance of ensuring safety against adversarial attacks.

Q5. Similarly, truthfulness can obviously be linked to harmlessness - still, I would have liked this to be discussed to better motivate the experiment.

LLMs sometimes generate responses that are not truthful, which can potentially mislead users and lead to serious consequences. In fields like healthcare, accuracy in factual information is extremely sensitive. When applying LLMs to these areas, special attention must be paid to the truthfulness of their outputs. Truthfulness is also a current focus in LLM alignment research, with studies such as LLaMA2-chat [2] and InstructGPT [3] both investigating and testing for truthfulness.

Q6. Lastly, I do not completely understand the relevance of controlled generation.

While not directly related to safety, exploring controlled sentiment generation allows us to test RAIN's versatility across a wider range of tasks. Recent works in LLM alignment, such as DPO [4] and RAFT [5], have also experimented in this task, indicating its relevance to the field.

Q7. A large part of your citations are preprints - if those papers have been published, please update their reference.

Thank you for your valuable suggestion. We have thoroughly reviewed all our references and identified four papers that have since been published. These publications are:

• Cho, Woon Sang, et al. "Towards Coherent and Cohesive Long-form Text Generation." Proceedings of the First Workshop on Narrative Understanding. 2019.

• Kreutzer, Julia, et al. "Can Neural Machine Translation be Improved with User Feedback?." Proceedings of NAACL-HLT. 2018.

• Lin, Stephanie, Jacob Hilton, and Owain Evans. "TruthfulQA: Measuring How Models Mimic Human Falsehoods." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

• Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We are committed to updating these citations in the forthcoming version of our manuscript.

Q8. I believe Figure 2 should be capital to the understanding of your method - however, the text helped me understanding the figure. I believe you should try to simplify what you display for the first contact of the reader with your method.

Thank you for your valuable feedback. We provided a simplified diagram, which abstracts away the internal details and utilizes a concrete example for clarity, in the “Supplementary Material” section of our OpenReview submission, within the .zip file, under the file name "Simplified_Diagram.pdf". The caption for this diagram is as follows: "A simplified explanation of RAIN. RAIN switches between a forward generation phase and a backward rewinding phase, incorporating a self-evaluation stage in between to accomplish self-alignment. The method mirrors human behavioral patterns: contemplating, weighing, and reflecting on the consequences before speaking. Notably, the method operates without the need for extra data and abstains from model updates."

References:

[1] Zou, Andy, et al. "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv preprint arXiv:2307.15043 (2023).

[2] Touvron, Hugo, et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv preprint arXiv:2307.09288 (2023).

[3] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

[4] Rafailov, Rafael, et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv preprint arXiv:2305.18290 (2023).

[5] Dong, Hanze, et al. "RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment." arXiv preprint arXiv:2304.06767 (2023).