Memorisable Prompting: Preventing LLMs Forgetting False Positive Alarm

• The article is difficult to follow due to inconsistent terminology, unclear mathematical notation, grammatical errors, structural issues, and logical gaps, which detract from its clarity and coherence. Overall, the writing does not reflect careful attention to readability and precision.

• The proposed approach appears limited in applicability, focusing primarily on classification tasks. However, this limitation is not addressed in the paper, leading to an incomplete understanding of the method's scope.

• The narrative includes redundant explanations and lacks depth, limiting the reader’s engagement with the insights behind the proposed approach.

• Baseline comparisons are insufficient; it remains unclear why traditional weak supervision methods, such as [1], combined with pre-trained language models were not included for a more comprehensive evaluation of the method.

• The proposed approach shows minimal innovation compared to prior work using transition matrices for prediction calibration, which reduces the originality and significance of the contribution.

[1] Ren, Wendi, et al. "Denoising multi-source weak supervision for neural text classification." arXiv preprint arXiv:2010.04582 (2020).

The entire paper is more of a technical report in that it improves upon the prompt strategy; however, it is difficult to read the entire paper. Even after reading the whole paper, I don't even know what kind of prompt form or few-shot form the authors employ.

The motivation of this paper is explicitly to make the LLMs remember the dependencies of responses and prevent making false positive answers. However, there are many fundamental errors in both model construction and experimental setups.

In method construction:

1. The paper is very redundant to introduce the basic definition, and there is a lack of formal definition of basic memorization matrix in preliminaries.

2. The introduction of the method is not clear, I think only two points need to be introduced clearly:

2.1. How is the memorization matrix constructed between the answer and the truth to each query when the truth cannot be obtained during reasoning? How many times does each query need to be answered, and how is the sparsity of the matrix addressed when there are too many categories?

2.2. Does the LLM actually capture dependencies between responses from the matrix? How does the LLM remember and avoid false positive responses? Are there any experiments to prove it?

3. In Section 4.0, the paper introduces the learning conditions, it is very confusing how the sample features are introduced in the method, for example in Eq.3, how to find completely different features that can make the LLM generate the same prediction? I don't think any specific distinct feature alone can do that.

4. Is it using the memorization matrix of few-shot hint samples to predict samples with no truth labels? What is the number of hint samples for each sample? How to determine the LLM is not mimicking the memory matrix, but actually recording the response to each query.

In related work: The paper does not fully investigate the related work and has no logic in expression. The paper is based on LLM reasoning and constructs a strategy to improve prompt.

In the experiments:

1. The paper is lack of experimental setup, and prompts should be introduced in detail, otherwise it is difficult to reproduce the work.

2. There is too little experimental analysis in the paper, and the experimental Tables and Figures are not clearly introduced. Besides, the paper adopts three LLMs. Is there no difference in performance among the three LLMs? Meanwhile, the paper lacks interpretability experiments to validate the motivation.

3. References to Figures1, 2 and 3 and Tables 1 and 2 are absent from the main paper. There are also several methods that are not detailed in baselines. What is used for correction? What is fot in the Tables? I think the paper should at least be standardized and clear.

There are multiple errors in the paper, multiple quotes, punctuation, basic grammar errors, the readability of the paper is poor, I think the author should take the paper submission seriously.

The justification for the Memorization masking assumption is confusing (line 225-234). Please lay out the specific constraints assumed made by this claim, and deduce the final assumption. If I understand correctly, this assumption is claiming that assuming if the model has learned the characteristics of a category well enough, then it should reliably predict the same Y' distribution for each Y regardless of the input feature X. The current justification is not clearly or rigorously written and need further editing.

Section 4 is hard to follow and not rigorously written. X and Y are sometimes referred to as variables and some other times as values/examples. For example, equation (4) does not "simplify" to equation (5) when the given assumption (ln. 258) is Y=1 for a given X=x1; this is only correct if the given assumption is X=x1 and Y=1.

Section 4.3. It is not clear how M is used in the transformation from $\vec{Y}$_Query to $\vec{Y}$_True. Additionally, "Our goal is to design a prompting scheme to enable LLMs to generate the correct annotation for each q from the corresponding $\vec{y}$" (line 303-304). What exactly is the prompting scheme? It is not discussed or included in the appendix.

Overall, there are fundamental flaws in the probabilistic framework and problem formulation. The core mathematical error lies in the proposed marginalization over Y: $P(Y'|X,\vec{Y}) = \sum_Y P(Y'|Y,X,\vec{Y})P(Y|X,\vec{Y})$. This equation is fundamentally incorrect as Y represents a ground truth label, not a random variable. The main objective to "obtain" $P(Y|X,\vec{Y})$ through $P(Y|Y',X,\vec{Y})$ shows a fundamental misunderstanding of the causal relationship in supervised learning. THe prediction Y' should not influence the probability of the true label Y. The learning objective should instead be formulated as maximizing the probability of correct classification, $\arg\max_G P(Y=y^*|X,Y)$. These issues reflect a fundamental misunderstanding of probabilistic modeling, making the proposed method mathematically unsound. While the research direction may be promising, the current formulation requires substantial revision to establish a valid theoretical foundation.