We would like to sincerely thank the reviewer j9mp for their thorough and highly constructive feedback. We will address questions here in detail one by one.

• On whether the "hidden stack is really a stack" and if "the learned grammar can be inspected." This is a very central question, and we will respond from two perspectives:
  • On the name and the global read mechanism: We call it a "stack" based on its core update mechanism. The information flow is dynamically managed through differentiable push and pop operations. The "global read" you observed is a reading mechanism we introduced to solve a key problem in practical training. As described in Appendix §B.4, a traditional "stack-top only" read can lead to restricted gradient flow and unstable training. Therefore, our design can be seen as: a module with stack-like update logic, supplemented by a more flexible global read mechanism to optimize the learning process.
  • On whether we can show that it captures grammar: Yes, we can.
    • Quantitative Evidence: In the formal language tasks in §4, StackTrans achieved good performance on tasks requiring precise grammar. For example, it scored 1.00 on the "Reverse String" task, whereas the standard Transformer only scored 0.55. This directly demonstrates that it has learned the underlying grammatical rules.
    • Qualitative Evidence: In Appendix §B.2 (Figure 5), we visualized the stack's operational patterns. The results clearly show that the model automatically learns meaningful, task-aligned patterns: it tends to push (store information) in the shallower layers of the network and pop (retrieve information) in the deeper layers. This interpretable behavior could be seen as the evidence that it has learned the underlying "grammar" of the task.
• On model details, compatibility with variants, and comparison with LoRA: One of the core advantages of our design is its modularity and non-intrusive nature. As stated in the paper, StackTrans "integrates a differentiable stack between Transformer layers while maintaining the integrity of the Transformer layers" and "decouples the stack operations from the attention computation."This design allows it to be naturally combined with various Transformer backbones. Therefore, we have strong reason to believe that it can also be combined with parameter-efficient fine-tuning (PEFT) methods like LoRA (e.g., by freezing the backbone LLM and fine-tuning only the StackTrans modules). This is a very promising direction for future research.
• On providing results for different model sizes and their significance: The reviewer may have overlooked the relevant sections, as we have provided detailed results in the paper:
  • Scaling Law Study: In §5, we systematically studied the model's scaling laws, clearly analyzing the impact of both model size and the amount of training data.
  • Performance Curves for Different Sizes: Figure 3(a) visually presents the loss curves as the model size increases from 360M to 7B, clearly showing that larger model sizes lead to better performance and outperform the standard Transformer.
  • Evaluation of a Specific Model Size: All comparisons in Table 2 are centered around our StackTrans-360M model, which is compared against models with larger parameter counts (such as the 1.1B OLMo and TinyLLaMA). The results are significant.
• On CFG bias and performance on general-purpose natural language tasks: Your question—"whether introducing a CFG bias would harm general-purpose performance"—is a core question that all methods with structured inductive biases must answer. We address this challenge head-on by evaluating our model on a wide range of general-purpose tasks.
  • It did not harm performance; it improved it. As shown in Table 2, we tested on 9 diverse general-purpose benchmarks, including commonsense reasoning (e.g. HellaSwag), question answering (e.g. OpenBookQA), and mathematical reasoning (e.g. GSM8K).
  • The results are significant. The outcomes show that not only did StackTrans not suffer from performance degradation, but it actually achieved significant performance gains. Our 360M model's average score (44.3) far surpasses baseline models that are 2-3 times its size in parameter count. This proves that our stack mechanism acts as an effective supplementary module that enhances the model's structured reasoning capabilities without sacrificing its general language abilities. While GLUE is a good benchmark, we believe the existing 9 benchmarks (for example, MMLU is a popular benchmark that is adopted by many pretraining LLM studies) already provide strong evidence of the model's generality.

Thank you for the positive feedback and for your decision to raise your rating. We are delighted that our responses have resolved your concerns and appreciate the opportunity to clarify points on the model configuration.

You are correct -- the total stack dimensionality (H * d_s) is not required to be equal to the Transformer's hidden size (d). This is a deliberate and central part of our model's architecture.

This is due to our low-rank design. Our standard StackTrans model intentionally uses a bottleneck where the total stack dimension is smaller than the model's hidden size (i.e., $H \cdot d_s < d$). This is achieved by first down-projecting the input hidden state before it is processed by the parallel stacks. The configuration where $H \cdot d_s = d$ is a special case we refer to as the "Full-Dimension" variant, which we tested in our ablation studies (Table 3). Those results confirmed that the full-dimension approach was significantly more computationally expensive without a corresponding performance benefit, validating our choice of the more efficient low-rank design.

To further prevent any misunderstanding, we wish to highlight the two distinct experimental configurations used in our paper:

• Section 4 (Formal Language Evaluation): For these more theoretical tasks, we used a smaller model configuration to ensure a fair comparison with prior work. This model uses five Transformer layers with a hidden size of d=64, 4 stack heads, and a stack dimension of d_s=8, as shown in Line 226 in our paper.
• Section 5 & Appendix F: For the general natural language evaluation, we used our large-scale models, with parameters ranging from 360M to 7B. The STACKTRANS-360M model, for instance, has 32 layers and a hidden size of 960, as detailed in Table 6.

Thank you again for giving us the opportunity to clarify this important design detail. We will be sure to emphasize this distinction in our revised paper for the benefit of all readers.

We would like to sincerely thank the reviewer nPse for their thorough and highly constructive feedback. We will address each of your points below.

• On the term "Stack" and "Global Read": Thank you for this insightful conceptual point. We use the term "stack" because the core update mechanism of our module is fundamentally based on differentiable push and pop operations. Regarding why reading only the top element is suboptimal, you are right that our explanation in the main text was too brief. As detailed in Appendix B.4, our initial experiments showed that a strict top-only access leads to "unstable and suboptimal model performance" because it "disrupts gradient flow and reduces parameter learning efficiency". In the revised paper, we will clarify this distinction between the stack-like update mechanism and the global read mechanism. We will also move a more detailed justification for the "global read" from the appendix to the main text to make the rationale clearer.
• On the "Bottom-to-Top" Design and its Motivation: We will revise the manuscript to introduce this concept much earlier, including in the abstract and introduction, to properly frame its significance as a core contribution. Regarding its effectiveness for context-free languages, our empirical results in Table 1 provide strong validation. StackTrans achieves scores of 0.92 and 1.00 on the DCF tasks "Stack Manipulation" and "Reverse String," respectively, whereas the standard Transformer fails with scores of 0.53 and 0.55. This demonstrates a clear and significant advantage in modeling these dependencies, directly validating our design choice.
• On Stack Size S vs. Number of Layers L: This is a very keen observation. You are correct that in our current implementation, which is designed for training parallelism, the stack processes hidden states layer-wise for each token. This means the effective sequence length fed to the stack at any given time is the number of model layers, L. We will add a note to the text clarifying this relationship between S and L in our specific implementation. This is also empirically supported by our hyperparameter experiment in Figure 4(c).
• On Table 1 Results: Thank you for pointing out this ambiguity. In the revision, we will add the citations directly into the table caption (which are already listed in Line 228 in the paper). We will also fix the citation typo in the footnote (the results are reported by [1]). Regarding the baseline in DuSell and Chiang [2024], in our initial experiments, we selected stack-augmented baselines, closely following the setup in [2] to ensure our results were solid. We are now working on the experiments for the baseline you mentioned. Preliminary results already show that our method, StackTrans, outperforms it by a significant margin. The detailed experimental setup and results will be provided in the revised paper.

[1] Jiaoda Li, Jennifer C White, Mrinmaya Sachan, and Ryan Cotterell. A transformer with stack attention. 2024.

[2] Delétang G, Ruoss A, Grau-Moya J, et al. Neural networks and the chomsky hierarchy. 2022.

• For descriptions for Chomsky hierarchy, We are extremely grateful for these critical corrections. We acknowledge that we were misusing some terms. We will perform a thorough revision of the entire manuscript to fix all instances of these errors. We will replace incorrect uses of "Chomsky hierarchy" with more precise phrasing (e.g., "modeling dependencies found in specific classes of formal languages"). We will also ensure the distinction between context-free and deterministic context-free languages is handled correctly. This will significantly improve the paper's formal precision.
• We will also revise those lines to address all points in "other comments".

For questions:

• Comparison with some baseline: As noted above, we will fix the citations in the footnote in Table 1 to properly contextualize our work among other stack-augmented models discussed in the literature. The detailed experimental setup and results will be provided in the revised paper.
• Layers in Figure 4: The experiments in Figure 4 were conducted using our StackTrans-360M model configuration. We will clarify in the caption that these hyperparameter settings used the 360M model, which has 32 layers.

Thank you for your insightful comments and suggestions.

Regarding the "bottom-to-top" stack design, your feedback has been invaluable in helping us refine its positioning. We will take your strong recommendation to heart and revise the key parts of our paper accordingly.

Furthermore, your excellent suggestion to probe the model's performance on longer sequences prompted us to conduct a deeper analysis, which has yielded strong confirmatory results.

For instance, in our formal language experiments, StackTrans maintains excellent performance, particularly on RE and DCF tasks, even when the evaluation sequence length significantly exceeds the model's depth.

This confirms our hypothesis that the model's overall generalization ability is not strictly limited by its depth. A powerful synergy exists: the self-attention mechanism inherently captures cross-token relationships, which can partially compensate for the truncation. It is indeed a balanced trade-off: it enables efficient, parallel training without sacrificing the model's crucial ability to generalize. We will incorporate a more detailed analysis of these length generalization experiments into the revised paper.

Thank you again for your diligence and for helping us improve the scientific accuracy and clarity of our paper.

We sincerely thank Reviewer FFsc for their review and valuable suggestions for revision.

• On the specific mapping of information stored in the stack: We thank the reviewer for pointing this out. The specific mapping is described in §3.2: The hidden state h_t \in \mathbb{R}^d from a Transformer layer is first passed through a down-projection matrix W_{down}, then reshaped and split into H independent heads, i.e., [h_{t}^{(1)}, ..., h_{t}^{(H)}] = \text{Reshape}(W_{down} \cdot h_t). Each h_t^{(i)} is then fed into its corresponding independent stack for operations. We will clarify this textual description in the revised version and update the caption for Figure 2 to more clearly illustrate this mapping from h_t to the multi-head inputs.
• On the "near 100% accuracy" claim: Your analysis of Figure 1(b) is correct, and our statement was indeed too broad. We originally intended to highlight that the model achieved 100% accuracy on all RE (Regular Expression) tasks and also demonstrated strong performance on DCF tasks. Out of the 13 formal language tasks, StackTrans did achieve 100% on 5 of them. We will revise the wording to be more precise: "achieved 100% accuracy on all regular language tasks and demonstrated excellent performance on the majority of deterministic context-free grammar tasks," to accurately reflect our experimental results.
• On the standardization of mathematical symbols: Thank you for the suggestion. We fully agree and will uniformly use the more formal \mathbb{R}^d to denote vector spaces in the revised version.
• On using tuple notation to simplify formulas: This is an excellent suggestion. Adopting the tuple notation (S_{t+1}[i], M_{t+1}[i]) will indeed make the formulas more compact and simplify the main text. We will adopt this suggestion in the revision and use the saved space to better elaborate on other key details.
• On zero vectors and the mask: This is a profound question. In our design, the mask M_t \in \mathbb{R}^S is a vector of probabilistic values indicating the "activeness" of each position in the stack. The stack itself, St_t, stores the actual hidden state vectors. The final effective stack is obtained through an element-wise product St_t \otimes M_t. Therefore, both are necessary: M_t controls which positions are valid in a differentiable manner, while the zero vectors in St_t are the actual padding values for invalid positions. While the idea of using only a mask is interesting, our current design ensures the entire process is fully differentiable. We will clarify this point in the revision.
• On the definition of the Reshape function: We apologize for the lack of clarity. Reshape is a standard operation in deep learning that changes the shape of a tensor. In this context, it reshapes the down-projected vector (with dimension H \cdot d_s) into H independent vectors of dimension d_s for the multi-head stacks. We will add this explanation in the revised paper.
• On the hyperparameter \lambda: Thank you for the reminder. We will add the value of \lambda used in our experiments. As a regularization term, we give \lambda a small weight in experiments, e.g., 0.001.
• On the visibility of Figure 3(c): We agree that the green curve in Figure 3(c) is difficult to discern. In the revised version, we will modify the colors and styles of the chart to enhance its clarity and readability.
• Grammar correction: Thank you for pointing out the typo. We will change "comes" to "come" in the revision.
• On the details of "data filtering": On line 257, we mentioned that our corpus comes from Dolma and Smoll. The data filtering followed standard LLM pre-training procedures, including deduplication, removal of low-quality text, and filtering of harmful content. We will add a sentence to briefly explain this in the revision.
• On the computational complexity of the ablation study: This is a great suggestion. In fact, we have already compared the training and inference efficiency for some ablation variants (single-head and full-dimension) in Table 4 of the appendix. The results show that our multi-head, low-rank design achieves the best performance with a minimal increase in overhead. We will more explicitly reference and discuss these results in the main text in §6.
• On the accuracy in the ablation study: In our ablation study (Table 3), we primarily used validation perplexity (PPL) as the core evaluation metric. This is the gold standard for assessing generalization ability in language model research and is highly correlated with downstream task performance. While we did not evaluate every ablation variant on all downstream tasks, the significant differences in PPL already provide strong evidence for the effectiveness of our design.
• On the size S for QueueTrans: Your point is well-taken; the optimal size for a queue might differ from that of a stack. In our ablation study, to ensure a fair comparison by controlling variables, we used the same stack/queue size S for all variants. Exploring the optimal hyperparameters for different data structures is a valuable direction for future work, which we will consider.
• On the discussion of single-head/full-dimension ablations: Table 3 and Table 4 show that the single-head stack is more efficient but performs slightly worse, whereas the full-dimension stack is extremely costly for limited performance gain. This precisely demonstrates that our multi-head, low-rank design is the optimal trade-off, achieving most of the performance gains at a cost close to the baseline Transformer. We will discuss this cost-performance advantage more deeply in §6.
• On the trend and training length in Figure 4: This is actually because, with a fixed total parameter count, having too many heads leads to a lower dimension per head, which weakens the model's expressive power. All experiments in Figure 4 used the same training dataset setting to ensure a fair comparison. We will clarify this.

Please excuse the late response. Thank you very much for the detailed rebuttal. Your comments address and clarify all my questions. I welcome your mentioned updates as that will add value to the publications. I'll keep the already high overall rating as is.

We sincerely thank the reviewer 1yQj for their high praise and strong support for our work. We are delighted that you found our paper to be "well-written, clear, and easy to follow," and that you recognized the value of our work and the thoroughness of our experiments.

We agree with your point that validating the advantages of StackTrans on larger-scale models (e.g., 70B+) would be a crucial and exciting next step. As you noted, our current experiments have successfully scaled the model from 360M to 7B parameters, and in this process, we observed consistent and stable performance gains and scaling trends. This provides strong preliminary evidence for our architecture's continued effectiveness on even larger models. Our decision to focus the main natural language evaluations on the 360M model while demonstrating scalability up to 7B was primarily due to computational resource constraints, a point we also mentioned in the limitations section. Our work lays a solid foundation for this new architecture and provides compelling evidence of its potential. We hope this will inspire the open-source community and institutions with greater resources to pursue larger-scale validation of this architecture in the future.