Number Cookbook: Number Understanding of Language Models and How to Improve It

5. Architecture or tokenization

   We believe it’s not necessary to prioritize either architecture or tokenizer exclusively. As our article demonstrates, both factors matter: architectural choices, such as position encoding (PE), are critical, and tokenizer behavior also impacts performance. Our aim is to provide guidance on both aspects rather than favoring one at the expense of the other.

   Additionally, we suspect that limited data diversity during training contributes to the challenges observed. Notably, even simple fine-tuning, without additional techniques, significantly improves model performance.

   In summary, we are confident that technical solutions exist, whether through better tokenizer and PE selection, or by developing new, more effective schemes. Our benchmark serves as a foundation for evaluating these approaches with clarity and comprehensiveness, supporting progress in this field toward more effective solutions.

6. Rationale

   We believe it is essential to provide a clear rationale for the benchmark suite, given the diversity of number-related tasks and representations. When discussing model NUPA, the focus should be on tasks that are (1) genuinely important and common, and (2) appropriately challenging—neither too simple nor overly complex—to ensure the benchmark’s utility and relevance. Our justification aligns with these principles.

   That said, we recognize that some statements in the original version may have been unclear or detracted from the main message. In the updated version, we have streamlined the content, keeping essential scenarios where these representations are relevant and omitting unnecessary details, such as how they are introduced in mathematics. We hope these revisions improve readability.

Thank you again for your thoughtful feedback. If you have further suggestions, we are happy to consider them.

[1] Xu et al. Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution, 2024

[2] Meta, Introducing Llama 3.1: Our most capable models to date, 2024

[3] Yang et al. Qwen2 Technical Report, 2024

3. Absolute and misleading statements
   We acknowledge that some statements in the earlier version of our paper may have appeared too absolute or misleading. These have been revised for greater accuracy and moderation in the updated submission. Examples include:

   • "Any student who has completed primary and secondary education should be able to accomplish them" -> "Because these tasks are extracted from the education curricula, students who have completed the stage of education are expected to solve them."
   • "for most practical tasks involving numbers (like arithmetic or math tests), all we care about is whether the answer is right, and there is no significant difference between being almost right and being completely wrong" -> "for most practical tasks involving numbers (like arithmetic or math tests), the correctness of the answer is the most important." In addition, this statement is a concession. We recognize and emphasize the importance of other metrics in just the next sentence "But having a smoother ...".
   • We maintain that NUPAs are essential and should be expected to be learned by LLMs even when tools or code are available for assistance. As discussed earlier, relying on tools is not always an optimal solution for numerical processing. However, we have expanded and clarified our explanation in the updated version to make this point more reasonable and better aligned with practical considerations.
   • The statement about $O(n^2)$ is not related to our conclusion and we are glad to remove it. However:
     1. In context, we discuss how RF-CoT handles multiplication by following an algorithm with $O(n^2)$ complexity. While table lookups and additions may simplify this, they still break the problem into a double loop structure inherent to multiplication.
     2. Even with shortcuts like a multiplication table, the complexity remains $O(n^2)$, as reducing constant terms (e.g., $O(n/2 * n/2) = O(n^2/4)=O(n^2)$) does not change the overall complexity.

   However, we want to emphasize that the scope of tasks and their representations were chosen carefully, with reference to the Chinese primary and secondary school curricula, ensuring that the tasks are both common and representative.

4. Concerning about high-compute low-innovation approaches

• While tokenization is an important factor, its effects on number processing remain poorly understood. There is no consensus on the best tokenization strategy, as current open-source models employ diverse approaches. Our work provides a valuable reference point for future model design. This area of research is still in its infancy, with many techniques proposed but their effectiveness in practical, diverse settings largely unproven. In this case, providing a comprehensive and clear benchmark can contribute to more systematic and orderly progress in this field.
• Although our paper includes extensive experiments, future researchers need not replicate all of them. For fine-tuning a model to enhance NUPA, only the fine-tuning experiments are required—there’s no need to re-run tokenization, PE, or data format experiments. Similarly, testing a model on NUPA only requires inference on our test set, which is computationally lightweight.
• Our benchmark is computationally efficient. Training and inference focus on a small set of numbers, requiring minimal compute resources. For example, fine-tuning Llama on a single GPU takes approximately 5 hours, with inference requiring only 8 hours on the same setup. While some experiments from scratch (e.g., in Sections 3.1 and 3.2) are more demanding, even these remain manageable: 0.1B models take about 5 hours, 1B models about 20 hours, and 3B models about 40 hours on a single H800 GPU. Importantly, these more intensive experiments are not required for end users.

We sincerely thank you for your thoughtful and constructive suggestions! We have revised the paper based on your feedback and hope the updates effectively address your concerns.

1. Numeracy in LLMs is a problem

• Although it is widely recognized that LLMs struggle with numeracy, there is still an absence of detailed benchmarks to evaluate specific challenges. Questions like which numerical tasks, number ranges, and task complexities are most difficult for models remain unanswered. A clear, detailed task definition is critical, rather than relying on vague notions of commonsense.
• We believe that NUPA without relying on external tools like Python is essential for an AGI candidate. Number processing is a high-frequency task, and dependence on such tools introduces significant overhead, increases complexity, and reduces parallelism [1]. Therefore, robust intrinsic numeracy is, in our view, critical for achieving efficient performance.
• Math ability is a key focus for LLM evaluation, with most models reporting metrics like MATH or GSM8k. Numerical processing is integral to this ability, and poor performance in NUPA directly hinders overall math capability. Notably, these metrics are reported without external tools, emphasizing the importance of intrinsic numeracy [2,3].
• While it is reasonable to use external tools for particularly complex problems, such as those involving very large or intricate numbers, models should not be expected to rely on tools for every numerical task. Therefore, it is crucial to establish a reference that identifies tasks the model can handle independently with high accuracy and those that necessitate external tool support. This distinction underscores the importance and value of a comprehensive benchmark.

2. Tokenizer

• While it is commonly suggested that tokenization affects digit-level performance, its impact remains underexplored. Numbers differ fundamentally from text, as noted in the revised Section 3.1. Further research is needed to understand tokenization's role in handling numbers. As highlighted in our paper, while the trend favors larger vocabularies, one-digit tokenization performs best for NUPA. This finding is novel and has not been adequately addressed by open-source model trainers, warranting consideration for future tokenization design.
• When we express surprise at LLMs’ struggle with digits, we mean that digits are foundational to arithmetic and math. If a model cannot reliably identify digits, its ability to solve complex math problems is questionable.
• From an architectural perspective, the poor performance on digits might not be surprising. However, considering the model's expected mathematical capabilities (as many models, such as GPT-4 and Qen2.5, claim to possess strong mathematical abilities), it does seem unexpected. We have revised the paper to clarify this point, but if you still find it unclear or misleading, we’re more than happy to provide further clarification.

6. Related work
   Thank you for your suggestion. In the updated submission, we have added a related work section that now includes relevant datasets and benchmarks. To summarize, our work differs from previous benchmarks by treating NUPA as an independent task, distinct from math, language, or commonsense abilities. Additionally, we provide a comprehensive and detailed analysis of diverse numerical representations and tasks, which sets our approach apart.

7. For these minor points:

   • The abstract has been revised and the vague terms have been removed.

   • More explanations have been added in the caption of figures. Specifically, the X-axis is the seen samples, Dn is the number length (digit).

   • The citation about the calculation hallucination has been added.

   • The discussion on innateness has been removed.

   • The citation of gpt-4o has been added. The original occurrence of "chatgpt" has been removed due to the space limitation.

   • Regarding few-shot learning: In practical applications of number processing, such as financial reporting or solving arithmetic problems, we believe models should be capable of handling numbers without reliance on few-shot examples. Moreover, we view zero-shot reasoning ability as a critical direction for future model development.

     Regarding the output format, we have reviewed the models’ outputs and found no issues, as our tasks require only simple numerical outputs. We have also tested some models with the results presented in Appendix A.3.1, and the conclusions remain consistent. Thank you for your valuable feedback!

Additionally, we have thoroughly revised and polished the paper to improve its readability and clarity. We hope the updated version is now more accessible and understandable. We are happy to make further improvements if you have any additional suggestions. Thank you for your constructive feedback!

[1] Hu et al. Case-Based or Rule-Based: How Do Transformers Do the Math?, 2024.

We sincerely thank you for your constructive feedback and valuable suggestions. We have revised the paper based on your suggestions and hope the updates address your concerns.

1. Training details
   We train our models using an autoregressive Transformer architecture based on Llama-3.1 (unless stated otherwise). Sections 3.1 and 3.2 cover training from scratch with all hyperparameters, except model size, aligned with the original Llama setup. The AdamW optimizer is used with a learning rate of 5e-5, weight decay of 0.01, and batch sizes of 256, 64, and 32 for 0.1B, 0.9B, and 3B models, respectively, following default settings in the Transformers library.

   Each model is trained on a consolidated dataset, comprising $10^7$ samples for each length (where feasible). Models are trained for one epoch using a cosine decay learning rate scheduler, and the best checkpoint on validation data is reported.

   While experiments weren’t initially replicated at submission, the updated version includes three replicates for key experiments, reporting means and standard errors in figures and tables.

   Our experiments were conducted on a cluster with Nvidia H800 GPUs (80GB memory). Training a 100M model from scratch takes 5–8 hours, a 1B model about 1 day, and a 3B model approximately 2 days on a single H800 GPU. Fine-tuning a pretrained model typically requires around 5 hours.

   These details are included in the updated version (Appendix A.4.1) for improved reproducibility.

2. In-domain and Out-of-domain The terms "in-domain" and "out-of-domain" refer to the length of numbers. As described in Section 3.1, the model is trained on numbers of lengths 1 to 8 (20) and tested on numbers of lengths 1 to 20 (100). Thus, lengths 1 to 8 (20) are considered in-domain, while lengths 9 (21) to 20 (100) are out-of-domain.

3. Section 2.1 and 2.2
   The primary goal of our work is to propose a comprehensive benchmark to formalize NUPA. We believe it’s important to provide clear reasoning behind the choice of representations and tasks. To enhance clarity, we have significantly streamlined this section in the updated submission. In Section 2.1, we have retained the essential scenarios where these representations are relevant while omitting unnecessary details, such as their mathematical introductions. Likewise, Section 2.2 has been reorganized and condensed for improved coherence. We appreciate your suggestion to enhance the paper’s readability and invite you to review the updated version for further details.

4. Figure 2
   Figure 2 has been updated for better readability. Additionally, to further enhance clarity, we have provided an interactive performance report as an anonymous HTML page here. This allows readers to interact with the figure by selecting models, tasks, and metrics for a more detailed exploration.

5. Interpretation about the results
   We recognize that some explanations in our initial submission may have led to misunderstandings due to less precise wording.

• For instance, when we state that "the model does not understand the concept of digit," we mean that the model cannot consistently solve digit-related tasks. Here, "concept" refers to a "set of digit-centered abilities" as defined in Section 2.2. We clarify that our paper acknowledges models can comprehend task instructions: "models can at least comprehend the task instruction." This statement has been rephrased for clarity in the updated submission. Thank you for pointing this out.

• Regarding "case-based reasoning", we intended it as the opposite of rule-based reasoning, where the latter implies that models learn the rules of tasks to solve them consistently. (See Reference [1] for more details.) However, case-based reasoning does not contradict dependence on architecture. As noted in Section 3, architectures like RoPE might enable case-based reasoning, potentially offering shortcuts based on sequence length. Recognizing the term’s potential to mislead, we have removed it in the updated version.

• We agree that architecture plays a role and have added this to the listing. However, we maintain that the training approach is more critical. For instance, within the same architecture (e.g., LLaMA), fine-tuned versions with fewer parameters demonstrate significantly better performance than the pretrained versions, as highlighted in Section 3.

Dear Reviewer,

As the discussion period is nearing its end in two days, we would like to follow up to ensure that our revisions and responses have addressed your concerns.

In particular, we have made several updates to the paper.

1. We have added detailed information on the model and training process to provide greater clarity on our methodology. (Appendix A.4.1.)
2. Sections 2.1 and 2.2 have been streamlined to focus more on the relevance and importance of the content, removing less pertinent details.
3. We have improved the phrasing and clarity in the section on result interpretation. (Section 2.4)
4. We have added a section about related work. (Section 5)

If you have any further questions or suggestions, we would be eager to continue the discussion with you over the next few days. Your feedback is highly valued, and we are keen to ensure that the manuscript fully meets your expectations.

Thank you again for your thoughtful review.

Best regards, Authors

Thank you for your suggestion and now we have finished our experiments with few-shot learning. Specifically, we test the open-source models with 5-shot examples in the same task and find that though in most cases, few-shot learning can generally improve performance, the conclusions in our paper are still satisfied.

Our results are shown in fig1 and fig2. For example, the performance also significantly decreases as the length increases or the tasks and representations become unfamiliar (like Add-Fraction, Add-Scientific or floordiv). And the performance of digit-related tasks are still unsatisfying.

Thank you for your reminder again. We believe it is an important supplementary and we will update them in the final version.

Thank you for your insightful review. We have revised the paper based on your suggestions, and we hope the updated version and our responses below address your concerns effectively.

1. Related work
   We acknowledge that, due to space constraints, some discussions of relevant literature were omitted in the initial version. We recognize their importance and have added these discussions in a newly created section. Specifically, we have incorporated your suggested papers, along with others, into this section. Additionally, the paper [9] has been cited in the original section 3.1 and we have talked about the results. In fact, we adapt the setting of the paper right-to-left tokenization in our experiments (we also emphasize this point in updated submission). However, our findings indicate that the one-digit tokenizer performs best, while left-to-right and right-to-left tokenizers yield equivalent results.

2. tokenizer

   1. Because the tokenization is bounded to the model, it is impossible to isolate the influence of tokenization in pre-trained models without introducing other confounding factors.
   2. In the updated submission, we include a comparison of fine-tuning pre-trained models with modified tokenization in Section 3.3 (see Table 17: Row "RoPE+1d," which represents the fine-tuned Llama model with a one-digit tokenizer while keeping other settings unchanged). Similar to other fine-tuned models with modifications, the performance of the fine-tuned model with the one-digit tokenizer is worse than both the vanilla fine-tuned model and the original pre-trained model. This aligns with our conclusion: this kind of modification should be directly applied in the pretraining stage, instead of an ad-hoc during finetuning.

3. presentation

   1. The colors in Figure 2 have been updated to enhance readability. The finetuned model is added in the figure for direct comparison with other models, as we believe presenting it in a separate figure would be less optimal. We can add an explanation at the caption. To further improve clarity, we have provided an interactive performance report as an anonymous HTML page here, where the readers can interact with the figure and select the models, tasks and metrics.
   2. A brief introduction to RFFT has been added at the beginning of Section 4 in the main paper, along with a detailed explanation and an example in Appendix A.5.1. Furthermore, all prompts and code are included in the supplementary materials for reference.
   3. The paper has been reorganized to avoid the excessive use of negative vspace, improving its overall structure and readability.

Thank you once again for your detailed and thoughtful feedback.

Dear Reviewer,

As the discussion period is set to conclude in two days, we would like to kindly follow up to ensure that our responses have addressed your concerns.

Specifically, we have revised and refined the presentation further, with careful attention to adding detailed content regarding related work. The references you provided, along with other relevant paper, have now been thoroughly added into the paper.

If you have any additional questions or suggestions, we would be delighted to engage in further discussions with you over the next few days. Your feedback is invaluable to us, and we are eager to ensure that the paper meets the highest standards.

Best regards, Authors

Thank you for your valuable review. We have revise the paper according to your suggestions and we hope the updated version and the following answers can address your concerns.

1. Related work

   Thank you for pointing out these related works. A related work section has been included in the revision now as section 5, which we hope will help readers better understand our contributions and novelty. We have included two of your suggested papers in the reference list and we find the other two papers about compositionality appear to be less relevant to our study. If you have further suggestions or feedback, we would be happy to consider them.

   We are also pleased to summarize the novelty of our work as follows:

   (1) Unlike previous benchmarks, where NUPA is intertwined with math, language, or commonsense abilities, our benchmark isolates NUPA as an independent focus, allowing for a targeted and detailed analysis.

   (2) While prior studies have highlighted the diversity of numerical representations and tasks, they often lack structured organization and comprehensive analysis of these aspects. In contrast, our work meticulously categorizes and analyzes numerical representations and tasks, offering a clear and thorough specification of their scope.

2. overly strong novelty claim
   Thank you for your reminder. When we refer to our work as an initial step, we mean we first emphasize NUPA itself as an independent task, characterized by representation complexity and diversity, separate from math or reading comprehension. We acknowledge that some statements in the previous version may be unclear and overstated. We have revised them into more accurate ones like "Our work provides a more detailed and comprehensive understanding of NUPA in LLMs" in the updated submission.

3. Add explanations
   We include the following explanation in the updated version:

• For the random tokenizer, we add a brief introduction at the beginning of the paragraph "random tokenizer" as follows:

  Introduced as ``sub-word regularization'' by [5,6], the random tokenizer splits words like "Hello world" into variable tokens such as "He/llo/ world" or "Hell/o/ world". Though not widely used in LLMs, [7] found that it enhances reasoning by introducing variability in generation path. Inspired by this, we apply this to the numbers, segmenting numbers into tokens with lengths randomly chosen between 1 and a predefined maximum, instead of using greedy left-to-right segmentation.

• We clarify the distinction between "length-related" and "length-agnostic" at the beginning of section 3.2 (Now in appendix A.4.3 due to the space limitation). Using integer addition as an example, its rules are length-agnostic since addition involves processing numbers digit by digit, unaffected by their length. However, during training, because the model is exposed to numbers within a limited length range (e.g., 1 to 8), it may develop a length-related rule that combines the original addition rules with an artificial constraint like "the output length must be between 1 and 8."

• We add an explanation of the PE results at the end of Appendix A.4.3. In summary, we believe that RoPE allows models to learn positional and length information more effectively, which in turn encourages the model to adopt a length-related rule as a shortcut.
  The question of "why RoPE facilitates learning length information" is intriguing but beyond the scope of this study. This phenomenon is likely tied to the architecture of the PEs. For instance, RoPE encodes positional information using a d-dimensional vector, whereas Alibi relies on a scalar, and NoPE uses no positional encoding at all. While we provide some intuition, a detailed analysis of the mechanisms behind PEs is a substantial topic on its own, and we look forward to further research in this direction.

• We add a explanation about what are and why we choose RoPEs, NoPEs and Alibi.

  RoPE, widely used in Llama and its derivatives, is the most classic relative PE. Then alibi, another relative PE, is proposed to address RoPE's length overfitting issues. NoPE (transformers without PE, relying solely on the causal mask to encode the position information) offers a surprisingly easy way to achieve length generalization. Therefore, we compare these three typical PEs to evaluate the performance on NUPA.