FACTOR: Factoring Complexity and Context Length in Long-Context Model Evaluation

Thank you so much for your attention to our paper. We really appreciate that you think of our benchmark originality and acknowledge the quality of our evaluation. We will try our best to address your concerns about dthe iversity of the dataset, computational cost, stability of the metrics, and others. If you feel that your concerns have been addressed, please consider raising your score.

• Task diversity is limited

Thank you for such an insightful comment. To increase the diversity of our current dataset, we introduce an new problem generation system for generating common-sense reasoning problems of large volume with fine-grained control. (More details about the common sense problem generation in $\text{\textcolor{blue}{Section 4.2, page 6}}$) To summarize, we see the potential in mapping between computational graph and natural language problems from the old FACTOR and we apply its spirit onto real-world context. The result is the new FACTOR in the revised paper, containing three subset of tasks classified based on hierarchical complexity in natural language structure (elaborated intensively in paper). The easy subset contains the old FACTOR, while the medium and hard subsets contains the reasoning problem in the real-world context. The latter two contains problems similar to the ones in GSM-8K and has 19 splits of problems, ranging from #ops = 2 to #ops = 19). To further increase the diversity of the tasks, we use three different real-world context template, together with both forward reasoning and backward reasoning modes. More details and evaluations can be read in the revised paper. Benefitting from this move, we can evaluate LLM on contextual problems that can be easily scaled up in difficulty. Below, we show results of Llama-3.1-70B-Instruct evaluation on these three subsets. You can clearly see that the rate for the accuracy degradation is very different, Easy subset has the slowest decay rate, while Hard subset decay the fastest.

• FACTOR is computationally expensive?

We appreciate the concern about computational cost when running the full benchmark. We admit that running the full benchmarks is computationally expensive. However, our proposed results modeling is here to help. We found that the exponential decaying trend actually fit the measured experiments quite well. Therefore, it provides you an opportunity to only subsample some of the ops from 1 to 39, then run an evaluation on them then extrapolate onwards on missing op between 1 and onwards seems to provide a reasonable solution. Below we subsample from the FACTOR EASY subtask using 8B and 70B. You can see that as we increase the subsample ratio or the stride, the MSE of the fitted log-linear curve starts to fluctuate. But mostly doesn’t differ from an order of magnitude. It seems that strides 3 and 4 are reasonably close to the baseline here which is stride equaling 1, or no subsampling. Therefore, the computation can be shrunk by ~3x - 4x with a reasonable loss in precision.

Also, one other way is to test on the Hard subset where the 8B model hardly gives reasonable results when op > 5, compared to EASY when 8B has a score at 20% accuracy even when op = 20, which would drastically shrink the number of ops to run to grasp the full landscape of model performance.

An interesting note when we approach this problem, we discover a pleasing insight about the error of the fitted CDO and CDF, that is when assuming the decay of accuracy is exponential with respect to the number of operations, the error is proportional to the inverse of square root over the total number of examples measured. We found that this law doesn’t require all ops need be measured, it is only related to the total number of examples you measured. More elaboration and proofs are in the $\text{\textcolor{blue}{Appendix D4.1 - D4.3 and D4.5, pages 22 - 23}}$ of the paper.

• Stability of CDO and CDF of the fitted curve when changing seeds

Thank you for the insightful question. We found that the result is very robust and stable. Below is the study that would cover the case you are specifically concerned about. We contrast between two different settings, for FACTOR EASY, we select the Qwen 2.5 7B model and run it for the entire 1 to 39 subsets. The difference is that for one scenario, we evaluate only 50 examples for every value, while for the other scenario, we evaluate 2000 examples for every value. (You can think of 2000 as we run the first case 40 times but each with a different seed). We compare the area under the curve for these two, for 50 it is 1470, while for 2000, it is 1490, differing by 1.3%. The gap is minimal. In the $\text{\textcolor{blue}{Appendix D4.4, page 23}}$ of the paper, we also present the figure of this experiment, we can see that the 50 example case has roughly the same shape as the 2000 version as well.

• RAG experiments are missing

Here we present thorough evaluations of studies of RAG on our dataset. Thanks to our well-conceived noise, FACTOR medium and hard effectively prevent RAG from solving the question entirely. (Details about the noise addition in $\text{\textcolor{blue}{Section 4.4, page 7}}$).

The result of our high-quality noise addition is that RAG is no longer effective in solving our tasks. (Referring to revised $\text{\textcolor{blue}{Section 3 page 4 - 5}}$) for more) We build a standard RAG system using Llama 3.1 8B Instruct as the decoder while using the al-mpnet-base-v2 for the retriever. We evaluated the system on various previous and standard long context datasets to show that the RAG delivers reasonable performance. Here we highlight the Variable Tracking (VT) from RULER, our method gives 86% acc, whereas LLM directly gives 99%. We use problem context 8K while the budget for RAG is to be 2K. We tested this RAG system on our FACTOR medium subset, and we found that it quickly dropped to zero. We also replaced the decoder with Llama 3.1 70B Instruct, and we found that it is the same pattern.

To make the case more convincing, we now enable iterative prefilling for the RAG side. We disregard the efficiency of the RAG system and test its performing ceiling. It now reaches 100% on the VT. However, it still struggles heavily on our FACTOR medium when op > 3. You can see that the accuracy quickly drops to zero. Therefore, RAG cannot determine the essential logic path in its chunk embedding space. Please refer to the revised paper for more details.

• What is the point of repeated sampling

Thanks for asking the insightful questions. We acknowledge that repeated sampling is one technique for scaling up inference time computation, but it isn’t the only way. On the other hand, many recent works (see citations below) show that repeated sampling (LMM, Aviral Kumar) leads to strong improvement in math and coding generation tasks.

Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787.

Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling llm test-time computationally optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

From our study, we found that repeated sampling is most effective in the first half of the ops range compared to the second half out of the 1 - 39 ops range. Please refer the Figure 8(b) from the appendix. Its effectiveness diminishes for harder problems. For op 15, sampling 64 times boosts the performance from around 45% to around 90%. However, for op 30, the increment from zero to 64 only increases the performance from around 5% to 20%. We hope this finding to be helpful for the community to know when reading our benchmark.

• Scoring Miscommunication

Thank you for expressing your concern. In our grading system for the FACTOR easy task, a partial answer of an incomplete list of variables is counted wrong. The score can either be 0 or 1 and only when the answer is precisely v3 and v5 does the score become 1. No partial result is allowed.

• Result holds for other tasks

In $\text{\textcolor{blue}{Table 5, page 9}}$, you can see that our evaluation of the FACTOR Easy subset produces rankings on correlate closely with the LMSYS ELO score, which is widely accepted as the popular benchmark for evaluating the LLM generation ability.

Thank you so much for your attention to our paper. We appreciate your acknowledgment of the comprehensiveness of our evaluation. We will try our best to address your concern about the diversity of the dataset, modeling the results, RAG, and others. If you feel that your concerns have been addressed, please consider raising your score.

• Lack of diversity of the benchmark and also miscommunication of the benchmark grading

In the new version of FACTOR, we address the mentioned narrowness from the old benchmark. To increase the diversity of our current dataset, we introduce a new problem-generation system for generating common-sense reasoning problems of large volumes with fine-grained control. (More details about the common sense problem generation in $\text{\textcolor{blue}{Section 4, page 6 - 8}}$) To summarize, we see the potential in mapping between computational graph and natural language problems from the old FACTOR and we apply its spirit to real-world context. The result is the new FACTOR, containing three subsets of tasks classified based on hierarchical complexity in natural language structure (elaborated intensively in the revised paper). The easy subset contains the old FACTOR, while the medium and hard subsets contain the reasoning problem in the real-world context. The latter two contain problems similar to the ones in GSM-8K and have 19 splits of problems, ranging from #ops = 2 to #ops = 19). To further increase the diversity of the tasks, we use three different real-world context templates, together with both forward reasoning and backward reasoning modes. Below, we show the results of Llama-3.1-70B-Instruct evaluation on all three subsets. The easy subset has the slowest decay rate, while the Hard subset decays the fastest.

Also, some clarification of the later questions about problem design and parsing implementation. For the Easy subset (the old FACTOR), we ask about all variables that have a certain value. We don’t have partial credit for questions: it either is correct because all variables are found out correctly or not correct. Therefore, no shortcut can be made in the general case, the model has to go through the entire computation graph and go for the exact number of operations listed. Remember that the problem is based on a randomly generated computation graph, while the variables to begin solving the graph have randomly generated values. Therefore, to consistently solve the problem correctly, the entire computation graph has to be traversed. The graph structure is random with no priors.

For the newly added common sense questions in the Medium and Hard subsets, the solution generated is based on a topological sort of computation DAG randomly generated. The number of ops is determined based on the length of the topological sort of the graph. Therefore, it is impossible to find a shorter path to solving the problem.

• Logistic Regression vs. Two Phase constant plus loglinear decay

We appreciate the thoughtful discussion about why you prefer logistic regression. To settle this argument, we identify three cases and fit both the log-linear and logistic regression. However, based on our observation from the studies, the decaying exponential seems to be a better option for accurately describing the trend. Here o1-mini is on Easy subset is the only case where we are dealing with this first plateau and then decay scenario for op > 0. We show visualization in $\text{\textcolor{blue}{Figure 7, page 7 - 8}}$ of the revised paper.

For FACTOR Easy with Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct

Consistently, as you can see below, the log-linear always gives a much better fit (sometimes by one order of magnitude). We use MSE for metrics. The O1-mini model has this phase transition from near 1 perfect to starting to decay. Our paper uses a two-phase step-wise function consisting of a constant plus an exponential decay. It achieved a lower and better MSE score compared to the logistic regression fit. As can be seen from the visualizations, once the model starts decaying, it drops steeply, rather than smoothly at first then accelerate as in logistic regression. As for the possible negative extrapolation, we now avoid it by clearly stating the support: the ops count for the curve has to be positive.

Thank you for your speedy response to our posts. Also, thank you for acknowledging our effort. We summarize the two lingering concerns you have and try our best to address them. If you feel that your concern is addressed, please consider raising the score.

• The comparison between exponential and logistic regression.

We appreciate your acute observation that the curve of the accuracy decay of LLM versus operation count (op) on our dataset has three different parts: a plateauing phase at the beginning when op is close to zero and the question is easy; the rapid decaying phase when op increases; and a plateauing phase for accuracy around zero when the op is high and LLM no longer capable of solving the problem. Therefore, it is natural to use logistic regression, a function so elegant that can cater to all three phases at once, showcasing your deep observation about our tendency. We want to assure the reviewer that we have taken this proposal seriously, and we also like this idea a lot. In fact, we have modified the analysis portion in $\text{\textcolor{blue}{Section 5.1, page 8}}$ to take this into our analysis. In contrast, we proposed to use two functions (a step-wise function) to explain what we observed from the curve. (In the paper, we write it as two-phase behavior in Section 5.1) First, a constant function to explain the initial plateauing stage, followed by an exponential decay function to explain the subsequent two stages. In the paper, we write the second stage as "log linear", so fitting log(y) = a'x + b'. Thus, y = exp(a'x + b'), a' < 0 since it is decaying. It can be written as 1/exp(ax + b) for a positive number a and b.

The two approaches are almost identical when x is positive and large since given a linear relationship of x, f(x) = ax + b. Logistic regression gives 1/(1 + exp(f(x))). When x is large, it is almost 1/exp(f(x)): this means that if a sigmoid curve can fit raw data well on the high magnitude positive regime, the raw data can also be fitted well by an exponentially decaying function. The only difference is at the transition between the initial plateau and subsequent decay, where logistic regression predicts it to be a smooth drop, where our stepwise function shows a sudden drop. Currently, as we have shown, the vast majority of the models barely have the initial plateauing phase due to the lack of reasoning ability. Therefore, the focus of today is on the decaying stage, where these two functions are almost identical. Only a very handful of models such as o1-mini are showing all three phases. On o1-mini, the step-wise approach is shown to better fit the data than logistic regression. When focusing on the decaying phase of the trend, we present a costly experiment running Gemini-1.5-Flash with high precision from op 1 to 160. We show the results in $\text{\textcolor{blue}{Figure 8, page 19}}$. According to the MSE score, exponential decay is better than logistic regression fit. However, in the short future, with the advent of a plethora of stronger reasoning models, some models might have different trends, and we will update our modeling function once the trend differs. Thank you again for proposing the logistic regression model and your acute observation.

• RAG

Thanks for pointing out your confusion. RAG experiments are the motivation of our benchmark. RAG experiments reveal the deficiency of the current long context benchmark: the current benchmark shows that the gap between the highly expensive to train and deploy long context LLM and the simple-to-build RAG counterparts is very close. The tremendous cost of building long-context LLM cannot be justified merely based on the current long-context benchmarks. Therefore, we need a new benchmark that truly differentiates these two, and therefore accentuates long-context LLM's value.

Specifically, to prove the current long context benchmark's inefficiencies in the paper, we first show that conventional RAG is strong on previous standard long context benchmarks that are supposed to be used to evaluate LLMs. However, because of RAG's strong performance, the advantage of LLM over RAG on long context jobs is marginal and questioning. Then, we show that our tasks, because of the reasoning nature, completely prevent RAG from doing it effectively. Even if we enable iterative prefilling, which is close to the upper bound of RAG, RAG performance continues to be ineffective. In comparison, LLMs' have superiority and a huge margin over RAG on our tasks. Does that help clarify your confusion about RAG experiments' results? Let us know whether you have more lingering concerns about RAG.

• How many tasks are in FACTOR benchmark?

For the old FACTOR (initially submitted), we only have one suite of tests that consists 39 different subsets of tests, they are from operation number of 1 to operation number of 39. After the expansion, the new FACTOR now consists of three suites: Easy, medium, and hard. While Easy is the old FACTOR. For medium and hard, each consists of 19 subsets from operation number of 2 to 19. For every data point presented, we test at least 200 examples of that kind.

• Could you present the results from Table 1 (now $\text{\textcolor{blue}{Table 5, page 9}}$) in a way that clearly highlights which models are performing better?

We acknowledge that there is confusion in our dataset description. More can be found now in $\text{\textcolor{blue}{Section 5.1 page 9}}$ of the revised paper. We re-organize Table 1 in the newly edited paper. Previously, we had two metrics, which makes it less intuitive to compare between models. Now, we propose to compare using the area under the curve (AUC), while CDF and CDO are also presented for a better understanding of the model behavior.

• o1?

In our initial submission, we actually have an o1-mini result in $\text{\textcolor{blue}{Figure 3 (c), page 8}}$. You can see that normally, models would have a score near zero for operation nearly 40, while O1-mini can go past operation 160. Showing its strong reasoning ability.

Thank you so much for your attention and the time taken to read the paper. We appreciate your acknowledgment of the methodology and significance of the benchmark. We will try our best to address your concern on filler text and diversity of the benchmark, together with other questions. If you feel that your concerns have been accounted for, please consider increasing your score. For better readability, we rearrange the problem responses.

• Only symbolic assignment and limited diversity

Thank you for such an insightful comment. To increase the diversity of our current dataset, we introduce a new problem-generation system for generating common-sense reasoning problems of large volumes with fine-grained control. (More details about the common sense problem generation in $\text{\textcolor{blue}{Section 4.2, page 6}}$) To summarize, we see the potential in mapping between computational graph and natural language problems from the old FACTOR and we apply its spirit onto real-world context. The result is the new FACTOR in the revised paper, containing three subsets of tasks classified based on hierarchical complexity in natural language structure (elaborated intensively in the paper). The easy subset contains the old FACTOR, while the medium and hard subsets contain the reasoning problem in the real-world context. The latter two contain problems similar to the ones in GSM-8K and have 19 splits of problems, ranging from #ops = 2 to #ops = 19). To further increase the diversity of the tasks, we use three different real-world context templates, together with both forward reasoning and backward reasoning modes. More details and evaluations can be read in the revised paper. Below, we show the results of Llama-3.1-70B-Instruct evaluation on these three subsets. You can discerningly see that the rate for the accuracy degradation is different, the Easy subset has the slowest decay rate, while the Hard subset decays the fastest.

• Filler text is random and not random is preferred

The filter text is indeed vital for our benchmark. Together with our well-conceived noise, FACTOR medium and hard effectively prevent RAG from solving the question entirely. (Details about the noise addition in $\text{\textcolor{blue}{Section 4.4, page 7}}$). One sentence to summarize how our close noise is added: benefitting from having a problem generator in our design of the dataset construction. We carefully devise a method that uses the same problem generator to generate noise statements that is super close in semantics and format and have strong links to the essential logic statements that are supposed to be used to solve the problems.

As a quantitative study of the noise quality, we conduct the following experiment. The experiment is based on FACTOR Medium, the commonsense reasoning subset. We compare two different types of noise: one is the aforementioned close noise generated by the problem generator, while another one is the noise I generated by asking the GPT-4o model to write a lengthy documentary related to the problem text. A RAG with 2K context with Llama 3.1 8B Instruct together with a full context Llama 3.1 8B Instruct on the 8K medium subset of FACTOR. Also, for the best theoretical performance of the RAG, we enable iterative prefilling of every token generated. The result is presented in the table below. As you can see, for long context LLM like Llama 3.1 8B Instruct, two types of noise behave similarly, the GPT-4O generated noise is even slightly harder in some ops. However, for the RAG, the close noise significantly worsens its performance compared to the generated noise, further proving the high quality of the noise.

• How many tasks and more details about the dataset construction

For the old FACTOR (initially submitted), we only have one suite of tests that consists 39 different subsets of tests, they are from operation number of 1 to operation number of 39. After the expansion, the new FACTOR now consists of three suites: Easy, medium, and hard. While Easy is the old FACTOR. For medium and hard, each consists of 19 subsets from operation number of 2 to 19. For every data point presented, we test at least 200 examples of that kind.

Thank you so much for your attention and time taken to read the paper and your insightful comments. We are very thankful that you like our paper’s initiative to evaluate model reasoning ability through fine-grained control of the level of difficulty. Also, your questions are very insightful in our development of the FACTOR benchmark. We revise our methods hoping to address your concern. If you feel that your concern has been taken into account, please consider raising your score. The question response orders are rearranged to better help the narrative and for better understanding.

• Filler text to be random text and RAG experiments missing?

The filter text is indeed vital for our benchmark. Together with our well-conceived noise, FACTOR medium and hard effectively prevents RAG from solving the question entirely. (Details about the noise addition in $\text{\textcolor{blue}{Section 4.4, page 7}}$). One sentence to summarize how our close noise is added: benefitting from having a problem generator in our design of the dataset construction. We carefully devise a method that uses the same problem generator to generate noise statements that are super close in semantics and format and have strong links to the essential logic statements that are supposed to be used to solve the problems.

The result of our high-quality noise addition is that RAG is no longer effective in solving our tasks. (Referring to revised Section 3 for more) We build a standard RAG system using Llama 3.1 8B Instruct as the decoder while using the al-mpnet-base-v2 for the retriever. We evaluated the system on various previous and standard long context datasets to show that the RAG delivers reasonable performance. Here we highlight the Variable Tracking (VT) from RULER, our RAG method gives 86% acc, whereas LLM directly gives 99%. We use problem context 8K while the budget for RAG is to be 2K. We tested this RAG system on our FACTOR medium subset, and we found that it quickly dropped to zero. We also replaced the decoder with Llama 3.1 70B Instruct, and we found that it is the same pattern. More details about our evaluation are in $\text{\textcolor{blue}{Section 3, page 3 - 5}}$.

To make the case more convincing, we now enable iterative prefilling for the RAG side. We disregard the efficiency of the RAG system and test its performing ceiling. It now reaches 100% on the VT. However, it still struggles heavily on our FACTOR medium when op > 3. You can see that the accuracy quickly drops to zero. Therefore, RAG cannot determine the essential logic path in its chunk embedding space.

• CDO metric hard to understand? Thank you for raising your concern about the interpretation of the CDO. We realize that having the metric defined as the log-linear curve interception with the y-axis has the issue of interpretability. We shift and now use the measured score at op = 1 to be the value to replace CDO, measuring the baseline performance of a model in that context length. Having op = 1 essentially reduces the task to a retrieval task, removing the complexity of logic and effectively only measuring the model performance in the given context. We thank the reviewer for your insightful comments.