Advancing Table Understanding of Large Language Models via Feature Re-ordering

1. Limited diversity of datasets. Although the method is evaluated on four datasets, expanding to more diverse and complex datasets would strengthen the empirical results. Additionally, exploring whether a similar order bias exists for regression tasks and whether the approach can be applied to those cases would further enhance the findings.

2. Scalability to larger datasets. The dynamic programming approach may potentially struggle with a larger number of features, requiring much more trajectory data to train the meta-controller. A more thorough discussion and empirical evaluation of the method's scalability would be beneficial.

1. Not enough ablation studies to debunk what really helps on performance. There are a few confounders which could be nice to be addressed. The proposed approach uses the ranking LLM like inference time ranking or model ensemble. When ranking the input fields, the base LLM model essentially takes as input ranking LLM classification results. Are we able to tell this improvement is from the input fields ordering or it's just because the model ensemble effect.
2. The most challenging part of the proposed method would be that whether the ranking LLM's help can be transferred across different tasks. The authors only investigated the transferablility of the ranking LLM across different models.

I have four major issues that stop me from giving it a higher note.

1. Fundamental Conceptual Issues with Feature Reordering

There is an inherent flaw in the approach of seeking an optimal feature order across the dataset instead of designing models capable of processing tabular data without such manipulations. Relying on reordering features to mitigate the ordering bias addresses a symptom rather than the root cause—the LLM's sensitivity to the sequential input of features. Ideally, models should be robust to feature order, especially given that in tabular data, the arrangement of columns is generally arbitrary and does not convey additional meaning. Furthermore, if the optimal feature order differs for each individual data point, the meta-controller may not converge to a single consistent order when trained on different subsets of the data. For instance, dividing the dataset into ten parts could lead to the meta-controller identifying different optimal orders for each subset, indicating that there is no universal optimal order. This variability raises questions about the practicality and scalability of the method, as it may not be feasible to train and maintain multiple controllers for different data partitions. It also suggests that the performance gains may not generalize well across diverse datasets or in real-world applications where data characteristics can vary significantly.

2. Weak Baselines and Comparisons

The baselines used in the experiments may not be strong enough to convincingly demonstrate the superiority of ROTATOR-LLM. Comparing against more robust and diverse baselines would provide a better assessment of the method's effectiveness. Some of the baselines that should be considered include:

1. Prompting the LLM to Reorder Features: Instead of using a fixed feature order, prompt the LLM to reorder the features itself and use that order. This could involve experimenting with different prompts or formatting examples to leverage the LLM's few-shot in-context learning abilities. By doing so, the LLM's inherent capabilities can be utilized without additional complexity.

2. Multiple Random Orders: Evaluating the model's performance using multiple random feature orders, possibly duplicating the information, to assess the impact of feature order variability without the need for a complex controller. This baseline would help determine if the benefits of feature reordering are due to the specific order found by the meta-controller or can be simply achieved by exposing the model to diverse orders.

3. Feature Importance-Based Ordering: Ordering features based on their importance as determined by traditional machine learning models (e.g., Random Forest, XGBoost). This could provide a simple yet effective baseline for feature ordering, leveraging established techniques in feature selection and importance weighting.

4. Iterative Feature Selection: The paper demonstrates that utilizing 10 features instead of 16 can yield improved performance on certain datasets. It would be valuable to assess whether recursive feature elimination alone is adequate in these instances. Alternatively, the authors could employ a greedy feature addition strategy, sequentially incorporating the features that provide the most significant performance enhancements first.

5. Comparison with Traditional Models: Since the task involves classification or regression over tabular data, it may be helpful to establish a baseline based on classical machine learning models like Random Forests or XGBoost. This would help contextualize the gains achieved by the proposed method and LLMs.

By not including these stronger baselines, the paper misses an opportunity to demonstrate that ROTATOR-LLM offers substantial improvements over simpler or more traditional approaches. Thus, forcing me to question the necessity of introducing the complex solution. It is unclear whether such a complex method is required to address the feature ordering bias. Simpler solutions, such as heuristic-based ordering strategies might provide comparable improvements without the added computational overhead. The paper does not provide sufficient justification for choosing a meta-controller over these simpler alternatives.

3. Limited Discussion of Limitations

The paper does not explore the limitations of the proposed method. For instance, it does not discuss scenarios where feature reordering might not lead to performance gains or could potentially degrade performance. How robust is the method across a spectrum of datasets remains unclear as the paper only validates it on a handful of datasets. Also, regarding transferability to other datasets: While the meta-controller shows some transferability across LLMs, it's unclear whether the approach is transferable across different datasets, especially those with different feature characteristics or domains.

4. Clarity and Presentation Issues

1. Writing Style: There are instances of awkward phrasing and grammatical errors. For example, sentences like "We demonstrate novelly discover and thoroughly explore the order bias..." contain mistakes that should be corrected. It is very minor in this paper, but rereading it should help. Above that, the citation style is a bit unfortunate. Please put the citations in between brackets.

2. Claims of Optimality and Maximization: The use of terms like "optimally" (e.g., line #370) and "maximally leverages" (e.g., line #477) are overstatements without rigorous proof or exhaustive empirical evidence. There is no indication that all possible permutations were tried to prove optimality and such claims should be moderated unless substantiated.

3. The methodology does not adequately address why duplication occurs in the reordered features, whereas the experiments do not show how it impacts model performance.

4. The paper overemphasizes attention entropy as a measure of attention concentration, but it is not clearly established how this directly correlates with performance improvements. Additional analysis or justification connecting attention patterns to model accuracy would strengthen this argument.

5. A visualization of how the meta controller generates feature trajectory would be helpful for a reader and lower the barrier to understanding the contribution.

1. The problem setting is not very clear. The paper focuses on the table understanding while the so-called "tabular feature" is not introduced clearly. Existing work may represent a table in markdown/JSON/XML formats to LLMs while the "feature" needs further definition. Are the features the cells in the table? Please explain the problem settings more clearly.

2. The baseline choices are too limited. In the paper, the proposed method is only compared with Default/Random. However, the meta-controller is trained on the training set while these baselines don't learn anything from the training data. The comparison is not fair toward the simple baselines.