Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

Q1:How does ModernNCA perform with different distance metrics, such as cosine similarity, especially for datasets with high-dimensional features?
A1: Please refer to A4 for reviewer 3moZ.

Q2: while the model shows high performance, the paper lacks specific scenarios or guidance on applying modernnca practically, such as when dealing with imbalanced datasets. Given that ModernNCA outperformed many models, have you tested its performance on imbalanced datasets?

A2: Thank you for this insightful question. To investigate the performance of ModernNCA on imbalanced datasets, we selected 64 datasets from the 180 classification datasets used in our main experiments. These datasets were identified as imbalanced based on a class ratio greater than 5 between the major and minor classes. In this case, we used F1 as the evaluation metric, as it accounts for class imbalance.

The average performance ranks of representative methods are listed below (lower rank indicates better performance):

We have the following two observations:

1. Competitive Performance: ModernNCA achieves a rank comparable to the strongest method, CatBoost, and outperforms other deep tabular methods, including TabR and MLP variants.
2. TabR Limitations: TabR struggles on imbalanced datasets compared to more balanced cases. A likely reason is that the imbalanced label distribution makes it difficult for TabR to retrieve meaningful neighbors effectively.

Therefore, ModernNCA demonstrates robustness on imbalanced datasets due to its reliance on a learned embedding space that clusters similar instances effectively, regardless of class distribution. This clustering property ensures that neighbors belonging to minority classes are still well-represented in the embedding space.

Moreover, ModernNCA's simplicity makes it amenable to standard techniques for handling imbalance. For example: we can adjust the weights of Neighbors with a learned ModernNCA model, where The influence of neighbors can be adjusted by assigning higher weights to minority class instances. We denote this version as ModernNCA (post-weight), which achieves the best performance among all other models.

These observations and practical suggestions will be included in the final version of the paper to provide more actionable insights for practitioners.

Q3: Could you provide more insights into cases where ModernNCA significantly underperforms compared to catboost or other tree-based methods?

A3: Please refer to A3 for reviewer Cjck.

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

Q1: Marginal Contribution. The paper’s contribution feels incremental rather than pioneering. The improvements in MODERNNCA rely on established techniques (SGD, SNS, and nonlinear embeddings) without introducing a fundamentally new concept or method. This makes the novelty limited, as it essentially optimizes an existing algorithm rather than providing a unique advancement.

A1: Thanks. We would like to clarify the value and context of ModernNCA’s contributions, emphasizing that while the core techniques used in the paper are possibly established, their effective integration to achieve strong performance on tabular tasks represents a significant advancement. We have provided a detailed response to a similar concern raised by review Qykr. In the following, we further highlight our technical contributions and significance.

1. The Value of Making Existing Ideas Work Effectively. While ModernNCA builds upon existing components like SGD and nonlinear embeddings, the paper demonstrates how to combine these elements to optimize the performance of Neighborhood Component Analysis (NCA) for tabular data. This approach bridges a gap between classical metric learning and modern deep learning, which, to the best of our knowledge, has not been successfully achieved before.
2. The mindset behind ModernNCA is grounded in solving real-world problems by leveraging what we already know and enhancing it through thoughtful integration and design. This philosophy aligns with many impactful works in machine learning, where the emphasis is not solely on theoretical novelty but also on advancing the utility and practicality of established methods. For example, the adoption of SGD for training NCA might seem incremental, but it significantly improves scalability and generalization in real-world scenarios; SNS improves efficiency while preserving or even enhancing performance, a critical requirement for handling large-scale datasets.
3. Revisiting Classical Methods for Modern Applications. ModernNCA is not merely an incremental optimization; it represents a paradigm shift in how classical ideas can be revisited and adapted for contemporary challenges. Our observations show that simple adjustments like switching to a soft nearest neighbor loss, introducing nonlinear architectures, and employing SNS unlock substantial performance gains, surpassing state-of-the-art methods in many cases; the fact that ModernNCA can consistently compete with and often outperform sophisticated models like CatBoost and TabR highlights the practical significance of these contributions.

We will incorporate these clarifications into the final version of the paper to better articulate the significance of ModernNCA's contributions.

Q2: Limited Novelty in Comparison to KNN Variants. The paper lacks direct comparisons with other KNN-inspired deep learning methods that have similarly benefited from modern optimization strategies. This limited scope of comparison weakens the argument for MODERNNCA’s distinctiveness and impact.

A2: Thank you for the comment. We would like to address the concerns regarding novelty and distinctiveness by clarifying the comparisons included in the paper and adding further context. First of all, we have included comparisons with two representative KNN-inspired deep-learning methods (TabR and TabCon) in our submission.

1. Existing Comparisons with KNN Variants. In this paper, we directly compare ModernNCA with multiple KNN-based methods, including:
   • Vanilla KNN: A baseline non-parametric approach using raw feature distances.
   • TabR: A strong neighborhood-based deep tabular model.
   • TabCon: Our proposed baseline model, which utilizes supervised contrastive loss during training as ModernNCA and applies KNN over the learned embeddings in inference (described in Appendix A.4).
2. These comparisons highlight the improvements ModernNCA achieves by combining classical ideas with modern deep learning techniques.

To strengthen the argument for ModernNCA’s distinctiveness, we extend our comparisons to include PFN-KNN $A$ , a recently proposed KNN-based method specifically designed for classification datasets. We evaluate these methods on 45 datasets from our ablation studies. The results, shown below, indicate that ModernNCA maintains its superiority over other KNN variants:

While ModernNCA could incorporate more "fancy" components, we consciously chose to focus on simplicity and practical utility. ModernNCA achieves strong results without unnecessary complexity, making it both effective and accessible.

In summary, ModernNCA establishes itself as a robust KNN-inspired method, leveraging classical principles with modern enhancements. Its superior performance across multiple benchmarks, even against advanced KNN variants like TabR and PFN-KNN, underscores its distinctiveness and practical impact. We will further emphasize these comparisons and their implications in the final version of the paper.

$A$ Retrieval & Fine-Tuning for In-Context Tabular Models. NeurIPS 2024.

Q3: Lack of Theoretical Analysis. The paper’s focus is primarily empirical, with little theoretical exploration of why the proposed enhancements lead to improved performance. A deeper theoretical perspective on the modifications would provide valuable insights and strengthen the work's academic contribution.

A3: Thank you for this suggestion. We acknowledge that the inclusion of a theoretical analysis would strengthen our work, and it applies to all deep tabular methods as well. However, conducting theoretical analyses is non-trivial, especially for deep methods. Often, one needs to make strong assumptions, which inevitably degrade the applicability of the analysis. To our knowledge, very few deep tabular methods have included theoretical analyses.

That said, nearest-neighbor-based methods have a solid theoretical foundation. Our modifications, though without formal theoretical analysis, were well-motivated to address the challenges we observed. We deliberately kept our modifications simple and intuitive and validated them with extensive experiments. Put together, we respectfully think that our empirical evidence offers valuable insights for future work to build upon.

Empirical Strength and Simplicity. Our approach aims to build on an established and interpretable method, NCA, by incorporating modern deep learning techniques. These modifications, such as Stochastic Neighborhood Sampling (SNS), are empirically validated across a large-scale tabular benchmark, demonstrating consistent improvements in performance.

Theoretical Implications: While we do not delve deeply into theoretical proofs, the proposed enhancements carry intuitive theoretical justifications:

• Stochastic Sampling and Generalization: SNS introduces randomness during training, which likely acts as an implicit regularization mechanism, forcing the model to generalize better by learning robust embeddings that are not overly dependent on specific neighbors. This aligns with general theories of stochastic training methods in deep learning.
• Soft-NN Loss: The use of a differentiable soft nearest neighbor loss directly aligns the embedding optimization with the prediction objective, theoretically reducing the mismatch between training and inference objectives compared to hard nearest neighbor methods.

We emphasize that our primary goal was to provide a strong empirical foundation for these techniques, which can serve as a baseline for both practical and theoretical advancements. Our approach’s simplicity is a strength, demonstrating that even straightforward extensions of a classical method can rival or outperform complex models.

In summary, while the theoretical exploration of our enhancements is limited in this paper, we have laid the groundwork for further theoretical studies and welcome future research that builds on our contributions to better formalize their effects.

Q4: MODERNNCA relies on Euclidean distance as the default metric. How adaptable is the model to other distance metrics, and do the authors have insights on how these choices might impact performance?

A4: Thank you for raising this question. We evaluated the adaptability of ModernNCA to alternative distance metrics, including cosine distance, L1-distance, dot product, and squared L2-distance. The results of these experiments are provided in Appendix B.2 (Table 5). Below, we summarize our key observations:

1. Performance Overview: On average, Euclidean (L2) distance performs well across both classification and regression tasks, making it a robust default choice.
   • Cosine Distance demonstrates superior performance on classification datasets (achieving an average performance rank of 4.5939 when compared with ModernNCA and 20 other methods across 300 datasets). However, its advantage diminishes on regression tasks, likely due to its focus on angular relationships rather than magnitude.
   • L1-Distance performs comparably to L2 on some tasks but has higher computational costs due to the absolute operations involved.
   • Squared L2-Distance slightly underperforms compared to standard L2, likely because the additional squaring step impacts gradient behavior during optimization.
2. Why we use L2 as the default choice:
   • Standard Practice: L2 has long been a standard metric in machine learning and deep learning, particularly in early methods like k-NN and NCA. Its simplicity, isotropic scaling, and compatibility with Euclidean geometry make it a reliable choice across diverse tasks.
   • Gradient Behavior: In optimization, L2 provides smooth and consistent gradients, which can enhance training stability. Metrics like L1 can lead to discontinuous gradients, making optimization more challenging.
   • Adaptability: Euclidean distance effectively balances both magnitude and directional information, which is particularly useful in high-dimensional embedding spaces.

In sum, while the techniques we employ (e.g., SGD, representation space learning) are not novel in isolation, their specific adaptation to tabular data and the resulting empirical performance represent a significant contribution. We position ModernNCA as a practical and competitive baseline, which, like foundational works in other domains (e.g., SimCLR, diffusion models), lays the foundation for future exploration and development.

We hope this clarifies the novelty and relevance of our work. We will incorporate these explanations into the final version of the paper for clarity.

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

Q: Lack of Novelty: While the paper shows strong empirical performance, the core modifications (using a representation space for distance calculations, employing SGD, and mini-batch training) have already been explored in prior research. This raises concerns regarding the originality of the contribution. The changes appear more like tunings of established techniques rather than introducing a fundamentally new method.

[Prior research example] J Kang et al., Deep metric learning based on scalable neighborhood components for remote sensing scene characterization, 2020.

If my understanding is incorrect, could you please clarify what is the novel concept introduced in this paper?

A: Thanks. We appreciate the opportunity to clarify what sets our work apart.

1. Reviving NCA for Tabular Data. While metric learning and neighborhood-based methods have seen extensive exploration in fields like computer vision (e.g., image retrieval) and remote sensing, their usage in tabular data, where they were originally designed, has been limited due to their less competitive performance. This remains the case even with the incorporation of deep learning techniques (please kindly see the remark in section 1 of the main paper). Therefore, we respectfully think how to effectively turn a classical method like NCA into a competitive tabular baseline itself is a significant and impactful contribution, regardless of whether each piece of our modifications has been explored somewhere in the literature.
   As an analogy, diffusion models have been brought into machine learning and computer vision since 2016; contrastive learning objectives have been well-known since early 2000. However, how to transform them into state-of-the-art generative models and self-supervised learning objectives is non-trivial, often involving insights into the detailed implementation. Likewise, NCA has existed for decades, but its potential for tabular data has not been fully realized. Our work demonstrates the importance of a thorough and thoughtful investigation in bridging the gap between theory and practical application.

2. Specifically, many prior works, including the mentioned paper by J. Kang et al. (2020), focus on vision-specific tasks. Their adaptations of NCA often incorporate metric learning losses (e.g., triplet loss) to fine-tune representations for image-based tasks. However, these methods do not address the specific challenges of tabular data, where feature sparsity, distribution skewness, and feature-engineering nuances differ significantly from vision domains. In contrast, our contribution is a focused rethinking of NCA tailored for tabular data, where design choices like Stochastic Neighborhood Sampling (SNS) and soft nearest neighbor loss in the inference stage play a pivotal role in addressing the scale and complexity of tabular datasets.

3. The Value of Simplicity. Simple methods that consistently work well across diverse datasets are crucial for practical machine learning. The performance of ModernNCA across 300 datasets, including classification and regression tasks, shows that simplicity need not sacrifice effectiveness.

4. Novel Insights for Tabular Data:
   We introduced several key ideas that are novel in the context of tabular data:

   • Stochastic Neighborhood Sampling (SNS): A principled and efficient approach for scaling neighborhood-based methods to large datasets, significantly reducing computational overhead while improving generalization.
   • Soft Nearest Neighbor Loss in Prediction: By aligning the loss function with the prediction strategy, we improve the robustness and scalability of neighborhood-based methods.
   • Deep Architectures for NCA: The paper explores how nonlinear representations and modern optimization strategies (e.g., SGD) fundamentally change the dynamics of NCA for tabular data, enabling it to outperform even tree-based methods like CatBoost.

5. Why This Work Matters: Metric learning is indeed a well-trodden path in vision research, but its adaptation to tabular data has been limited, particularly in creating methods that are efficient, scalable, and effective across diverse tabular datasets. ModernNCA:

• Outperforms contemporary deep tabular models like TabR and FT-Transformer on 300 datasets, including challenging real-world scenarios.
• Demonstrates how simple yet carefully integrated changes can transform a classical method into a state-of-the-art approach for a new domain.

Thank you for your response. As acknowledged by the authors, the primary novelty of this paper lies in applying the well-established NCA, a technique commonly used in other domains, to the tabular domain to achieve improved performance. However, based on the additional insights provided by the authors and results discussed in related works such as TabReD, the proposed approach does not demonstrate clear superiority over other DNN-based methods or GBDTs in terms of performance. This raises concerns that the reported results may reflect thorough tuning on specific datasets rather than a generalizable improvement.

Even without referencing external works, the results reported in this paper do not show significant superiority over other methods. Given that the methodology is not novel and the performance improvements are marginal, the primary contribution of this paper is limited to the application of NCA to the tabular domain. Additionally, the paper does not provide substantial insights or theoretical advancements in the discussion section, further limiting its impact.

In its current form, this paper lacks the innovation and contribution expected of ICLR-quality work. Therefore, I maintain my current score.

Dear reviewer Reviewer Qykr,

Thank you for your additional comment. We provide further responses as follows.

Q1: As acknowledged by the authors, the primary novelty of this paper lies in applying the well-established NCA, a technique commonly used in other domains, to the tabular domain to achieve improved performance. 

A1: We would like to clarify that the tabular domain is the domain where traditional machine-learning approaches like NCA were originally developed. Before the rise of deep learning, many machine learning methods assumed that the input data was pre-embedded into a fixed-dimensional feature space, aka, tabular data format. What we argued in the rebuttal is that before the application to the other domains like image recognition, nearest-neighbor methods have been widely applied to tabular data, but the performance seems suboptimal as validated in Table 2 (L432-444) in the paper. 

Therefore, in our humble opinion, our contribution is NOT “applying well-established techniques to a new domain” BUT “modernizing them for the domain they should have thrived in.” We systematically revisit NCA and revitalize it to extend its success from other domains back to the initial tabular domain.

Q2: However, based on the additional insights provided by the authors and results discussed in related works such as TabReD, the proposed approach does not demonstrate clear superiority over other DNN-based methods or GBDTs in terms of performance. …. Even without referencing external works, the results reported in this paper do not show significant superiority over other methods. 

A2: We would like to clarify that our primary contribution is not to claim superiority over existing leading methods, but to demonstrate a previously believed inferior approach can indeed achieve decent, competitive performance in the tabular domain. We respectfully view such a contribution as significant, as it opens up new directions for future research in tabular data. It has a profound implication that an appropriate integration of traditional methods and deep learning techniques could lead to strong approaches to tabular data. (We also humbly think that achieving SOTA accuracy is not a requirement for paper acceptance.)

Regarding the discussion in TabReD, ModernNCA was mentioned as impressive on standard classification and regression tasks. It was reported as less effective only under distributional changes, which is how TabReD split the training and test data. We note that addressing such shifts and improving out-of-distribution (OOD) generalization is a fundamental and ongoing challenge not just for nearest-neighbor-based methods, but for many machine learning approaches that rely on empirical risk minimization. We plan to explore methods for enhancing neighbor search and robustness under these conditions in future work. 

We note that TabReD was put on arXiv on Jun. 27, 2024 (version 1). The discussion of our approach was added in their version 4 on Oct. 24, 2024, after the ICLR 2025 deadline (Oct. 1, 2024). According to the ICLR 2025 Reviewer Guide (https://iclr.cc/Conferences/2025/ReviewerGuide), “We consider papers contemporaneous if they are published within the last four months.” We would also like to reiterate the point made by Reviewer Cjck: "Importantly, I don't think that performance on new benchmarks is a significant issue for the submission, as it still provides a valuable contribution to tabular DL models based on nearest neighbors. Improving performance in new scenarios is worthy of extended future investigations." We agree with this sentiment and will ensure that we acknowledge the limitation of nearest-neighbor-based methods and outline potential avenues for improvements in the final version of the paper.

Q3: This raises concerns that the reported results may reflect thorough tuning on specific datasets rather than a generalizable improvement. 

A3: We have evaluated our approach on 300 tabular datasets, with various evaluation criteria such as average rank, critical difference, paired t-test, and shifted geometric mean, which we believe have covered a diverse range of existing tabular problems. We humbly believe such a scale is beyond the scope of “specific datasets.”

Q3: Limitation of the method. Are there any cases where it may perform poor. Maybe some post-hock meta-analysis akin to the one in tabzilla paper (https://arxiv.org/abs/2305.02997) of what are the datasets where ModernNCA performs worse.

A3: Inspired by Tabzilla, we conducted a meta-analysis using dataset meta-features (e.g., number of features, feature-to-sample ratio, feature sparsity, range of feature values) to train a decision tree that predicts when the performance of ModernNCA falls behind XGBoost. This analysis is based on 25 meta-features,similar to those used in Tabzilla, and identifies scenarios where ModernNCA tends to underperform compared to XGBoost. The observations are summarized below (detailed results are available in Supplementary Material):

1. High Feature Sparsity: Datasets with a higher average sparsity of features pose challenges for ModernNCA.
2. High Feature-to-Sample Ratio: ModernNCA struggles on datasets with many features but relatively small sample sizes, leading to a large feature-to-sample ratio.
3. High Feature Skewness: Datasets with significant skewness in feature distributions (i.e., those deviating markedly from normal distributions) negatively impact ModernNCA's performance. This meta-analysis highlights specific dataset characteristics where ModernNCA might face limitations, providing insights for future work and potential areas for improvement. We will incorporate the discussions in the final version.

Q4: Some recent benchmarks demonstrate that retrieval-based models might not be universally superior (https://arxiv.org/abs/2406.19380)

A4: Thank you for pointing out this perspective. Based on our initial investigation using the TabRed benchmark, we agree that ModernNCA may lose its superiority when there is a significant distribution shift between training and test sets. This limitation arises because the prediction strategy of ModernNCA (as well as TabR) heavily relies on identifying neighbors of the test instance in the embedding space. Distributional changes can compromise the effectiveness of neighbor retrieval, making it challenging to maintain robust performance.

That said, we would like to emphasize that addressing distributional shifts and improving out-of-distribution (OOD) generalization remains an open problem not only for retrieval-based models but for all kinds of machine learning methods that rely on empirical risk minimization. We plan to investigate approaches to enhance neighbor search under such conditions in the future, such as incorporating domain adaptation techniques or robust embedding learning strategies. We will also discuss this limitation and its implications in the final version of the paper.

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

Q1: Additional relative improvement compared to a strong baseline (e.g. a well-tuned MLP) would provide useful additional signal besides the average rank metric (e.g. what is the scale of such improvements)

A1: Thank you for the suggestion. In addition to the average rank (Figure 1), critical difference diagrams (Figure 2), and significant t-tests (Table 1), we further evaluate ModernNCA using two additional criteria to provide a clearer picture of its effectiveness:

• Shifted Geometric Mean (SGM) classification Error and Shifted Geometric Mean (SGM) nRMSE for classification and regression respectively, as proposed in [A]. Here, nRMSE normalizes RMSE by dividing it by the standard deviation of the target values, making it more interpretable across datasets.

$\mathrm{SGM}_{\varepsilon}:=\exp\left(\sum\limits _ {i=1}^{N _ {\mathrm{datasets}}}\frac{w _ {i}}{N _ {\mathrm{splits}}} \sum\limits _ {j=1}^{N _ {\mathrm{splits}}}\log(\mathrm{err} _ {ij}+\varepsilon)\right).$

The results are shown below. ModernNCA achieves the best SGM values (the lower, the better) among all methods. For nRMSE, ModernNCA performs best among deep tabular methods and is only slightly outperformed by CatBoost and XGBoost in regression cases across 300 datasets.

• Relative improvements over a strong baseline (MLP): We compute the relative improvement compared to a well-tuned MLP as proposed in [B]. We use the relative performance -1 as the final score as in [B].

The results are listed below. ModernNCA achieves the highest relative improvement for classification tasks compared to other methods. For regression tasks, ModernNCA demonstrates significant relative improvements, performing better than other deep tabular methods and second only to CatBoost and XGBoost.

[A] Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data. NeurIPS 2024.

[B] TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling. CoRR 2024.

Q2: Providing raw unaggregated results for the core (or even for all) the experiments is very usefull for quick sanity-checks and comparisons in future work. So that others can assess results on the individual datasets by consulting the paper text.

A2: Thank you for the suggestion. The individual results for all datasets are provided in Table 10 and Table 11 in the appendix of the paper. These tables include raw, unaggregated results for both classification and regression tasks, allowing for detailed comparisons and facilitating reproducibility in future work.

The latest TabReD paper, showing that Modern NCA doesn't outperform MLP on its dataset, as shown in Table 3 and Figure1 in paper "TABRED: ANALYZING PITFALLS AND FILLING THE GAPS IN TABULAR DEEP LEARNING BENCHMARKS".

Table 3:

MLP rank 5

ModernNCA rank 5.6

Figure 1:

relative performance percentage change over MLP is -1.05%

Thanks for the valuable feedback. In this rebuttal, we address the reviewers' suggestions and concerns in a Q&A format:

Q1: [Datasets] In light of the recent studies about tabular datasets, it is unclear how representative the benchmark is.

A1: We evaluate ModernNCA on a large-scale benchmark containing 300 datasets with sizes ranging from 1,000 to 1 million instances. This benchmark covers a diverse range of tasks, including binary classification, multi-class classification, and regression. Additionally, the datasets span various domains such as education, physics, and finance. Given the extensive scope of these datasets, we believe the improvements demonstrated by ModernNCA over other methods are robust and representative.

Thank you for pointing out recent studies on tabular datasets. We will address these in the final version of the paper. Specifically:

• [A] Highlights properties such as dataset count and age in evaluations. Our benchmark includes 300 datasets, exceeding the dataset counts in related studies, making our results more comprehensive.

• [B] adopts a data-centric perspective to evaluate tabular datasets using 10 Kaggle datasets and emphasizes the role of feature preprocessing and engineering. We believe our study complements [B] by using a much broader dataset range while incorporating similar preprocessing techniques.

• [C] focuses on datasets with latent temporal characteristics, where training and test splits have different distributions based on time. While this is not explicitly addressed in our current work, we consider this an interesting future research and extension of ModernNCA.

Q2: [Metrics] The metrics such as ranks or wins do not show the scale of performance gaps between methods. It is unclear how significant are the wins and losses of ModernNCA (not from statistical perspective, but from the practical perspective).

A2: In the paper, we evaluate performance using several metrics to provide a comprehensive comparison:

1. Average Rank (Figure 1): This metric summarizes relative performance across datasets, making it easy to interpret which method consistently outperforms others.
2. Critical Difference Diagrams (Figure 2): These are based on the Wilcoxon-Holm test with a significance level of 0.05, helping to visualize statistical differences among methods.
3. Significant $t$-Test (Table 1): This test identifies statistically significant wins or losses between ModernNCA and comparison methods at a 95% confidence interval.

Regarding practical significance, win/loss ratios provide an intuitive and unbiased perspective on how often a method outperforms others across diverse datasets. These ratios are effectively “normalized” and easy to understand, offering insight into which method is a better choice on average. While performance scales (e.g., accuracy differences or RMSE gaps) could be directly analyzed, they vary significantly across datasets with different characteristics. Normalizing such metrics would make them harder to interpret, potentially obscuring practical insights.

Additionally, please refer to A1 for Reviewer Cjck, where we include additional metrics, such as relative comparisons with the strong baseline MLP and alternative metrics like the Shifted Geometric Mean (SGM) classification error for classification and Shifted geometric mean of normalized RMSE for regression. These provide further context on performance gaps and practical significance.

In summary, our metrics are designed to balance statistical rigor with practical insights, ensuring the results are interpretable and actionable for selecting the most suitable algorithm on average.

Q3: [Presentation] The proposed NCA extensions are not conceptually novel, so I believe that the story on the first six pages could be more compact. Perhaps, some of the details and discussion can be moved to appendix.

A3: Thank you for the suggestion. We agree that compactness can enhance readability, and we will consider moving some details from the first six pages to the appendix in the final version. We will carefully refine the presentation to balance conciseness with the importance of preserving the narrative and technical contributions that highlight the challenges and solutions involved in making NCA perform effectively on modern tabular tasks.

That said, we would like to emphasize the value of the detailed exploration in the main text. Over the past two decades, despite the potential of NCA, these extensions have not been successfully applied to tabular datasets. Our paper demonstrates how vanilla NCA can be significantly enhanced with modern techniques, achieving strong results across diverse datasets. This step-by-step approach not only validates the practical utility of each extension but also provides insights that are valuable to the research community.