Thank you for the detailed review. We reran ablations and robustness checks; answers are below.

Responses to Questions

Q1. Low Performance of GPQA and MuSR

Why are we seeing negative correlation by LRMF on the GPQA benchmark across both experiments?...

• Further analysis of model performance on GPQA and MuSR revealed that, for many models, these benchmarks were excessively difficult, resulting in accuracy on multiple-choice questions that was close to random chance.
• This substantial difficulty hindered clear differentiation between models on these benchmarks.
• Additional experiments with five different train/test splits showed extremely high variance in correlation coefficients for GPQA and MuSR, suggesting instability in predictions.
• We attribute this instability to the excessive difficulty of these benchmarks, where irrelevant data from other models contributes noise instead of useful signals. |Method|all|bbh|gpqa|ifeval|math|mmlu-pro|musr| |-|-|-|-|-|-|-|-| |LRMF |0.587(±0.032)|0.644(±0.021)|−0.110(±0.135)|0.406(±0.018)|0.536(±0.020)|0.653(±0.015)|0.094(±0.194)| |MLA |0.414(±0.014)|0.396(±0.017)|0.261(±0.106)|0.283(±0.026)|0.421(±0.034)|0.405(±0.033)|0.296(±0.058)| |NCF_F |0.242(±0.065)|0.295(±0.054)|0.137(±0.056)|0.082(±0.061)|0.190(±0.036)|0.283(±0.062)|0.104(±0.077)|

Q2. Factor Selection

Could you tell us more about the specific factors used in the NCF baseline and how you chose them?...

• We used only the factors consistently available across all models in the Hugging Face model card JSON: “architecture type,” “model type,” and “parameter size.” We didn’t include factors like “dataset size” or “batch size” because they weren’t reliably present. In the homogeneous setting, such missing factors would likely be similar across models anyway, so excluding them has little impact.

Q3. Sensitivity of A^(X)

I'm wondering how sensitive the LRMF results are to how you constructed the instance similarity graph A^(X). Have you experimented with different embedding models for prompts or different values of k when defining neighbors?

• We compared several widely used embedding models, including Snowflake/snowflake-arctic-embed-l-v2.0, infloat/e5-mistral-7b-instruct, and all-mpnet-base-v2, and found that the choice of embedding model had almost no diffrence on the results. We attribute this to the fact that the essential requirement is to connect top-k tasks with rough similarity; any model capable of capturing coarse-grained similarity is sufficient for this purpose.
• For the parameter k, we experimented with values ranging from 2 to 100, selecting the value that yielded the best AUC-ROC on the validation set. However, differences in performance across different k values were minimal. |embedding model|k=2|5|10|20|50|100| |-|-|-|-|-|-|-| |Snowflake/snowflake-artic-embed-l-v2.0|0.57|0.57|0.57|0.57|0.57|0.57| |infloat/e5-mistral-7b-instruct|0.57|0.57|0.57|0.57|0.57|0.57| |all-mpnet-base-v2|0.57|0.57|0.57|0.57|0.57|0.57|

Other Comments

The quality of lineage information feels like a potential weak spot ... how missing or incorrect lineage data might impact results.

The reliability of lineage metadata extracted from Hugging Face is indeed a critical issue, as this metadata often lacks completeness—for instance, merged models frequently omit details about their base models. To quantitatively assess the impact of noise/incompleteness of lineage data, we performed additional experiments by randomly modifying lineage data (adding or removing lineage links). Notably, we observed distinct responses between the two methods:

• LRMF:
  • Robust to incomplete (missing) lineage data (40% removal led to only a ~10% decrease in correlation).
  • Highly sensitive to incorrect lineage additions (40% addition caused a ~50% decrease in correlation).
  • Implication: For LRMF, accuracy is better preserved by omitting uncertain lineage information rather than risking incorrect additions.
• MLA:
  • Demonstrated robustness against both random additions and removals of lineage data.
  • The resilience likely stems from random lineage connections averaging out model-specific variations, pulling results toward a global mean.

Detailed results from these experiments are available here.

Absolute Performance Metrics

The evaluation focuses on relative performance (using Pearson correlation) to rank models correctly. While this makes sense, I think some discussion on predicting absolute performance scores would be valuable for certain applications.

• We also used AUC-ROC as an absolute performance metric for binary predictions (solvable or not) and observed high scores (0.8–0.9) across all methods.
• However, high AUC-ROC can occur even when true correlation is zero, especially in homogeneous settings. Therefore, we report correlation coefficients as the primary evaluation metric in the main paper.

We would like to express our sincere gratitude for your careful review and insightful comments on our research. The feedback from the reviewers has significantly improved the quality of our paper. Based on your suggestions, we have conducted several additional experiments, which we invite you to examine along with our responses.

What if there is noise in the model lineage?

The paper demonstrates incorporating lineage, it doesn't extensively analyze the sensitivity of its LRMF model to varying degrees of noise or incompleteness in this foundational lineage data. A critical aspect would be understanding at what threshold of data inaccuracy the benefits of lineage regularization begin to diminish or even become detrimental. Furthermore, over smaller subsets of data this effect might be more pronounced.

To quantitatively assess the impact of noise/incompleteness of lineage data, we performed additional experiments by randomly modifying lineage data (adding or removing lineage links). Notably, we observed distinct responses between the two methods:

• LRMF:
  • Robust to incomplete (missing) lineage data (40% removal led to only a ~10% decrease in correlation).
  • Highly sensitive to incorrect lineage additions (40% addition caused a ~50% decrease in correlation).
  • Implication: For LRMF, accuracy is better preserved by omitting uncertain lineage information rather than risking incorrect additions.
• MLA:
  • Demonstrated robustness against both random additions and removals of lineage data.
  • The resilience likely stems from random lineage connections averaging out model-specific variations, pulling results toward a global mean.

Detailed results from these experiments are available here.

LRMF vs. MLA

For both homogeneous and heterogeneous cases how does one decide to use model lineage averaging or Lineage-Regularized Matrix Factorization? The paper does not discuss in depth about the failure nodes and provide insights into the emperical analysis.

• LRMF generally achieves superior performance compared to MLA across most evaluation settings.
• We recommend selecting the method with the best validation performance, as no inconsistencies were observed between validation and test rankings.
• Based on further experiments above, MLA is preferable in scenarios where lineage information contains a large number of unreliable entries.

Low Performance on GPQA and MuSR

Further, the paper still needs to shed light on the poor performance of GPQA

• Further analysis of model performance on GPQA and MuSR revealed that, for many models, these benchmarks were excessively difficult, resulting in accuracy on multiple-choice questions that was close to random chance.
• This substantial difficulty hindered clear differentiation between models on these benchmarks.
• Additional experiments with five different train/test splits showed extremely high variance in correlation coefficients for GPQA and MuSR, suggesting instability in predictions.
• We attribute this instability to the excessive difficulty of these benchmarks, where irrelevant data from other models contributes noise instead of useful signals.

We would like to express our sincere gratitude for your careful review and insightful comments on our research. The feedback from the reviewers has significantly improved the quality of our paper.

Adding New Models in Our Method

When a new model is to be evaluated, does the entire optimization problem have to be re-run or is there a more efficient approximation that can be done? How stable are the obtained model embeddings? When adding a new model or several models for example, how different will the results be?

This is a situation that frequently occurs in the large-scale recommendation domain, and it can be optimized at low cost using various methods. For example, a popular approach is to warm-start the optimization using the previously obtained feature vectors as initial values. This enables obtaining solutions much faster than cold-start. In that case, we can also expect the embedding variations to be minimized.

Comparison with Existing Routing Methods

The comparison is done only with one previous work that relies on scaling laws. Comparing to another performance prediction technique, e.g. from the model routing works, would help highlight the importance of the model lineage information and could increase the impact of the paper.

We are tackling a new problem: 'When creating a new model that can effectively solve a certain problem through fine-tuning or merging, which model should be used as the base?' Our goal is to estimate the performance superiority of such unknown new models. This differs from the problem setting of 'Routing (determining which existing model should be used to solve a new problem),' so, to the best of our knowledge, no existing method proposed for Routing can be directly applied. Routing assumes that candidate models are already given and that these candidate models have already been evaluated on several samples, whereas we are estimating the performance of unknown models from cold start.

However, the kNN Routing concept proposed by Shinitzer et al. (2024) can be applied and utilized with an extension that incorporates Lineage. While they use the average of scores for top-k similar tasks (= Task Lineage Average), Model Lineage Average is the application of this concept in the model direction. MLA underperforms LRMF in most cases but shows good performance in some cases. We consider this result to indicate that methods that worked well in Routing were also effective in some settings.

Questions to Authors

Are the 1,000 random samples the same for all models or are they randomly selected for each model? What is the resulting size of the instance matrix?

• The same 1,000 random samples are consistently used across all models, ensuring fair comparison. The resulting instance matrix size is [number of models] × 1,000.
• In Appendix C’s setting (each model evaluated only on t unique samples), the matrix size remains the same but becomes sparse; LRMF still achieves superior performance.

Thank you for the response to my questions and concerns, and for clarifying the differences and parallels between the focus of the paper and model routing. I have raised my score since I believe that these clarifications and the additions in the responses to the other reviewers are very helpful to better understand the contribution and power of the proposed approach.

Thank you for the detailed review. We reran ablations and robustness checks; answers are below.

Limited Baselines

The paper currently uses one baseline method (NCF with factors) ...

• We address the problem of predicting model performance before any post-training steps such as fine-tuning or model merging. This can be viewed as a model-side cold-start problem.
• Polo et al. (2024) and Kipnis et al. (2025) cannot be used under this problem settings, because they require the target model to have been evaluated on some samples.
• Similarly, scaling law methods (e.g., Ruan et al., 2024) are not suitable for performance estimation in the context of post-training data or model merging, since they extrapolate performance from factors like parameter count or pretraining data volume, which are essentially unchanged in these scenarios.

Regularization Term Ablation

The proposed method LRMF uses both “lineage regularization” and “task regularization”, ... I would expect an ablation study on the two loss terms to show that.

• Most correlation improvement is due to the Model Lineage regularization term.
• Appendix B.1 (Fig. 4a/4b) shows AUC-ROC ablation for each regularization term.
• In response to feedback, we expanded the ablation study to more comprehensively explore and visualize regularization effects using correlation coefficients. (See new figure at here )
• The case λ_M=0,λ_X=0 (i.e., NCF/MF) fails in cold-start (correlation = 0).
• Model lineage (λ_M > 0) enables strong performance even without task lineage (λ_X = 0).

Evaluation Metrics and Consistent Reporting

An improvement of “up to 7-10 percentage points” is described in the abstract and conclusion. ... It’s confusing how the number of 7-10 was derived.

• We use two metrics: Pearson correlation (for model ranking) and AUC-ROC (for instance-level accuracy).
• The “7–10 percentage points” originally referred to AUC-ROC improvement.
• For consistency, we will instead report Pearson correlation improvement: “up to a 25-point increase.”

Cold-Start Problem

The paper claims that the method addresses the cold-start problem in the abstract and Line 251, however I only find the relevant results in appendix C...

• All experiments address the cold-start problem: predicting test-model performance with no evaluation data based solely on training-model evaluation results and lineage information.
• Appendix C shows results under a stricter setting where training-model evaluation data are limited (5–500 samples per model). Even then, LRMF outperforms MLA and NCF with factors in nearly all cases.

Responses to the Questions

Line 302: Clarification of Metrics

Line 302: Clarification question about evaluation metrics. Did you predict instance-level accuracy (i.e., either 0 or 1), ...

• That is correct. We first predict instance-level accuracy (0/1), then compute average scores at the overall or benchmark level and calculate correlations. Thus, the constrained matrix factorization is performed once on the entire set of benchmarks.

Line 314: Train/Test Split

Line 314: Is there one fixed train/test split or do you sample multiple times?

• In response to feedback, we conducted five different samplings and computed mean and variance. Trends remained consistent, although score stability varied greatly across benchmarks. For GPQA and MuSR—where LRMF underperforms—variance was about ten times larger than in other benchmarks. We found these benchmarks are very difficult for most models, with performance close to random; including other model scores in such cases adds noise and degrades prediction.

Line 341: Factor Diversity

Line 341: I don’t quite understand the reasoning of “this limited factor diversity explains …”...

• NCF with factors relies only on whether a model is FT, merged, or base. Information about which data or learning method was used is all collapsed to “trained,” so on instruction-following benchmarks like IFEval—where data or method differences strongly affect scores—NCF cannot capture feature differences, resulting in lower predictive performance.

Line 393: Weighted Connections

Line 393: I’m curious about your thoughts on how to incorporate weighted connections...

• We can represent weighted edges by using merge ratios for model merging or fine-tuning token counts as weights. This can be implemented by using a soft adjacency matrix A.
• The graph kernel in the regularization term already reflects multi-hop relations: if edges exist between A–B and B–C, A and C embeddings become closer indirectly.