ON EXTRAPOLATION IN MATERIAL PROPERTY REGRESSION

We thank the reviewer for the rigorous reviews. We have carefully and thoroughly addressed the concerns and sincerely hope the reviewer will consider raising the evaluation score.

Response to weaknesses

1.1: About the problem scope.

A: This work focuses on the critical challenge of extrapolation in MPR—predicting property values beyond the training data, which is essential for AI-powered material discovery with novel properties. The i.i.d tests suggested by the reviewer has been well-studied and is not the focus of this paper.

1.2: About the practicality of the proposed method.

A: In Section 4.4, we have demonstrated that our method can effectively recall materials with novel properties, highlighting its practical value for material discovery.

2.1:About method novelty

A: We would like to emphasize that novelty is not limited to technical advancements but can take various forms. In our work, we reframe MPR as a material-property matching task specifically tailored for extrapolation, introducing a novel application of matching techniques to this critical challenge and offering fresh insights to the field of MPR.

2.2: About the rationality of our method for addressing extrapolation.

A: While the proposed matching optimizations focus on modeling the relationship between x and y within the training set, this allows the model to learn the compatibility between x and y, enabling it to infer the compatibility between unseen x and unseen y. This neural network capability to generalize beyond the training data is not unique to our approach but rather reflects a broader principle observed in related domains, such as zero-shot learning (where the model is trained solely on seen images and classes, then tested on unseen images and classes) and information retrieval (where the model is trained solely on seen queries and documents, then tested on unseen queries and documents).

2.3: the algorithm "may" achieve high performance if there is significant variance in the value encoder, hyperparameters, and other factors.

A: As shown in Table 2 and 3, our method outperforms existing methods under a wide range of datasets, backbones and metrics, demonstrating its robustness. Although we conducted hyperparameter searches, these were based on performance on the validation set, not the test set. For fairness, we also report the results of hyperparameter searches for the baselines below (in 3.3), providing additional evidence of our method’s superiority. If the reviewer has specific suggestions for additional ablation studies, we would be glad to conduct further experiments.

3.1: It is unclear why the target value range is the same in both the test and validation environments.

A: As provided in Section 4.1, we clarify that the target value ranges of the validation and test sets are indeed disjoint across all seven tasks. In Table 1, we merged the validation and test ranges for brevity, which may have caused your misunderstanding. We have uploaded a newer revision, which displays the label ranges for training, validation, and test sets separately in Table 1.

3.2: About the lower and upper bounds of the label range.

A: To address the reviewer's concern, we extend the bounds from dataset-specific values to broader ones, namely [-10, 10] for all datasets, which can encompass the theoretical range for most materials-related properties. As shown in the following table, even with these broader bounds, our method maintains superior performance, demonstrating its robustness to bound settings.

3.3: About the hyperparameter search space for the comparative methods.

A: We followed the reviewer's suggestion and conducted hyperparameter searches for the baselines. As shown in the table below, our method still achieves SOTA performance in 12 out of 14 comparisons. Specifically, we performed task-agnostic hyperparameter searches, such as learning rate optimization, as detailed in Section 4.2. For task-specific hyperparameters, we identified that Ranksim, ConR, C-Mixup, and FOMA include key settings that significantly influence performance. We adhered to the hyperparameter search protocols outlined in their respective papers and reported the best MAEs in the table.

Response to questions

Q1: Can the author demonstrate the effectiveness of this algorithm on established imbalanced regression benchmarks or in environments where labels are sparsely shared between the training and evaluation sets?

A: We would like to emphasize that our benchmark, although challenging, is significant for material discovery. To solve the reviewer's concern, we construct a setting where training labels overlap slightly with the target. Specifically, we randomly chose 10 samples from the validation and test set separately and moved them to the training set, forming an imbalanced regression scenario. We compared our method and SOTA baseline methods in the following table. Both methods exhibit a performance improvement compared to the original setting and our method is still effective in most tasks.

Q2: how the method intuitively aligns with the concept of extrapolation.

A: - This question has been addressed in weakness 2.2.

Dear Reviewer N2Xe,

Thank you once again for your valuable and constructive feedback on our submission. As the discussion period progresses toward its end, we kindly request any additional comments or thoughts you may have regarding the clarifications and results we have provided in our rebuttal. We fully understand that this is a busy time, and we sincerely appreciate the effort and time you have dedicated to reviewing our work. If there are any remaining questions or concerns, we would be happy to address them promptly.

Best regards, The Authors

Dear Reviewer N2Xe,

Thanks again for your insightful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need. In our previous response, we have carefully studied your comments and made detailed responses summarized below:

• Provide further explanation of the novelty, practicality and rationality of our method.
• Clarify the label range of the validation set to ensure experiment fairness.
• Conduct additional experiments (dataset-agnostic label interval for MEX & hyperparameter search for baselines) to ensure fairness of the comparisons.
• Perform additional experiments to demonstrate the effectiveness of our method on imbalanced MPR.

We hope that the explanation of our method and the provided new experiments have convinced you of the contributions of our submission.

Please do not hesitate to contact us if there's additional clarification or experiments we can offer. Thanks!

Thank you for your time!

Best, Authors

Thank you for the detailed explanations provided by the authors. Below, I have outlined the remaining concerns. In summary, I believe that the proposed MPR with extrapolation (MPRE) problem could serve as a valuable contribution to the community, provided that it is both (1) practical and (2) solvable.

1. Practicality (about (1))

The authors claim that solving the MPRE problem can aid in material discovery. However, the experiment in Section 4.4 resembles an anomaly detection task and does not report precision metrics. Do the authors have examples of applications where the proposed method can be used outside the MPRE context?
If the authors can provide such examples, the proposed MPRE problem has the potential to serve as a benchmark dataset. Without this, the concern remains that the problem may be too narrowly defined.

2. Solvability (about (1) and (2))

I am curious whether the proposed MPRE problem is indeed solvable. While I appreciate the experiments where the authors added 10 samples to the training data, this does not fully address my concerns. Even with just 10 additional samples, the MAE values decreased significantly. As mentioned in my initial review, the reported MAE values appear high relative to the target value. Are the authors confident that MPRE is a solvable problem? Additionally, does the proposed setting (using 70% of the training data) have physical or scientific justification?

3. Fairness (about (2))

Could the authors provide more details about the validation setting? Specifically, I am curious whether the model is retrained on a dataset that includes the validation data after hyperparameter tuning.

4. Method Effectiveness (about (2))

I understand that the label encoder approach proposed by the authors is related to extrapolation. However, it is well-known that over-parameterized deep learning models often lead to overfitting. Additionally, the NCE loss proposed by the authors appears to focus on reducing confidence around the labels, making the model more robust to noise (as evidenced by improved extrapolation performance with larger lambda values). Have the authors considered methods to mitigate overfitting within the target range of the training data, particularly to ensure effective extrapolation? I suspect the validation setting plays a crucial role here, and I am curious whether early stopping based on validation performance is a key factor in the effectiveness of the proposed approach.

Thank you for your insightful feedback. We have provided detailed explanations for each question below and hope these address your concerns.

Q1.1: about the precision metric.

A: The detection task in Section 4.4 evaluates whether an extrapolative sample (label 1) is correctly classified as extrapolative (label 1) or not (label 0). In the test set, all samples are extrapolative (label 1), meaning there are no false positives. As a result, the precision is always 1 for all methods in this setting, rendering it uninformative for comparison.

Q1.2: about more applications of MEX

A: First, while our focus is on extrapolation related to material properties, we do not consider this a narrowly defined problem. Materials are fundamental to the development of human society and are deeply interconnected with critical domains such as energy and the environment. For instance, breakthroughs in discovering innovative catalysts for renewable energy storage could significantly advance efforts to combat climate change. Moreover, we believe that other types of scientific data [1,2], including small-molecule drugs, proteins, and genetic information, also face similar extrapolation challenges. In the future, we will explore more possibilities of MEX in these areas.

Q2.1: about the solvability of the problem.

A: We firmly believe that MPRE is a theoretically solvable problem. This is because the mapping between a material’s structure and its properties is governed by fundamental physical laws (e.g., quantum mechanics at the atomic scale). Regardless of the material distribution, these physical principles remain invariant. Therefore, the better a model is at capturing these robust principles, the better its extrapolation performance will be. Moreover, recent work [3] in machine learning has shown theoretically that encoding proper non-linearities can enable effective extrapolation, further supporting the feasibility of solving MPRE.

Q2.2: about the data splitting.

A: The 70%-15%-15% split for training, validation, and testing is a widely adopted practice in the machine learning community, and material property prediction tasks are no exception [4].

Q3.1: about the validation setting.

A: Details of the validation setting are explicitly provided in Section 4.1 and Table 1. The validation set is entirely disjoint from both the training and test sets, specifically designed to mimic extrapolation scenarios during training. This design aligns with standard practices in areas like domain generalization (where train/val/test lie in different domains) and zero-shot learning (where train/val/test involve disjoint classes). If the reviewer has specific concerns, we would be happy to address them.

Q3.2: training with validation data.

A: We strictly adhere to the standard train-validation-test pipeline. The model is never retrained on datasets that include validation data.

Q4.1: about overfitting.

A: Experimental results presented in Section 4.3 and Section 4.4 indicate that our methods exhibit minimal overfitting within the training label range. For example, as discussed in Section 4.4, MEX is less likely to mistake extrapolative labels as within the training label interval. While our method is an initial attempt to address the MPRE challenge, we believe it opens the door for future improvements in this area.

Q4.2: about early stopping.

A: Early stopping based on validation performance is standard practice in machine learning to avoid overfitting, and we consistently applied this technique across all methods during our experiments.

References

[1] Chan, Alvin, et al. "Deep extrapolation for attribute-enhanced generation."

[2] Padmakumar, Vishakh, et al. "Extrapolative Controlled Sequence Generation via Iterative Refinement."

[3] Xu, Keyulu, et al. "How neural networks extrapolate: From feedforward to graph neural networks."

[4] Chang, Rees, et al. "Towards Overcoming Data Scarcity in Materials Science: Unifying Models and Datasets with a Mixture of Experts Framework."

For Q1.1

The authors should also evaluate the algorithm on in-range samples to measure precision.

For Q1.2

The authors have designed the MPRE problem under the assumption of an extreme extrapolation scenario. The reason I argue that the problem is narrowly defined is that the authors do not consider regression for in-range samples at all (This concern was partially resolved by the additional exp.). As stated in the authors' response to Q2.1, if immutable physical principles exist and it is possible to learn these principles from in-range samples, there is no reason not to test the model on in-range samples. The problem proposed by the authors is both interesting and necessary; however, as the authors are introducing a new benchmark, it is essential to provide robust justification for its validity.

For Q2.1 & Q2.2

Unfortunately, the paper [4] does not partition the data based on the target value. Contrary to the authors' opinion, I am not convinced that deep learning models can reliably learn physical rules invariant to the range of target values (referred to by the authors as the material distribution). Indeed, the additional experimental results provided by the authors in their rebuttal also demonstrate a sharp improvement in model performance when additional samples are utilized.

For Q3

The proposed method appears to have a higher dependency on the validation set compared to other approaches. This suggests that the presence or absence of a validation set (for hyperparameter tuning) could have a significant impact on the performance of the method. The authors could identify the hyperparameters using the validation set and then retrain the model on a dataset that includes the validation set using the identified hyperparameters (see [R-1]). Alternatively, the authors could experimentally demonstrate that the validation set plays a minimal role in the proposed method.

For Q4

Is it correct that the proposed method relies entirely on early stopping as a tool to prevent overfitting? If so, the effectiveness of early stopping should also be thoroughly discussed in the paper.

[R-1] Erik Englesson, et al. "Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels"

We appreciate the reviewer for acknowledging our studied problem as significant and our perspective of solving extrapolation as novel. We sincerely believe that we have effectively addressed the reviewer’s main concerns and hope that the reviewer may consider raising the evaluation score to reflect the improvements made in our paper.

Response to weaknesses

1: About method novelty.

A: We would like to emphasize that novelty is not limited to technical advancements but can take various forms. In our work, we reframe MPR as a material-property matching task specifically tailored for extrapolation, introducing a novel application of matching techniques to this critical challenge and offering fresh insights to the field of MPR.

2: scalability and computational complexity.

A: First, in Section 4.5, we delve into the runtime performance of the MEX framework, illustrating that its computational efficiency is comparable to conventional regression techniques. Additionally, we demonstrate that the computational expense can be further reduced without sacrificing performance by tuning two key hyperparameters: the number of candidate labels (C) and the number of optimization iterations (T). Taking the Formation Energy task as a case study, we adjusted C to 500 and T to 3, reducing the processing time to 0.0031 seconds per sample, which is on par with the baseline LDS.

3.1: Traditional methods, like DFT, are supposed to be compared to find the gap between DL-based extrapolation and traditional methods.

A: DFT is not directly compared here as as the dataset was annotated using it.

3.2: more DL-based methods should be compared

A: As extrapolation remains an underexplored area, there are currently few established methods. Although earlier studies [1,2,3] examined extrapolation, they were limited to simplified experiments without practical methodologies. To the best of our knowledge, DIR methods could serve as competitive baselines. For instance, LDS and Ranksim, included in our experiments, claim effectiveness in this domain and demonstrate strong performance on disjoint label ranges. If the reviewer has additional suggestions for suitable baselines, we will make every effort to include them and provide experimental results.

Response to questions

Q1: About the diversity of the dataset.

A: The selected datasets are representative of MPR benchmarks and are frequently employed by other works, such as Matbench [4]. We have uploaded a revision, where the distribution of atom numbers and lattices and more characteristics for each dataset are provided in Appendix A.1.

Q2: are there any insights into adapting Noise Contrastive Estimation (NCE) instead of other methods?

A: The proposed matching optimizations focus on modeling the relationship between x and y within the training set, this allows the model to learn the compatibility between x and y, enabling it to infer the compatibility between unseen x and unseen y. We chose to adapt NCE due to its simplicity and its ability to achieve these goals effectively. Nevertheless, we believe that other methods that can fulfill these objectives are also applicable.

Reference

[1] Xu, Keyulu, et al. "How neural networks extrapolate: From feedforward to graph neural networks."

[2] Florence, Pete, et al. "Implicit behavioral cloning."

[3] Shen, Xinwei, and Nicolai Meinshausen. "Engression: Extrapolation for nonlinear regression?."

[4] Dunn, Alexander, et al. "Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm."

Dear Reviewer JB71,

Thank you once again for your valuable and constructive feedback on our submission. As the discussion period progresses toward its end, we kindly request any additional comments or thoughts you may have regarding the clarifications and results we have provided in our rebuttal. We fully understand that this is a busy time, and we sincerely appreciate the effort and time you have dedicated to reviewing our work. If there are any remaining questions or concerns, we would be happy to address them promptly.

Best regards, The Authors

Dear Reviewer JB71,

Thanks again for your insightful suggestions and comments. Since the deadline of discussion is approaching, we are happy to provide any additional clarification that you may need. In our previous response, we have carefully studied your comments and made detailed responses summarized below:

• Provide further explanation of the novelty and rationality of our method.
• Perform additional experiments to demonstrate the computational complexity of our method.
• Explain the choice of baseline methods and datasets.

We hope that the explanation of our method and the provided new experiments have convinced you of the contributions of our submission.

Please do not hesitate to contact us if there's additional clarification or experiments we can offer. Thanks!

Thank you for your time!

Best, Authors

We thank the reviewer for the detailed and thoughtful review comments. We have addressed the concerns raised and humbly hope that the reviewer will consider increasing the evaluation score.

Response to weaknesses

1. Dataset Limitations

Dataset choice: While our dataset size may seem small compared to large-scale CV and NLP datasets, it is standard for MPR benchmarks. As we stated in Section 4.1, we curate our datasets from Matminer, one of the most widely used tools for materials data, with over 750+ citations.

Dimensionality of data points: In MPR, material samples are not represented as continuous vectors like image or word embeddings. Instead, a sample material with $n$ atoms is represented by $(h_i,x_i)_{i=1}^n$, where $h_i\in\mathbb{N}^{+}$ denotes the atom type and $x_i\in\mathbb{R}^3$ represents the Cartesian coordinates of the $i$-th atom. The materials in our dataset are crystal, so they have additional information such as the lattice matrix ($3\times 3$ matrix capturing the crystal's fundamental translation symmetry). This raw information will be transformed into a graph for further processing. We have uploaded a revision, where more characteristics for each dataset are provided in Appendix A.1, to further clarify the details of our dataset.

Design of $y_{target}$: The decision to separate training and target data fully aligns with our objective of evaluating true extrapolation capabilities. Extrapolation remains a significant challenge for neural networks, and prior studies [1,2,3] have explored this issue with the same disjoint data configurations, though typically for simpler data structures. In materials science, extrapolation is particularly crucial, as researchers aim to discover materials with properties beyond those observed to date, driving advancements in technology and innovation.

2. Baseline Choice

While DIR methods are not specifically designed for extrapolation, our baseline LDS and Ranksim claim their effectiveness in this area and report performance on disjoint label ranges. To date, as extrapolation is an underexplored challenge in deep learning, we believe that DIR remains one of the most viable approaches for addressing extrapolation in the absence of purpose-built methods. If the reviewer has suggestions for additional suitable baselines, we will try our best to provide experimental results.

Response to questions

Q1: clarify the Geometric Mean metric

A: This metric was first introduced in [4] and is defined as $(\Pi_{i=1}^N e_i)^{1/N}$, where $e_i:=|y_i - \hat{y}_i|$ represents the L1 error of each sample. As described in [4], "GM aims to characterize the fairness (uniformity) of model predictions using the geometric mean instead of the arithmetic mean over the prediction errors." This metric is widely used in DIR research.

Q2: the performance seems to decrease as $\lambda$ increases, suggesting that NCE might negatively impact results

A: We would like to first clarify that $\lambda$ represents the weight of the absolute matching relationship $L_{abs}$ rather than the relative matching relationship $L_{NCE}$. As $\lambda$ increases, $L_{abs}$ may dominate the optimization process, causing the model to struggle in distinguishing between negative and positive values, which can affect performance.

Q3: how does this method handle out-of-distribution data or outliers?

A: Our matching-based approach focuses on aligning the sample and label feature spaces, making it less dependent on the sample distribution. Even when presented with an outlier sample, our method can still effectively optimize the model by aligning the features of the sample with those of its corresponding label.

Q4: Could the authors consider an additional setting where the target and training data overlap slightly?

A: We follow the reviewer's suggestion and consider a setting where training labels overlap slightly with the target. Specifically, we randomly chose 10 samples from the validation and test set separately and moved them to the training set, forming a DIR-like scenario. We compared our method and SOTA baseline methods in the following table. We can observe that (1) both methods exhibit a performance improvement compared to the original setting and (2) our method is still superior to SOTA baselines in most tasks.

Q5: How well does the method scale with higher-dimensional targets?

A: Theoretically, our method can be extended to higher-dimensional targets with minimal modifications for negative target sampling. A straightforward approach would be to scale the one-dimensional Gaussian in Eq. 4 to a multi-dimensional Gaussian. However, since material target properties are typically single-dimensional, we leave the exploration of this extension in future work.

Q6: Is there a threshold in the matching function M(x, y) to determine when a match is strong enough?

A: While MEX is inherently threshold-free, users can define a threshold using heuristic rules, such as setting it to the average M(x, y) value from the training set samples.

Q7: How does the model avoid overfitting in the label encoder, potentially transforming it into a look-up table? Does the noise component mitigate this risk, supporting extrapolation?

A: We sincerely thank the suggestion. However, it is non-trivial to implement a look-up table in this context, as such approaches are generally suited for discrete labels, whereas our task involves continuous material properties. The noise component indeed plays a critical role in reinforcing proper matching relationships between labels and an anchored sample, which implicitly maintains inter-label consistency, thereby mitigating the risk of overfitting and supporting robust extrapolation.

Q8: could gradient-based methods be viable for this estimation?

A: Certainly, gradient ascent and other optimization methods could be applied for inference in our framework. We opted for Monte Carlo sampling due to its derivative-free nature, which allows for efficient inference while maintaining strong performance.

Q9: What is the dimensionality of the input data sample x?

A: This question has been addressed in weakness 1.

Reference

[1] Xu, Keyulu, et al. "How neural networks extrapolate: From feedforward to graph neural networks."

[2] Florence, Pete, et al. "Implicit behavioral cloning."

[3] Shen, Xinwei, and Nicolai Meinshausen. "Engression: Extrapolation for nonlinear regression?."

[4] Yang, Yuzhe, et al. "Delving into Deep Imbalanced Regression."

Dear Reviewer oJTR,

Thank you once again for your valuable and constructive feedback on our submission. As the discussion period progresses toward its end, we kindly request any additional comments or thoughts you may have regarding the clarifications and results we have provided in our rebuttal. We fully understand that this is a busy time, and we sincerely appreciate the effort and time you have dedicated to reviewing our work. If there are any remaining questions or concerns, we would be happy to address them promptly.

Best regards, The Authors