Post Hoc Regression Refinement via Pairwise Rankings

We thank the reviewer for their thoughtful and constructive feedback. We address the concerns regarding the reference set’s sampling strategy, distribution alignment, and diversity with new empirical analyses and clarifications below.

Q1: Will a more advanced sampling strategy improve the results?

We agree that the reference set is central to RankRefine's effectiveness. While we used random sampling for simplicity, we now include additional experiments to study the impact of biased sampling. We use synthetic data where the reference and query labels are drawn uniformly within some ranges. We simulate predictions by injecting zero-mean Gaussian noises to the query labels.

As shown in Table 1, we systematically restrict the reference set to increasingly narrow subsets of the query labels' range to simulate non-uniform sampling (e.g., bias toward certain property values). Even under severe reference bias (e.g., 70% range coverage centered away from query), RankRefine maintains strong performance. For example, with 60% ranker accuracy and 70% reference set bias, we still achieve a 12.3% reduction in MAE over the base regressor.

The fact that 0% reference bias achieves the best performance indicates that sampling from the full label range (uniform, unbiased sampling) yields the best performance. However, even clustered references retain strong utility. While more advanced strategies such as adaptive, query-aware sampling (e.g., using the base regressor’s prediction to guide selection) could further improve performance, we leave this as a promising direction for future work.

Table 1. RankRefine performance, measured in β (β = $\frac{MAE_\text{refined}}{MAE_\text{non-refined}}$, lower is better), with non-uniform sampling of the reference set. We restrict the reference set range to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples. Best performance is achieved when the reference set spans the entire range of the ground truth labels, but clustered reference sets still provide substantial regression improvements.

Q2: Would performance degrade if the reference and test distribution differ largely?

To test the impact of distribution shift between reference and query, we created two challenging scenarios where the reference labels and query labels are drawn uniformly from different range values:

• Disjoint distributions: reference labels in [-1, 0] or [1, 2], query labels in [0, 1]. We averaged the results of the two cases, that is, between {ref = [-1, 0], query = [0, 1]} and {ref = [1, 2], query = [0, 1]}.
• Partial overlap: 50% overlap between reference and query label range

In both cases, we fix the reference label range width and vary the ranker accuracy. Despite the mismatch, RankRefine still improves the base regressor when the ranker is moderately accurate (≥65% in the disjoint case, ≥55% in the partial overlap case). These findings suggest that RankRefine is robust to distribution shift that is common in real-world settings with scarce or heterogeneous data.

Table 2. β for disjoint and partially overlapping reference set.

Q3: How does label and scaffold diversity in the reference set affect performance?

Label diversity plays a key role. Table 1 demonstrates that reference sets spanning the full range (0% reference bias) of ground-truth labels consistently achieve the best results, underscoring the importance of covering the ground truth labels spectrum.

We agree that scaffold diversity is also an important factor in molecular applications. However, we did not explicitly control for it in our experiments, as our goal is to present RankRefine as a general-purpose method applicable across domains. Measuring or enforcing scaffold (i.e., design) diversity is specific to molecular data and does not easily transfer to other domains where structural diversity is harder to define. We view scaffold-aware reference selection as an important, domain-specific extension of our work and leave it for future.

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged by the positive remarks on the quality, clarity, and originality of our submission. Below, we address each of the reviewer’s concerns in detail and provide new empirical results to support our claims.

Q1: How bad is a biased regressor? In many practical applications, the regressor may systematically under or overestimate.

We agree that this is an important, practical concern. Our experiments on the TDC ADMET datasets (Figure 3 in the main paper) demonstrate that RankRefine works in real-world scenarios. We have also conducted new experiments where we systematically inject constant bias into an otherwise unbiased regressor. Specifically:

1. We start from synthetic predictions (arbitrary labels injected with random noise) with zero-mean Gaussian noise, yielding a mean absolute error (MAE) close to one standard deviation of the ground truth labels.
2. We then add a constant offset, increasing in magnitude from 0% to 70% of the label standard deviation (SD).
3. We report the refined-to-original MAE ratio (β = $\frac{MAE_\text{refined}}{MAE_\text{non-refined}}$, lower is better) for varying ranker accuracies.

The results (see Table 1) demonstrate that RankRefine remains effective even under substantial regressor bias. For example, with a 60% bias and only 60% ranker accuracy, RankRefine achieves β = 0.9649, i.e., a 3.51% gain.

Table 1. RankRefine performance, measured in β, on a biased regressor. We inject constant bias into an otherwise unbiased regressor, where the magnitude of the offset is measured as a percentage of the ground truth labels standard deviation. RankRefine offers improvement even when the regressor is significantly biased (e.g., 60% of the SD of the ground truth labels)

Q2: Does the LLM simply memorize some of the ranking answers?

This is a valid concern. To test for potential memorization, we used a proprietary compound activity dataset (75 compounds; IC50 labels) collected by our collaborators (experimental biochemists). This dataset is unpublished and therefore not part of any public training corpora, ruling out direct memorization. ChatGPT-4o achieved 60.14% pairwise ranking accuracy on this private dataset. This result indicates generalization and not memorization.

Q3: Have you considered non-uniform sampling of the reference point? Intuitively, references that aren’t clustered together should be better.

We agree with the reviewer’s intuition that a clustered reference set would reduce the performance gains. We evaluated the effect of non-uniformity on synthetic data (similar to Q1) by restricting the reference set range to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples.

As shown in Table 2, RankRefine’s performance remains stable as long as the reference set overlaps with the query label range. Even with severe bias (e.g., 60% range), the method still improves the regressor estimate. For instance, with 60% ranker accuracy and 70% reference bias, RankRefine achieves a 12.30% MAE reduction (β = 0.8770). We also confirm, as mentioned by the reviewer, that RankRefine performs best when the reference set labels are spread across the full range of the ground truth labels and not clustered.

Table 2. RankRefine performance, measured in β, with non-uniform sampling of the reference set. We restrict the reference set range to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples. Best performance is achieved when the reference set spans the entire range of the ground truth labels, but clustered reference sets still provide substantial regression improvements.

Q4: Pairwise rankings could be expensive for a human in practice.

While 20 comparisons per query may seem expensive, we note that many scientific domains already rely on human expert ranking (e.g., drug design). Our method is also compatible with low-cost and parallelizable rankers, like LLMs (which we have shown is not simply memorizing answers) or crowd workers, offering a promising tradeoff between accuracy and human effort.

I appreciate the authors response to my questions and concerns, particularly the biased estimator and non-uniform sampling results. That said, could the authors explain why the performance is noticeably worse with a ranker accuracy of 70%? Presumably beta should decrease monotonically with higher ranker accuracy, at least in the case with 0% bias.

We thank the reviewer for their thoughtful assessment and constructive feedback. We appreciate the recognition of RankRefine's novelty, versatility, and empirical strength across diverse benchmarks. Below, we address each of the reviewer’s concerns in detail and hope our clarifications and new experimental results strengthen your confidence in our work. We believe RankRefine offers a practical and broadly applicable contribution, especially in settings where ranking information is more accessible than precise regression labels.

Q1: The method relies on some assumptions such as unbiased regressor and may not hold in real-world scenarios.

We agree that this is an important, practical concern. Our experiments on the TDC ADMET datasets (Figure 3 in the main paper) demonstrate that RankRefine works in real-world scenarios. We have also conducted new experiments where we systematically inject constant bias into an otherwise unbiased regressor. Specifically:

1. We start from synthetic predictions (arbitrary labels injected with random noise) with zero-mean Gaussian noise, yielding a mean absolute error (MAE) close to one standard deviation of the ground truth labels.
2. We then add a constant offset, increasing in magnitude from 0% to 70% of the label standard deviation (SD).
3. We report the refined-to-original MAE ratio (β = $\frac{MAE_\text{refined}}{MAE_\text{non-refined}}$, lower is better) for varying ranker accuracies.

The results (see Table 1) demonstrate that RankRefine remains effective even under substantial regressor bias. For example, with a 60% bias and only 60% ranker accuracy, RankRefine achieves β = 0.9649, i.e., a 3.51% gain.

Table 1. RankRefine performance, measured in β, on a biased regressor. We inject constant bias into an otherwise unbiased regressor, where the magnitude of the offset is measured as a percentage of the ground truth labels standard deviation. RankRefine offers improvement even when the regressor is significantly biased (e.g., 60% of the SD of the ground truth labels)

Q2: How does RankRefine compare with other ranking methods or models that could potentially be used as rankers in different domains?

Thank you for this suggestion. While RankRefine is agnostic to the choice of ranker, we recognize the value of exploring different types. To that end, we experimented with a Random Forest (RF) classifier as the ranker on the Caco2 dataset (TDC ADMET benchmark). Despite being a simple model, it achieves 75.78% pairwise ranking accuracy.

This highlights the flexibility of RankRefine, it can leverage any ranker (few-shot LLMs, learnt classifiers, domain-specific heuristics), making it adaptable to different domains.

Q3: What is the optimal size of the reference set?

Thank you for raising this point. As shown in Figure 3 (main paper), RankRefine’s performance improves as reference set size increases, up to a point. We observe that performance saturates around 20 samples, with diminishing gains beyond that (e.g., 30 samples). This suggests that RankRefine is not only effective but also efficient, requiring only a small number of reference samples to yield performance improvements.

Additional Comments

We also note the reviewer’s concern about theoretical assumptions (e.g., Gaussian errors, independence). While these simplify analysis, we validate RankRefine extensively in real-world settings (e.g., molecular property prediction, tabular data) where such assumptions do not strictly hold. The consistent empirical improvements across diverse tasks suggest that RankRefine is robust beyond the theory’s assumptions.

On the concern of ranking cost, we note that:

• Pairwise comparisons can be parallelized.
• Smaller reference sets (≤ 20) are sufficient for substantial improvements in most tasks.
• RankRefine only requires inference-time computation, no retraining is involved, preserving efficiency.