Thank you for the constructive feedback! We sincerely appreciate all your comments, which have been invaluable in improving the paper. We hope the following clarifications and additional results address your concerns.

On Sequence length bias

Weakness 3: The authors multiply individual token probabilities to compute the overall sequence probability. This provides a bias towards shorter sequences, by assigning decreasing probabilities for longer sequences. Coincidentally, this puts negative correlations at a disadvantage. Is this something that needs to be corrected for, e.g. by considering the average token probability?

Question 3: Sequence Length Bias: How much is the previously mentioned sequence length bias expected to impact performance, by putting, e.g., negative numbers at a disadvantage due to the additional minus sign. Would the consideration of the average token probability be suited to fix this issue?

We’d like to first clarify the question regarding the sequence length bias.

In our setup that uses the GPT-4o tokenizer, both positive and negative correlation values are represented with the same number of tokens. Specifically, both forms include four tokens: sign, integer part, decimal point, and fractional part.

For example, the response

{

"coefficient": "0.5"

}

is tokenized as ' "', '0', '.', '5'.

while

{

"coefficient": "-0.6"

}

is tokenized as ' "-', '0', '.', '6'.

In our decoding process, we always treat the token immediately following the colon as the sign token, trimming the prefix (' "') before it. For positive numbers, the sign token can be an empty string or '+', while for negatives it is '-'. As a result, negative numbers do not suffer a disadvantage due to extra token length.

We will add the above clarification to the paper.

On the impact of a single token sequence on token probabilities

Weakness 2: From algorithm 1 and the description of it Sec. 2 I get the impression that parsing of correlational values is done from a single sequence of token generations with the corresponding captured logits. Wouldn't this procedure immediately skew the probability of alternative token sequences in the second step, as the next tokens are now conditioned on the single chosen token of the first step? Would a branching beam-search for individual paths of token generations capture and normalize probabilities more truthfully?

Question 2: Token Probabilities: Could the authors comment of the impact of a single token sequence on token probabilities. Would a multi-beam search provide more truthful estimates?

Thank you for highlighting this important point. We agree that a full multi-beam search would provide a more faithful estimate of the true joint token probability, as it considers all possible token branches at each decoding step. In our current single‑pass approach, we first sample the sign prefix and then condition subsequent decoding steps on that choice, approximating

$p(sign, maginitude) = p(sign)p(magnitude | sign)$

rather than the true joint. We will clarify this limitation in the paper.

However, in practice, we find that this approximation works well for three reasons:

1. High sign accuracy: As shown in Section 4, the LLM achieves a sign accuracy of 78.8% across our benchmark. This means that in the vast majority of cases, the single-pass decoding correctly captures the true sign of the correlation, and thus most of the probability mass is assigned to the correct branch.
2. High sign confidence: In addition to being accurate, the LLM is typically very confident in its sign prediction. Across the 2,096 correlations in our benchmark, the median probability gap between the two signs is 99.8% (77.7% on average). This indicates that the model significantly prefers one sign direction, so the probability mass of the alternative branch is negligible. Therefore, the joint probability missed by not performing beam search on the less likely sign branch has minimal practical impact.
3. Scalability: A full branching beam search would require exponentially more decoding paths—and, in practice, significantly more LLM calls—as the beam width increases at each token position. Since LCP is designed to process tens of thousands of variable pairs, such an approach would be expensive. The single‑pass strategy thus provides a practical balance, maintaining strong empirical performance while ensuring computational efficiency.

On Narrow Variance Prediction and Naive Baselines

Weakness 1: In section 4, the authors compare against baselines which mostly follow the purpose of constructing a uniform or mixture of Gaussian distributions from LLM predictions. KDE and Gaussian baselines mainly seem to suffer from choosing a too narrow variance (which is indicated by a 95%-coverage of over 92% of the random/uniform baseline, and with the absolute mean distance errors of KDE, Gaussian and LCP being the same value, indicating a good overall mean prediction of the LLM). I was wondering whether the most naive way of simply directly prompting 'to which extend a correlation with Pearson coefficient of [CORR] between variables [X] and [Y] within the provided context is surprising/hypothesis-worthy.' might not provide reasonable baseline and eliminate performance degradations due to incorrectly scaled variances.

Question 1: Narrow Variance Prediction / Naive Baseline: I would like to ask the authors to comment on whether performance degradations of the baselines are only due to too narrow choice of variance. Do other baselines exists that do now suffer from such problems? Would a naive approach that directly asks for the 'surprise' levels or hypothesis-worthiness provide a more fair baseline?

On Narrow Variance Prediction The performance degradations of the KDE and Gaussian baselines are indeed mainly due to their overly narrow variance. As shown in Figure 3, both achieve good accuracy but suffer from poor calibration, resulting in overconfident predictions. Specifically, the Gaussian prior tends to have lower sign accuracy than KDE and LCP because it uses only the verbalized mean prediction from the LLM, ignoring other possible values in the underlying distribution. This makes it more prone to sign errors, especially when the correlation magnitude is small (e.g., < 0.2).

On Naive “Surprise” Baselines:

We thank the reviewer for proposing alternative baselines. To evaluate this, we implemented two new baselines on our benchmark:

1. Binary Surprise: The LLM receives all relevant context and the observed correlation coefficient, and is asked to classify the correlation as “surprising” or “not surprising”.
2. Level Surprise: The LLM receives the same context and observed correlation and is asked to rate the surprise level from 1 (least surprising) to 5 (most surprising).

We then ranked correlations by the reported surprise scores. The results are:

LCP outperforms both naive baselines. The verbalized surprise approaches have two main drawbacks:

1. Lack of transparency: The LLM’s “surprise” assessment is opaque and may not consistently align with user expectations. In contrast, LCP is calibrated on a validation set of common-sense correlations, making its definition of “surprise” more interpretable and transparent.
2. Limited granularity: Verbalized surprise produces only coarse ratings, leading to many ties in ranking and degrading retrieval effectiveness. LCP, by constructing a continuous predictive distribution, supports more nuanced and flexible assessment.

Moreover, LCP is more general: users can define discrete surprise levels if desired, but still benefit from the underlying continuous prior.

We agree with the reviewer that including these baselines is important for improving the paper. We will report the new baseline results in the final paper.

On the top-ranked correlations

The authors perform an experiment in Sec. 5 to retrieve "expert-flagged, hypothesis-worthy correlations" and present average precision and rank, what are the top identifies samples and do they intuitively make sense?

The top identified correlations make sense intuitively. For example, one is the strong positive correlation between the number of Divvy bike docks (Divvy is Chicago’s public bike-sharing system) and a community’s socio-economic status, suggesting that bike stations are more concentrated in wealthier areas. This hypothesis has been investigated by domain experts in [1]. We will include a more detailed discussion in the full paper.

[1] "Riding tandem: Does cycling infrastructure investment mirror gentrification and privilege in Portland, OR and Chicago, IL?", Research in Transportation Economics.

We sincerely thank the reviewer for their constructive feedback and valuable suggestions.

Question 1: Would LCP be sensitive to the kernel functions and the sample size used to estimate the distribution?

Thank you for this important question. We conducted a preliminary sensitivity analysis to assess the effect of both kernel function choice and sample size on LCP’s robustness.

Kernel Function:

We compared three common kernel functions (Uniform, Epanechnikov, and Triangle) against the Gaussian kernel used in the main paper. For each kernel, we tuned the bandwidth parameter $\sigma$ on the same held-out validation set and evaluated on a benchmark of 2,096 correlated pairs:

The results show that LCP’s performance is stable across different kernel functions, maintaining strong accuracy and calibration. This indicates that LCP is not sensitive to the specific choice of kernel function. However, because each kernel has a different shape, the optimal bandwidth parameter $\sigma^*$ should be re-tuned using a held-out validation set for best results..

Sample Size: We interpret sample size as the number of distinct correlation values collected from top‑k decoding. In our setup (k=20), we enumerate an average of 65 values per variable pair. Section 4 shows this is sufficient for constructing a robust prior and achieving strong empirical results.

Additionally, we find that the optimal $\sigma$ for smoothing is stable across different validation set sizes, as shown below:

We hope these clarifications are helpful. Please let us know if you would like further details.

Question 2: In the prompts, what does Source Table mean? Does that mean the tabular data is also a part of the input?

Source Table is the name of the table. The tabular data itself is not part of the input. We will clarify this in the final version.

Weakness 1: Such assessment depends on LLMs' own prior. which may not be consistent across different LLMs; and furthermore, it lacks a clear explanation (the prior from LLMs itself is still mysterious ) about why some correlations are selected, and some are not.

We appreciate the reviewer’s point about the limitations of relying on LLMs and have discussed this in Section 9. Our primary motivation in this work was to develop LCP as a scalable, fully automated tool for surfacing interesting correlations from large candidate pools, which necessitated a human-out-of-the-loop design. Before applying LCP to identify hypothesis-worthy correlations, we perform a calibration phase to ensure that the LLM prior aligns with broadly shared human expectations, assigning high likelihood to commonly known or intuitive correlations. As shown in Section 4, this calibrated prior achieves strong accuracy and low mean absolute error on widely recognized correlations. At application time, correlations that receive low likelihood under the prior are deemed surprising, representing deviations from these established expectations. While the internal formation of the LLM’s prior remains somewhat opaque, the process by which LCP defines and ranks surprise is transparent to the users.

We thank the reviewer for the insightful feedback and detailed comments! Your feedback will help us improve the paper significantly.

On the inherent bias of LLMs towards specific numeric values

Weakness 1: The method asks the LLM to directly provide the correlation score and extracts a distribution over possible values instead of a single output by looking at the top-k token distribution from the logits. However, LLMs are notoriously biased when generating numerical outputs, specific numbers encompassing more weight because they occur in more diverse contexts (e.g. pi or rounded decimals). This issue is not addressed and may affect the reliance on the distribution over the scores (in Algorithm 1).

Question 1: How do you address the inherent bias of LLMs towards specific numerical values (see weakness 1)?

Thank you for raising this important point. We acknowledge that LLMs exhibit bias toward rounded decimals—for example, in our experiments, among 293 predictions in the range [0.3, 0.4), 282 were exactly 0.30. To mitigate this, we apply kernel smoothing to the decoded distribution, which spreads probability mass across nearby values and avoids sharp spikes at rounded numbers like 0.30. Additionally, the objective of LCP is to capture general human expectations rather than produce exact point estimates. Thus, biases toward rounded values do not significantly impact downstream robustness. For instance, if the LCP prior is centered at 0.30 and the observed correlation is 0.35, kernel smoothing ensures that 0.35 still receives high probability, resulting in low surprise. We will include this clarification in the full paper.

On the sensitivity of LCP to kernel standard deviation

Question 2: How does varying the value of the kernel standard deviation affects the robustness of the method in practice?

We conducted a preliminary analysis to test the sensitivity of LCP to kernel standard deviation. We evaluated three kernel standard deviation ($\sigma$) values (0.2, 0.4, 0.6) on our benchmark and the Nexus hypothesis‑worthiness task; the result is as follows:

Increasing $\sigma$ leads to a smoother prior, which improves coverage of credible intervals (from 77.4% to 91.9%) but also increases the mean absolute error between distribution mean and observed correlations (from 0.26 to 0.33). Sign accuracy remains stable across all settings.

Importantly, while the calibration of the prior is sensitive to $\sigma$, the relative ranking of surprising correlations, which drives downstream tasks like hypothesis-worthy pair discovery, remains robust across a wide range of $\sigma$ values. This is because moderate changes in smoothing shift the absolute likelihoods but generally preserve the order of correlations. In the above Nexus evaluation, all tested $\sigma$ produced the same Precision@k.

Choosing $\sigma$ involves a trade-off between sharpness and calibration: smaller $\sigma$ yields narrower, less calibrated intervals, while larger $\sigma$ offers better calibration but more conservative estimates. We select $\sigma$ = 0.4 as it achieves strong calibration without impacting discovery performance. Overall, our method’s ranking-based results are robust to changes in $\sigma$, but proper calibration remains essential for trustworthy probability estimates and credible intervals.

On counterfactual assessments

Question 3: Can you provide examples of counterfactual assessments (as described in Section 6)?

We provide a full example in Appendix E and will further clarify this in the full version. Additionally, all counterfactual contexts are open-sourced in the benchmark folder of our anonymous repository (shared in the original submission). Here are two representative examples:

1. An example where the LLM predicts correctly: “drinking water access” and “infant mortality” are originally negatively correlated.

Counterfactual context: “Imagine a world where every source of drinking water is heavily contaminated, so regions with the greatest water access suffer the highest rates of health problems.”

2. An example where the LLM predicts incorrectly (Appendix E): Horsepower and acceleration time (measured as 0–100 km/h time), which are originally negatively correlated.

Counterfactual context: “Imagine a world where cars are powered by highly unstable, experimental engines. More horsepower means a greater chance of catastrophic failure, forcing drivers to accelerate slowly and cautiously to avoid explosions.”

On Iterative approaches

Question 4: Existing work relies on iterative improvements over a prior hypothesis (e.g. [1,3] above). Have you investigated whether integrating the output prior into the prompt as an initial guess to be critiqued can lead to improvements over future iterations?

Using the LCP output as a “first guess” in an iterative critique loop is indeed an interesting idea. In the current work we focus solely on single‑pass prior elicitation and calibration, so we haven’t yet experimented with feeding the fitted prior back into the prompt for successive refinements. That said, we want to clarify that across our benchmarks, a single‐pass LCP call already achieves strong performance on empirical correlations. More importantly, LCP is meant to scale over large corpora of variable pairs: requiring only one LLM invocation per pair keeps both latency and cost manageable. We include this discussion in the full paper.

The effect of temperature on LCP

Question 5: Have you studied the effect of the temperature hyperparameter on the output density?

We have not explicitly varied the temperature hyperparameter in our experiments; instead, we use the default (unscaled) logits from the model to compute the output distribution. In our approach, we control the sharpness of the resulting density through the kernel bandwidth parameter sigma during the calibration phase. Conceptually, both temperature and sigma affect the spread of the distribution (but via different mechanisms): lowering the temperature makes the output probabilities more peaked, similar to using a smaller $\sigma$ for kernel smoothing. Our results show that overly sharp distributions lead to poor calibration. We chose to adjust only the kernel bandwidth because this approach is both more transparent and more convenient: smoothing with $\sigma$ allows us to tune the density shape directly in a single step, without having to balance two interacting parameters (temperature and bandwidth). Moreover, even if temperature were adjusted, kernel smoothing would still be necessary to produce a continuous density, so controlling everything via $\sigma$ is simpler and more interpretable.

On the structure of the paper

Weakness 2: While hypothesis assessment is at the core of the motivations, a small portion of the paper is actually dedicated to this problem (Section 5) and the core of the method mainly focuses on calibration. I think that the paper would gain from either reducing the emphasis on hypothesis assessment and developing more downstream tasks, or including more datasets/experiments on hypothesis assessment.

We agree that hypothesis assessment is a central motivation and appreciate the suggestion to strengthen this aspect. In the current paper, we prioritized the calibration part, as it forms the foundation for reliable hypothesis assessment. We recognize that expanding downstream experiments or including more diverse datasets for hypothesis assessment would further strengthen the paper and provide a more comprehensive evaluation. This is a valuable direction for future work, and we plan to extend our framework and experiments in this area. Thank you for this thoughtful feedback.

This question makes total sense to us. To assess the impact of temperature and sequence length on LCP, we conducted preliminary experiments varying the temperature from 0.0 (original setup) to 1.0 on our benchmark of 2,096 correlated pairs.

We found that increasing the temperature to 1.0 did not substantially affect the sharpness of the output density: the average standard deviation remained nearly unchanged (0.084 for temperature = 1.0 vs. 0.085 for temperature = 0.0). Performance metrics were also similar:

We observed that most LLM responses were still short, with structured JSON output, regardless of temperature. This is likely because LCP prompts the model to generate a specific JSON format, causing the initial tokens (```json) to be highly probable and limiting sequence length variability. For the small subset of responses with extra reasoning steps before the JSON (29 cases), the average standard deviation of the output distribution was even lower (0.037), much smaller than the learned optimal σ=0.4 used in LCP.

In summary, in our current setup, temperature does not significantly affect the sharpness of the resulting density. We agree that a more comprehensive study—for example, explicitly forcing longer output sequences and examining the correlation with output variance—would be valuable, and we are happy to expand on this analysis in the final version.

Thank you so much for your detailed and constructive comments, and we truly appreciate them. We hope the clarification and additional results provided below address your concerns.

On the scope and motivation of LCP

W1: The main concern I have is that the scope of the method is unclear. ...

LCP is defined for pairs of numeric attributes, producing a predictive distribution over the Pearson correlation coefficient. It supports continuous real‑valued variables directly (Pearson’s r requires both variables to be numeric, but they can be either continuous or discrete). Any missing‑value handling is performed upstream. LCP itself only consumes the descriptions of two variables and never their raw values.

Extending this to more than two variables would require eliciting a multivariate distribution (e.g. a Gaussian copula), which we have not experimented with as it is outside our current scope. In principle, one could apply LCP to each pair in a set of n variables, but inferring a coherent joint distribution would need additional machinery (e.g., ensuring positive definiteness).

Q1: Authors should discuss the choice of Pearson’s correlation measure and more generally discuss, ...

Rationale for Focusing on Pearson’s Correlation and the Two-Variable Setup

Analyzing correlations between two real-valued variables is a foundational step in data exploration and hypothesis generation, especially when working with large structured datasets. In practice, many analysts start with simple pairwise correlations to uncover relationships and generate initial hypotheses for deeper analysis. For instance, underwriting teams use correlations to evaluate new features, and city planners examine correlations between vaccine coverage and factors like grocery access or public transit. We focus on Pearson’s correlation because it is the most widely used measure of linear association in exploratory data analysis [1].

[1] "Thirteen Ways to Look at the Correlation Coefficient.", The American Statistician

Extension to Partial Correlation and Higher-Order Relationships

Our framework is readily extensible to partial correlations or more complex relationships, as it only requires natural-language context and elicits a scalar coefficient from the LLM. To adapt the method for partial correlation (i.e., correlation between X and Y after adjusting for confounders Z), one could augment the prompt: “Given these variable descriptions, after controlling for covariates Z, what is the expected Pearson correlation between X and Y?”, The rest of the method would remain unchanged. That said, robustly evaluating LLMs’ ability to reason about partial correlations requires dedicated benchmarks and further empirical study, which is out of the scope of this paper. This paper takes the first step by focusing on zero-order (pairwise) correlation.

On the domains of datasets used in our evaluation

W1: Based on the datasets used in the benchmark, what domains of datasets can we expect the method to perform well?

Q5: Please provide more details on the types of questions in the benchmark. ...

In the original submission, we provided an anonymous repository with all benchmarks. Below is a summary of the dataset domains:

• Cause–Effect Pairs: Well-known relationships such as altitude vs. temperature and CO2 emissions vs. energy use, spanning physics, biology, and climate science.
• Kaggle‑Sourced Pairs: Public datasets including relationships such as horsepower vs. fuel efficiency and BMI vs. blood pressure, across social sciences, economics, healthcare, and sports.
• Chicago Open Data Pairs: Civic and environmental pairs, e.g., crime rate vs. education, air quality vs. hospital admissions.

These examples span diverse domains, demonstrating both the breadth of our benchmark and the generalizability of our method. We agree that this clarification is important and will include it in the paper.

On incorporating domain-specific knowledge

W1: If I have domain-specific knowledge about the dataset that may not be accessible to the LLM, can it be used to influence the prior. ...

We would like to clarify that in our method, the LLM is not the sole source of contextual knowledge. Our method defines the prior as $p_{LM}(r_{X,Y} | C_{X, Y})$ where $C_{X, Y}$ is entirely user-provided and can include variable descriptions, table summaries, or any domain-specific knowledge.

In the Nexus evaluation, where data undergo domain-specific transformations, we explicitly describe these steps in the prompt. This demonstrates how users can incorporate domain-specific knowledge—such as aggregation at the census tract level or spatial joins—that goes beyond the LLM’s pretraining. LCP can effectively leverage such context to rediscover expert-labeled correlations (Section 5). We further explored this in Section 6 by providing counterfactual contexts, testing whether the LLM relies on explicit prompt context or defaults to memorized knowledge.

That said, we acknowledge that we did not incorporate explicit human priors into LCP in this work. Our main objective was to develop a scalable, fully automated tool for surfacing interesting correlations from large candidate pools. Incorporating richer user-supplied priors is an excellent direction for future research.

On finer-grained details about the approach and the experiments

W2: Some finer-grained details of the results should be provided to show the robustness of the method. ...

Q3: The for loop in step 5 over top-k tokens was unclear in Algorithm 1. What are the top-k tokens referring to? ...

Thank you for pointing this out. We will include finer-grained details in the paper. Meanwhile, here are additional details on the robustness of our method:

Decoding strategy and parameter choices:

In Algorithm 1, the “top‑k tokens” are the k most likely tokens produced by the LLM at each step when generating the correlation value. This value is represented by four tokens: the sign, integer part, decimal point, and fractional part (which may cover up to three digits). For example, "0.56" in the response {"coefficient": "0.56"} is tokenized as ' "', '0', '.', '56'.

At each decoding step, we consider the top 20 tokens (the API’s maximum) and enumerate all valid combinations to form candidate correlation values. We use a single decoding pass (no retries), and with k=20, this yields an average of 65 distinct correlation values per variable pair. As shown in Section 4, this provides sufficient diversity for building a robust prior and achieving strong empirical results. We will clarify these details further in the paper.

Sensitivity to variable transformations:

By construction, Pearson’s r is invariant to affine transformations (scaling and shifting) of the input variables. Empirically, in the Nexus evaluation, where data is aggregated and joined across tables, our method remains effective (Section 5).

Stability of Sigma Across Validation Set Sizes and Pair Types:

The validation set for tuning σ is randomly sampled to represent the overall data distribution, without bias toward specific correlation types. We also explored how varying the validation set size affects the optimal σ through subsampling experiments, summarized below:

As shown, the optimal σ remains stable across different validation set sizes. This supports our explanation in the paper (lines 151–161) that σ corrects for systematic model bias, which is consistent as long as the architecture, prompt, and task remain unchanged. We will include a more detailed sensitivity analysis in the full paper.

Q4: Nexus results indicate LCP aligns with human judgements on hypothesis-worthiness. ....

Hypothesis‐worthiness in our Nexus evaluation is not a separate objective baked into the prompt. It emerges directly from our calibrated LLM prior. Concretely:

1. Calibration aligns LCP with human expectations: We tune σ so that the LLM’s elicited density places high probability on correlations that people usually expect, making the prior a proxy for human intuition.

2. Surprise scores surface interesting pairs: In Nexus, we simply prompt for the Pearson coefficient and rank observed correlations by their likelihood under LCP.

On the relationship between calibration and surprise

W3: The application of the method seems to be at odds to the training. ...

Thank you for raising this important point. We will clarify the relationship between calibration and surprise in LCP. In our framework, calibration and surprise are complementary: calibration aligns the LLM prior with broadly shared human expectations by assigning high likelihood to well-known or intuitive correlations (e.g., “altitude vs. temperature,” “horsepower vs. fuel efficiency”). The calibration set is built to reflect these canonical relationships.

After calibration, the prior encodes what is expected in the domain. At application time, any correlation assigned low likelihood by the prior is flagged as surprising, meaning it deviates from established patterns. We do not distinguish between “improbable” and “surprising”—both are defined as unlikely under the calibrated prior. Thus, the prior serves both to encode common knowledge (via calibration) and to flag novel or non-obvious patterns (via low likelihood, i.e., high surprise). We will clarify this relationship in the full paper.

Q2: What are some of the formal assumptions required for ...

To clarify, our correlation discovery system does not use an LLM for hypothesis generation—it computes empirical correlations directly from tabular data. The LLM is only used for hypothesis assessment, so there is no risk of self-evaluation or circular reasoning. By “hypothesis generation,” we simply refer to interpreting discovered correlations (e.g., inferring that higher crime may lower housing prices from a strong negative correlation).