Extracting Rare Dependence Patterns via Adaptive Sample Reweighting

We thank the reviewer for the detailed and constructive feedback. Below we address all raised points grouped by topic. All figures and tables are available at https://tinyurl.com/mwafx6kh.

• Rare dependence in reality (Claims & Methods & W1) We clarify that we have evaluated our method on a real-world dataset (Sachs et al., 2005) in Sec. 5.2. We agree that real-world support is important and include additional real-world experiments (see first & second responses to Reviewer MSVk).

• Consensus datasets (Claims & Methods) We tested our method on three standard dependence benchmarks: Data Generation III (DG III) in Appendix with $\tau=1$, i.e. totally dependent; Sinusoid in (Ren et al., 2024) and ISA in (Gretton et al., 2007). As shown in the table, RHSIC slightly underperforms full HSIC due to data splitting to learn reweighting function (split ratio=0.5), but outperforms HSIC using half of the data. In practice, when we are unsure whether rare dependence exists, it is preferable to use HSIC first, and then apply RHSIC if HSIC fails.

  Method	RHSIC		HSIC		HSIC(n/2)
  Data Generation	Type I ↓	Power ↑	Type I ↓	Power ↑	Type I ↓	Power ↑
  DG III	0.03	0.99	0.04	1	0.04	0.67
  Sinusoid	0.04	1	0.05	1	0.02	1
  ISA	0.09	0.16	0.1	0.26	0.1	0.15

• Test-train splitting (W2) We agree that splitting for reweighting may cause a performance drop compared to full-data baselines, as shown above. However, RHSIC significantly outperforms them when rare dependence exists, suggesting the splitting loss is limited and outweighed by the gain in detecting rare dependencies. To examine the effect, we list different split ratios on DG III $\tau=0.1$. A 0.5/0.5 ratio generally performs well, consistent with prior work [1,2,3]. |Tr:Te|Type I ↓|Power ↑| |-|-|-| |7:3|0.07|0.68| |5:5|0.05|0.8| |3:7|0.01|0.74| |HSIC|0.04|0.17|

• Reference variable C selection (Comments) When do UI/CI tests where only X and Y are observed, we consider C=X or Y. Once more information is available, e.g. more observed variables as in causal discovery, C can be a third variable to leverage such information. Take the DG III $\tau=0.1$ as an example, in the causal graph X<--$\epsilon_b$-->Y<--Q, we observe X, Y, Q and want to test X $\perp$ Y. Proposition C.2 shows that C=X, Y, or Q are all valid here (i.e., do not introduce spurious dependence). RHSIC with C=Q performs the best since Q directly controls the dependence between X and Y, while C=Y, a child of Q, brings some power loss though still acceptable. C=X is ineffective since X $\perp$ Q. |Method|Type I ↓|Power ↑| |-|-|-| |RHSIC (C=Q)|0.05|0.8| |RHSIC (C=Y)|0.1|0.57| |RHSIC (C=X)|0.01|0.06| |HSIC|0.04|0.17|

  In practice, if only X, Y are available, we recommend testing with both C=X and C=Y and selecting the lower p-value.

• Proof for Prop. 3.6 (Theoretical) We are sorry for omitting the proof for Proposition 3.6 in the submission, and we include it below. The proof uses the independence preservation under measurable transformations.

  Proof. Since the data are i.i.d., $D_{tr} \perp D_{te}$. As $\hat{\beta} = f(D_{tr})$ is a measurable function of $D_{tr}$, it follows that $\hat{\beta} \perp D_{te}$ (Thm 4.3.5 in [4]). $\square$

  We also agree that the appendix numbering could be improved, and we will revise it to clearly indicate where each proposition is proven.

• (Q1) We have updated Fig. 3 (see the link above) and will revise the surrounding text for clarity. We use it to visualize that the original HSIC may fail to detect dependence in rare dependence settings, even if we have infinite samples.

• (Q2) We should make sure the reweighted p.d.f., $\tilde{\mathbb{P}}(X,Y)=\beta(C)\mathbb{P}(X,Y)$, is still a well-defined one, i.e. $\int_{\mathcal{X\times Y}}\tilde{\mathbb{P}}(X,Y)dXdY=1$, which gives us $\mathbb{E}[\beta(C)]=1$. We will clarify this in our manuscript.

• (Q3&4) Thank you for your careful reading. We will correct the terminology in L169 and formally define KCIT and RKCIT in Section 4.

• (Q5) Thanks for your question. In our rules, we employ KCIT to detect non-rare dependencies and RKCIT to detect rare ones. While RKCIT is capable of identifying non-rare dependencies as well, its reliance on data splitting may introduce unnecessary statistical inefficiency. Therefore, we prefer KCIT in such cases. Assumption 4.1 guarantees the reliability of KCIT when it rejects the null hypothesis. However, if KCIT fails to reject, this could be due to the presence of a rare dependence, in which case RKCIT serves as a complementary test.

We sincerely appreciate your thoughtful feedback and recognition of our work. Thank you for your time.

[1] Interpretable distribution features with maximum testing power. NeurIPS 2016.

[2] An adaptive test of independence with analytic kernel embeddings. ICML 2017.

[3] Learning adaptive kernels for statistical independence tests. AISTATS 2024.

[4] Casella & Berger, Statistical Inference.

We appreciate the reviewer's constructive comments and helpful feedback. Please see below for our response. All figures and tables are available at https://tinyurl.com/mwafx6kh.

• On kernel choice (Comments & Q1) Indeed, RHSIC/RKCIT performance depends on kernel choice. Our theoretical claims require a characteristic kernel. In practice (e.g., Examples 1.1 and 3.1), we use the Gaussian kernel with median heuristic width, as in standard HSIC.

  We conducted additional experiments comparing kernels and found that Gaussian consistently performs best. The poor performance of the polynomial kernel is expected, as it is not characteristic. Laplace is characteristic but has heavier tails than Gaussian; its performance drop suggests that lighter-tailed kernels may be preferable.

  Method (Kernel)	Type I ↓	Power ↑
  RHSIC (Gaussian)	0.05	0.8
  RHSIC (Laplace)	0.08	0.6
  RHSIC (Polynomial)	0.04	0.42

• On related work (References) We thank the reviewer for pointing out this related work. While our goals differ — ours focus on hypothesis testing, theirs on representation learning — both leverage dependence-maximization criteria to learn meaningful functions. We will cite and discuss [XZ2024] in the manuscript.

  [XZ2024] proposes a feature learning framework maximizing dependence with the target using an extension of Rényi maximal correlation (RMC) in $L^2$ space, with functions parameterized via neural networks. In contrast, we focus on detecting rare dependence using HSIC/KCI, widely adopted in independence testing, and learn a reweighting function to assign sample-wise importance. Theoretically, RMC provides a general measure of nonlinear dependence via optimal function pairs in $L^2$. Modal decomposition captures the full spectrum of the cross-covariance operator, with the leading mode corresponding to RMC. Restricting to RKHS and aggregating all squared singular values yields HSIC. While RMC is more general, its estimation and statistical inference in $L^2$ with neural networks might be more challenging. In contrast, our methods inherit the kernel-based formulation of HSIC, allowing efficient estimation by kernel trick and asymptotic distribution analysis with statistical guarantees.

• (Q2) Examples 1.1 and 3.1 illustrate two types of rare dependence and show a shared solution. Example 3.1 is local, with $X \perp Y$ holding in a subregion. Example 1.1 shows global but weak dependence, where the signal exists throughout but is largely buried by noise. Both of them motivate our method, which adaptively reweights samples to better reveal underlying dependence patterns.

• (Q3) We thank the reviewer for pointing this out. We acknowledge the inconsistency in notation: X, Y are introduced as random variables in Sec. 2 but later referred to as variable sets. We will unify the notation to "random variables or sets of variables". Most examples in our paper treat X and Y as variables for simplicity and illustration purpose. The phrase “C is a subset of X or Y” assumes that X and Y are sets of variables; when they are individual random variables, C is either X or Y. We agree this was ambiguous and will rephrase accordingly.

• (Q4) We thank the reviewer for this question. Our reweighting framework is general and does not assume rare dependence lies in a rectangular region. In nonlinear Example 1.1, axis-aligned boxes are purely for visualization; they are not a limitation of the method. The nonlinearity is captured by the reweighting function $\beta(C)$, which is flexible and nonlinear. It assigns higher weights to samples more likely to exhibit dependence, based on their values of C. This allows the method to adapt to complex dependence structures - as long as some signal is present along the reference variable C. Hence, our method naturally handles nonlinear rare dependence.

  We say that C can be either X or Y, but not both. As mentioned in Proposition 3.3, C being either X or Y preserves the independence between them and avoids introducing spurious dependence. However, using $C=(X,Y)$ in Eq. (15) makes the reweighted distribution $\tilde{\mathbb{P}}_{XY}=\beta(X,Y)\mathbb{P}_X\mathbb{P}_Y$ no longer factorizable in X and Y. This means the reweighed distribution changes the original independence relation. We will revise the manuscript to clarify these points.

• On notation and presentation (Comments)

  We appreciate the reviewer's comments. We will define "characteristic kernel" upon its first mention (end of Sec. 2), and revise ambiguous notation — in particular, replacing $\mathcal{H}$ with $\mathcal{F}$, denoting RKHSs of X, Y, Z as $\mathcal{F}_X, \mathcal{F}_Y, \mathcal{F}_Z$. We will also provide intuitive explanations and examples for each theorem or proposition when it appears, and streamline the notation for clarity.

We sincerely thank the reviewer for your recognition and your valuable feedback that helps improve the presentation of our submission.

We thank the reviewer for the insightful and constructive feedback. Please see below for our response. All figures and tables are available at https://tinyurl.com/mwafx6kh.

• On Motivation (Reference & W1) We thank the reviewer for this valuable suggestion. We agree that the motivation can be strengthened, especially regarding real-world relevance. We reviewed the literature on local and context-specific independence [1–5], which supports the practical significance of rare dependence. For instance, [3] applies CSI to physical dynamics such as friction; [5] demonstrates local dependence in reinforcement learning, e.g., autonomous driving; [6] discusses CSI in biomedical inference, e.g., dose-response effects in antibiotic treatment. These works are insightful and we will cite and integrate these as examples.

  Rare dependence is common across domains. In economics, income and consumption may appear weakly related in low-income groups but become strongly coupled at higher income levels. In psychology[7], the impact of social media on adolescent mental health becomes significant only with excessive usage. Similar rare dependencies occur in medicine[8], physics[9], and sociology[10], highlighting the practical relevance of detecting rare dependencies.

  We will revise the motivation accordingly and include references [1–5] and these real-world examples to better contextualize the relevance of rare dependence in practice.

• On real-world datasets (Experimental & W3)

  We agree that additional real-world evaluation would further strengthen the work. In response, we have added an experiment (see link above) using monthly JPY/USD exchange rates (E) and U.S. federal funds rates (F) from 1990 to 2010, sourced from FRED. While the original HSIC fails to reject independence (p = 0.2174), RHSIC detects dependence (p = 0.0005) using F as the reference variable. The learned weights assign higher importance to the samples in 2001 and 2008. These correspond to the Dot-com recession and the global financial crisis, respectively — showing that our method not only detects rare dependence but also provides interpretable insights.

• On local (in-)dependencies (W2 & Q1)

  We thank the reviewer for this thoughtful question. We apologize for the misleading wording "opposite" and will revise it accordingly. We intended to highlight a difference in focus — which we now realize is not necessarily exclusive: prior works aim to model precise conditional independence structures, while our goal is to detect rare dependence overlooked by existing tests.

  We agree that our methods can deal with context-specific or local (in-)dependence problems. For example, Example 3.1 and Data Generation II & III in our experiments involve local dependence, and our method performs well in these cases. Moreover, the learned sample weights can help identify independent regions — low-weight samples often correspond to locally independent areas. We will clarify this point in the revision.

• (Q2) Indeed, one advantage of our method is that the learned importance weights provide a natural way to identify subgroups of data that contribute most to the rare dependence signal. As shown in our real-world dataset analysis and local-independence discussion above, these weights help uncover meaningful patterns and guide downstream interpretation, enhancing interpretability.

  We agree that explicitly extracting high-weight subgroups could further extend the utility of our method. In particular, the learned weights can support fine-grained causal structure discovery by highlighting context-dependent relationships. Moreover, the structures recovered by our approach (e.g., the algorithm in Sec. 4) can provide valuable inputs for downstream tasks. Both aspects are relevant to applications discussed in [4,5,6], as suggested by the reviewer. We view this as a promising direction for future work.

• (Q3) Yes. Since kernel matrices can be constructed for discrete variables using Kronecker kernels [11], our method — which operates on these matrices — directly applies to the discrete case.

We sincerely thank you for your constructive feedback, the time you have dedicated, and your recognition of our work. We especially appreciate the valuable suggestions regarding future directions, which we find highly inspiring and will consider in our subsequent research. Thank you.

[7] A Systematic Review: The Influence of Social Media on Depression, Anxiety and Psychological Distress in Adolescents.

[8] Opioid-induced Hyperalgesia: A Qualitative Systematic Review.

[9] Frequency-Dependent Local Interactions and Low-Energy Effective Models from Electronic Structure Calculations.

[10] Time Spent on Social Network Sites and Psychological Well-Being.

We appreciate the reviewer's constructive comments and helpful feedback. Please see below for our response. All figures and tables are available at https://tinyurl.com/mwafx6kh.

• (Weakness & Reference) Thank you for your comments and for pointing out the work by Zhang et al. (2023), which we will cite in our revised version. Our work and Zhang et al. (2023) originate from different motivations and aim to address distinct problems. Our research is motivated by the specific challenge of detecting rare dependence, a topic that remains unexplored to the best of our knowledge. Rare dependence is widespread across many real-world domains and confirmed by lots of literature, such as psychology[1], medicine[2], physics[3], sociology[4], etc. Our focus is on detecting rare dependencies that are otherwise missed by existing independent tests. The proposed RHSIC and RKCIT are statistical tools for independence and conditional independence testing designed to recover rare dependencies. We emphasize that discovering causal relations in the presence of rare dependence is one important application of our methods, not the ultimate goal. Our method is not restricted to causal discovery.

  On the other hand, ReScore (Zhang et al., 2023) aims to optimize the structure recovery of score-based causal discovery methods with fewer spurious edges and more robustness to heterogeneous data. They propose an effective model-agnostic framework of reweighting samples to boost differentiable score-based causal discovery methods. The core idea of ReScore is to identify and upweight less-fitted samples, as these samples provide additional insight into uncovering the true causal edges—compared to those easily fitted through spurious associations. Although RD-PC, which employs our proposed RHSIC and RKCIT with correction rules, is also a sample reweighting-based method for causal discovery, its objective is fundamentally different from that of ReScore. ReScore is designed to mitigate the influence of spurious edges by focusing on less-fitted samples, whereas RD-PC aims to recover true causal edges that are easily overlooked or erroneously removed due to rare dependence. Exploring the connection between rare dependence and less-fitted samples may offer an interesting direction for future research.

  We appreciate the opportunity to clarify this and will update the manuscript accordingly.

• (Theoretical claims)
  Thank you for your insightful comments. We agree that the uniform bound in Theorem 3.7 applies to the empirical risk without regularization terms, rather than to the optimization problem (9). We acknowledge that this discrepancy exists and appreciate the opportunity to address it. We should have discussed this in detail in the paper. A gap remains between the theoretical aspect and the current algorithmic implementation. We have included a discussion in our updated manuscript. In practice, however, when analyzing data, we have to introduce a trade-off by employing the normalized RHSIC with two penalties in (9): the first term to control the smoothness of the $\hat{\beta}$ function, and the second to constrain the deviation of $\hat{\beta}$ from one to select as many samples as possible. These two penalties allow us to balance theoretical rigor with practical performance in practice.

  We appreciate the reviewer for raising this issue. To address this concern more thoroughly, we will revise the manuscript by adding an ablation study in the Experiment Section to quantify the regularization’s effect and discuss this theoretical-practical tradeoff explicitly.

We are encouraged that the reviewer recognizes our work as "well-written" and our theoretical analysis as "in depth." We also appreciate the reviewer’s acknowledgment of our comprehensive synthetic and real-world experiments.

Besides, we would like to add that our contribution goes beyond the classical results that the reviewer mentioned; we identify a new problem setting, rare dependence in statistical testing, and establish a general framework to solve this problem. In real-world scenarios, such rare dependencies often arise due to noise or imbalanced data distribution, making them difficult to detect using standard tests. To this end, we hope this work provides a foundation for various potential practical extensions, such as interpretable subgroup discovery, robust causal inference, and scientific analysis in domains like medicine, economics, and social science.

We sincerely appreciate your thoughtful feedback and recognition of our work. Thank you for your time and efforts.

[1] A Systematic Review: The Influence of Social Media on Depression, Anxiety and Psychological Distress in Adolescents.

[2] Opioid-induced Hyperalgesia: A Qualitative Systematic Review.

[3] Frequency-Dependent Local Interactions and Low-Energy Effective Models from Electronic Structure Calculations.

[4] Time Spent on Social Network Sites and Psychological Well-Being.