Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence

Dear Reviewer 4ZU6,

We sincerely thank you for your profound review. Below, we provide answers to the questions and concerns you raised.

Weaknesses:

1. Although introduced only in 2021, SMI has rapidly gained traction due to its ease of implementation and parallels with other slicing methods (e.g., sliced Wasserstein distances). By revealing fundamental limitations of SMI, and slicing‑based measures more generally, our work clarifies when these approaches can mislead practitioners and suggests reconsidering slicing strategies. We thus expect it to inspire further research into efficient, projection‑based dependency measures across various fields.
2. It is true that we focus on simple datasets, as they suffice to demonstrate the true scale of SMI's shortcomings. However, motivated by your feedback, we conducted four additional experiments using complex high-dimensional data from [r.a]. Due to space limitations, we are unable to include the results here. Please refer to 'Weakness 1' in our response to Reviewer GJD2.

Questions:

1. We chose the KSG estimator as our baseline precisely because it requires no auxiliary optimization, a key advantage of SMI over neural estimators such as MINE, which rely on iterative network training. Once one adopts a more computationally involved backbone for MI estimation, the primary appeal of SMI diminishes. Moreover, in our experiments the dimensions remain within the practical regime for KSG to yield stable estimates, as also shown by the low variance. Additionally, we use SMI-MINE in Section 6.2 to show the redundancy bias for this method. Overall, the obtained empirical results support our theoretical findings.

2. We share your confusion. The mentioned observation highlights the core "curse of slicing" uncovered by our work. Although one might expect sliced estimators to perform well, our theoretical and empirical results demonstrate that they can fail even at moderate $d$.

3. In light of our findings on SMI, one can recommend one of two strategies for high‑dimensional dependence estimation. The first one is indeed dimension‑reduction followed by MI estimation, i.e., we project the data onto a low-dimensional subspace that preserves most of the dependence (e.g., via PCA, CCA, or learned neural embeddings) and then apply a standard MI estimator (e.g., KSG) in that reduced space.

   However, if we still want to use the idea of slicing, then one may slice both the joint distribution $\mathbb{P}\_{X, Y}$ and the product of marginals $\mathbb{P}\_{X} \otimes \mathbb{P}\_{Y}$ onto the same random subspace. This results in estimating $\mathrm{D}\_{\mathrm{KL}}(\mathbb{P}' || \mathbb{Q}')$ and in this way we overcome some drawbacks of SMI/$k$-SMI (e.g., each slice now retains nontrivial dependence even when redundancy is low). The trade‑off is that one must employ KL‑divergence estimators for arbitrary distributions, rather than direct MI estimators.

   On the whole, we regard this important point as a direction for future work.

4. We begin our proof by recalling the MI formula for two $d$-dimensional Gaussian vectors: $\mathrm{I}(X; Y) = - \frac{d}{2} \log(1 - \rho^2)$. By rotational invariance, for any fixed unit vectors $\theta,\phi\in\mathbb S^{d-1}$ the one‑dimensional projections satisfy $\left(\theta^T X; \phi^T Y\right) \sim \mathcal{N} \left(0,\begin{pmatrix}1 & \rho \theta^T \phi \\\ \rho \theta^T \phi & 1 \end{pmatrix} \right).$

   By the definition of SMI, $\mathrm{SI}(X ; Y) = \mathrm{I}(\theta^T X; \phi^T Y \vert{} \theta, \phi) = - \frac{1}{2} \mathbb{E}[\log(1 - \rho^2 |\theta^T \varphi|^2)].$

   Since $|\theta^T \varphi|^2 \sim \mathrm{Beta}(1/2, (d - 1)/2)$ by Corollary A.5, we substitute its density and write in Eq. (6):

   $\mathrm{SI}(X ; Y) = - \frac{1}{\mathrm{B}(\frac{1}{2}, \frac{d-1}{2})} \int_0^1 \log(1 - \rho^2 x) (1 - x)^{\frac{d-3}{2}} x^{-\frac{1}{2}} \ \mathrm{d}x$

   Next, we use the identity $\ln(1 - z) = - z \ \_2F_1 (1, 1; 2; z)$, yielding

   $\mathrm{SI}(X ; Y) = \frac{1}{\mathrm{B}(\frac{1}{2}, \frac{d-1}{2})} \int_0^1 x^{\frac{1}{2}} (1 - x)^{\frac{d-3}{2}} \ \_{2} F_1 (1, 1; 2; \rho^2 x) \ \mathrm{d}x$

   We then simplify the Beta‐prefactor via the relationship between beta and gamma functions and the property $\Gamma(z + 1) = z \Gamma(z)$: $\frac{1}{\mathrm{B}(\frac{1}{2}, \frac{d-1}{2})} = \frac{\Gamma(\frac{d}{2})}{ \Gamma(\frac{1}{2}) \Gamma(\frac{d-1}{2}) } = \frac{\frac{2}{d} \Gamma(\frac{d}{2} + 1)}{ 2 \Gamma(\frac{3}{2}) \Gamma(\frac{d - 1}{2}) } = \frac{1}{d} \frac{\Gamma(\frac{d}{2} + 1)}{ \Gamma(\frac{3}{2}) \Gamma((\frac{d}{2} + 1) - \frac{3}{2}) }$

   At the same time we group the powers in the expression under the integral $\mathrm{SI}(X ; Y) = \frac{\rho^2}{2d} \frac{\Gamma(\frac{d}{2} + 1)}{ \Gamma(\frac{3}{2}) \Gamma((\frac{d}{2} + 1) - \frac{3}{2}) } \int_0^1 x^{\frac{3}{2}-1} (1 - x)^{(\frac{d}{2} + 1) - \frac{3}{2} - 1} \ \_{2} F_1 (1, 1; 2; \rho^2 x) \ \mathrm{d}x$

   and recognize the remaining integral as the Euler‐type transform, which takes the following form (recall that the generalized hypergeometric function is symmetric in its upper parameters $a_i$ and separately symmetric in its lower parameters $b_j$): $\_{p+1}F_{q+1}\left(c,a_1,\dots,a_p; d,b_1,\dots,b_q; z\right) =\frac{\Gamma(d)}{\Gamma(c)\Gamma(d - c)} \int_{0}^{1}t^{c - 1}(1 - t)^{d - c - 1} \ \_p F_{q}\left(a_1,\dots, a_p; b_1,\dots,b_q; z t\right)\ \mathrm{d} t.$

   valid when $\mathrm{Re}(d) > \mathrm{Re}(c) > 0$. Thus, we obtain $\mathrm{SI}(X ; Y) = \frac{\rho^2}{2 d} \ \_{3} F_2 \left( 1, 1, \frac{3}{2}; \frac{d}{2} + 1, 2; \rho^2 \right)$

   To prove an upper bound we introduce the variable $\eta = (1 + \theta^T \varphi) / 2$, so that in the limit $\rho \rightarrow 1$ SMI becomes

   $\mathrm{SI}(X ; Y) = - \frac{1}{2} \mathbb{E}[\log(1 - |\theta^T \varphi|^2)] = - \frac{1}{2} \mathbb{E} \log(1 - (2\eta - 1)^2) = - \frac{1}{2} \mathbb{E} \log 4\eta (1 - \eta) = -\log 2 - \mathbb{E}[1 - \eta]$, where we used that $\eta \overset{d}{=} 1 - \eta \sim \mathrm{Beta}(\frac{d-1}{2}, \frac{d-1}{2})$ by Corollary A.5. Finally, the Beta‑moment identity $\mathbb{E} [\log \eta] = \psi(\alpha) - \psi(2\alpha)$, where $\psi$ denotes the digamma function, yields

   $\mathrm{SI}(X ; Y) = \psi(d - 1) - \psi(\frac{d-1}{2}) - \log 2$

   To obtain an upper bound, we apply digamma inequalities $\log\left( x + \frac{1}{2} \right) - \frac{1}{x} \leq \psi(x) \leq \log(x + e^{\psi(1)}) - \frac{1}{x}, x > 0$. Thus, we have

   $\mathrm{SI}(X ; Y) \leq \log\left( d - 1 + e^{\psi(1)} \right) - \log(d/2) - \log(2) + \frac{1}{d-1} = \log\left( 1 + \frac{e^{\psi(1)} - 1}{d} \right) + \frac{1}{d-1}$

   To simplify the bound, one can note that $e^{\psi(1)} - 1 < 1, \log(1 + x) < x$, and $1/d < 1/(d-1)$

5. In our paper, we consider limitations arising in infinite sampling regimes (so the saturation arises solely from the inherent biases of SMI itself). However, both MI and sliced MI estimators built as distribution‑free lower bounds from $N$ samples are subject to the same (additional) finite‑sample limit, i.e., $\hat{I}, \hat{SI} < \log N$ with high probability ([a], [b]). This “saturation” of the estimates is a sampling‑limit phenomenon and applies equally to MI and its sliced average. Besides, we observe empirically that SMI saturates noticeably faster in practice, for instance in Figure 3 the estimate plateaus at roughly $2$ nats even though $\ln N \approx 9.2$ for $N = 10^4$ samples (indicating that the infinite-sampling limit is reached much sooner than the finite-sample limit). By contrast, the measures themselves behave very differently in high dimensions: the full MI grows linearly in $d$, $\mathrm{I}(X; Y) = \Theta(d)$, whereas the SMI decays as $O(1/d)$ (Lemma 4.1).

   Additionally, we note that some estimators (e.g., MINDE [r.b] and MIENF [r.a]) are neither lower nor upper bounds and thus avoid this particular theoretical limitation in finite-sample regimes. These estimators, however, employ generative models and are several levels above SMI in terms of computational complexity.

6. Section 4.1 extends our analysis to non‑uniform slicing methods (i.e., max‑SMI and optimal SMI) to show that the same redundancy bias persist even when projections are learned rather than chosen at random. We abbreviated this discussion because the majority of applications still rely on (averaged) SMI, but the key takeaway is that no variant of SMI escapes the "curse of slicing".

7. This is a third-party plugin we used to seed random generators and gather experimental results. To avoid using it, please, replace bebeziana.seed_everything(seed, to_be_seeded) in source/evaluate/run.py with numpy.random.seed(seed)

   We are deeply sorry for the inconvenience. The supplementary material will be fixed to include this piece of code.

8. Yes, we use "$\ll$" to denote an absolute continuity. Writing $\mathbb{P} \ll \mathbb{Q}$ means that whenever $\mathbb{Q}$ assigns zero probability to an event, so does $\mathbb{P}$. We will add this definition where the symbol first appears.

9. We enforce the energy constraint via standardization and whitening. Standardization rescales each coordinate by $D^{-1/2}$ with $D=\mathrm{diag}(\mathrm{Var}[X_i])$ and $D^{-1/2}_{ii}=\mathrm{Var}[X_i]^{-1/2}$, while whitening replaces $X$ with $\Sigma^{-1/2}X$ where $\Sigma=\mathrm{Cov}(X)$ and $\Sigma^{-1/2}$ is computed via scipy.linalg.sqrtm.

10. Thank you for catching this! It will be corrected.

We would like to once again sincerely thank Reviewer 4ZU6 for their work. If any concerns remain, we will be happy to address them as well.

[r.a] Butakov I. et al. "Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression." Proc. of ICLR 2024"

[r.b] Franzese G. et al. "MINDE: Mutual Information Neural Diffusion Estimation." Proc. of ICLR 2024

Dear Reviewer GJD2,

We sincerely thank you for the thoughtful questions you posed, which prompted us to clarify key points throughout.

Weaknesses:

1. "It is true that we focus on simple datasets, as they suffice to demonstrate the true scale of SMI's shortcomings.

   However, motivated by your feedback, we conducted four additional experiments using complex high-dimensional data from [a]. This work proposes two distributions of synthetic images: one with high redundancy (images of Gaussian blobs, where each pair of pixels contains just enough information to reconstruct the latent variables used to generate the image) and one with low redundancy (images of rectangles, where only border pixels contain the information required to reconstruct the latent variables). For further details, please refer to the original article. We are deeply sorry that we are not able to provide pictures illustrating samples from these datasets due to the restrictions imposed by AC.

   Using this data, we conducted experiments by varying the ground truth MI (divided by the latent dimensionality) from $0$ to $5$ in steps of $1$ and report mean SMI values. The rest of the experimental setup matches Section 5. The results are presented in the table below.

   MI / $d_\text{latent}$	$0$	$1$	$2$	$3$	$4$	$5$
   Gaussian images, $16 \times 16$	$0.01$	$1.39$	$2.86$	$4.30$	$5.42$	$5.97$
   Gaussian images, $32 \times 32$	$0.01$	$1.39$	$2.86$	$4.30$	$5.42$	$5.98$
   Rectangles, $16 \times 16$	$0.08$	$0.81$	$1.40$	$1.68$	$1.74$	$1.75$
   Rectangles, $32 \times 32$	$0.16$	$0.46$	$0.75$	$0.92$	$0.97$	$0.97$

   Note that in the high-redundancy case, SMI saturation only becomes noticeable around $5$ nats. In contrast, for the rectangle images, saturation occurs much earlier (starting at $2$ nats and reaching a plateau at $3$-$4$ nats) and becomes even more pronounced as dimensionality increases.

   We will extend these results to higher MI values and higher projection dimensionalities and include them as an additional plot in our paper.

Questions:

1. When the components of the random vector exhibit high redundancy, i.e., when the “effective” dimension is substantially smaller than the ambient dimension, $k$‑SMI narrows the gap to the true MI. This is captured in Corollary 4.5, where we analyze a covariance matrix distinct from that of Lemma 4.1. In that setting, $k$‑SMI coincide with MI; however, in such cases, the merits of SMI are eliminated completely, as it is essentially a low-dimensional regime.

2. We thank the reviewer for highlighting this important point. In fact, where all dependence lies within a $k$‑dimensional subspace, there is an implicit assumption that the data are intrinsically low‑dimensional from an information‑theoretic standpoint. This prior knowledge induces a bias in favor of methods that explicitly exploit low‑dimensional structure. Consequently, $k$‑mSMI, by projecting onto the most informative $k$ directions, achieves substantially lower variance and improved sample efficiency than a generic MI estimator operating in the ambient space. In finite‑sample settings, this implicit bias translates into more accurate and stable estimates of dependence, making $k$‑mSMI preferable whenever the true information content concentrates within a lower‑dimensional manifold. We emphasize, however, that this advantage is not unique to $k$‑mSMI but arises for any estimator that leverages the same low‑dimensional inductive bias (e.g., [a,b]).

Thank you again for these perceptive inquiries. We hope our revisions address them fully, and we remain ready to address any additional questions you might have.

[a] Butakov I. et al. "Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression." Proc. of ICLR 2024

[b] Gowri G. et al. "Approximating mutual information of high-dimensional variables using learned representations." Proc. of NeurIPS 2024

Dear Reviewer uLwu,

Thank you sincerely for reviewing our article! In the following text, we provide responses to your concerns. We hope that all of them are addressed properly.

Weaknesses:

1. We will add the subscript $\mathbb{E}\_{\mathbb{P}\_{X, Y}}$ to the expectation to make explicit the underlying joint distribution.

2. We denote by $\mathrm{W}$ an orthogonal matrix uniformly distributed according to the Haar measure on the orthogonal group $\mathrm{O}(d)$, by $\theta$ a unit vector uniformly distributed on the unit sphere $\mathbb{S}^{d-1}$, and by $\mathrm{A}$ a matrix with orthonormal columns sampled uniformly from the Stiefel manifold $\mathrm{St}(k, d)$. We will state them explicitly in both the main text and the appendix for clarity.

   In practice, we generate $\mathrm{W}$ by sampling a $d \times d$ matrix with i.i.d. standard Gaussian entries, performing a QR decomposition, and taking the orthonormal $\mathrm{Q}$ factor (this is also shown in the proof of Lemma A.4). To obtain $\theta$, we sample a $d$-dimensional vector with i.i.d. standard Gaussian entries and normalize it to have unit lenght. For $\mathrm{A}$, we draw a $d \times k$ matrix with i.i.d. Gaussian entries, perform a QR decomposition, and use the first $k$ columns of the resulting orthonormal $\mathrm{Q}$ matrix.

3. To distinguish the two uses of "d", we have typeset the dimensionality $*d*$ in italic and the differential operator $\textrm{d}$ in roman; nonetheless, we will consider the dimensionality symbol with an alternative letter in the revised manuscript.

Questions:

1. Thank you for pointing this out. This claim is in fact a direct consequence of Lemma 4.1. There we consider two jointly Gaussian $d$-dimensional vectors $(X, Y)$ with (component‑wise) correlation coefficient $\rho \in (-1, 1)$. We prove that, for any fixed $\rho$, $\lim_{d \rightarrow \infty} \mathrm{SMI}(X; Y) = 0$, whereas the standard linear correlation $|\rho|$ remains constant. Hence, in high dimensions SMI reports vanishing dependence even though a nonzero correlation persists. We will make this remark explicitly in the manuscript.

2. In the “Suboptimality of Random Slicing” paragraph (Section 3) and again in Section 4.1, we discuss max‑SMI and optimal SMI, both of which employ non‑uniform slicing via an inner maximization over projections. While these methods mitigate some saturation effects of random SMI, they continue to suffer from redundancy bias and incur a substantial increase in computational cost by requiring an auxiliary neural network. This contrast highlights our core conclusion: there exists an inherent trade‑off between estimator complexity and the capacity to capture rich, high‑dimensional dependencies.

3. In Example 1 of the original SMI paper, the authors consider two Gaussian vectors with an arbitrary covariance matrix, which prevents derivation of a closed‑form SMI. In that setting one can only upper‑bound the integral by maximizing over all one‑dimensional projections, yielding a simpler expression in terms of the canonical correlation coefficient (CCA). However, this bound does not coincide with the exact SMI and is not tightly related to CCA; the asserted tight connection to CCA in the original work is unfounded, as we demonstrate. To obtain a tractable analytic result while still illustrating peculiar properties of the SMI, we therefore restrict ourselves to a simpler block‑diagonal covariance structure.

   To clarify the connection between the Jacobi ensemble (Lemma A.4) and the distribution of $|\theta^T \phi|^2$, where $\theta, \varphi$ are independent and uniform on the sphere $\mathbb{S}^{d-1}$, first observe that this random variable (as well as $(1 + \theta^T \phi) / 2$) arises in the proof of Lemma 4.1. The Corollary A.5 incorporates the result for the distribution of this inner product under two transformations. We outline two possible ways to justify the claimed distribution and thus establish its relation to the Jacobi ensemble.

   Lemma A.4 shows that, for a uniformly‐distributed orthogonal matrix $\mathrm{W}$, the product $\mathrm{W_{11}} \mathrm{W_{11}^T}$, where $\mathrm{W_{11}}$ is the upper-left $k \times k$ block of $\mathrm{W}$, follows a Jacobi ensemble with parameters $a = 1/2$ and $b = (d - 1)/2$. Concretely, setting $k=1$ in the Lemma A.4 implies that the first column $\mathrm{w_1}$ of $\mathrm{W}$ satisfies $\mathrm{w_1} \sim \mu_{\mathbb{S}^{d-1}}$, and establishes that $\mathrm{w_{11}^2}$ has density propostional to $\lambda^{-1/2} (1 - \lambda)^{(d-1)/2 - 1}$, i.e., $\mathrm{w_{11}^2} \sim \mathrm{Beta}\left( \frac{1}{2}, \frac{d-1}{2} \right)$.

   Since $\mathrm{w_1}$ and $\varphi$ are i.i.d., it follows that $\mathrm{w_1^T} \varphi \overset{d}{=} \mathrm{w_1^T} e_1 = \mathrm{w_{11}}$ in distribution, so we recover that $| \theta^T \varphi |^2 \sim \mathrm{Beta}\left( \frac{1}{2}, \frac{d-1}{2} \right)$.

   An equivalent derivation starts with the distirubion of the inner product $u = \theta^T \varphi$ (which is $p(u) = (1 - u^2)^{(d - 3)/2} / \ \mathrm{Beta}((d - 1) / 2, 1/2), u \in [-1, 1]$) and then applies the change of variables $v = u^2$, recovering the same result.

   We will incorporate a more detailed explanation in the proof.

4. By "redundancy" we mean the replication of the information across multiple axes. More formally, redundancy corresponds to low differential entropy. As the covariance matrix becomes increasingly ill‑conditioned, the differential entropy diverges to $-\infty$, indicating a growth in redundant information. Proposition 4.4 shows that embedding any two variables into a higher‑dimensional space via a full‑column‑rank linear map leaves k‑SMI exactly equal to the original MI. Corollary 4.5 then considers the extreme case of a rank‑one Gaussian covariance, i.e., all coordinates carry identical information. It demonstrates that k‑SMI remains at its maximal value $-\frac{1}{2} \log(1 - \rho^2)$, whereas classical MI strictly decreases when information is duplicated. Consequently, unlike MI, which penalizes repeated information, SMI "rewards" redundancy by preserving or boosting its value.

   Our experiments in Section 6.2 also confirm this bias, showing that learning representations by maximizing SMI produces collapsed, highly redundant embeddings rather than decorrelated features.

The clarifications will be added to the upcoming revision of our manuscript. We hope that you will find our answer satisfactory. If there are some parts of our response which require further clarification, please, let us know.

We again sincerely thank Reviewer uLwu for carefully reading our article and pointing out the parts of our theoretical framework which require additional explanation.

Dear Reviewer Ly5f,

Thank you for your valuable feedback and insightful comments. We have carefully addressed your concerns.

Weaknesses:

1. In fact, we have already extended our empirical analysis to k-SMI: Figure 6 (2‑SMI) and Figure 7 (3‑SMI) in the main manuscript (line 236) demonstrate that SMI continues to exhibit rapid saturation as the slice dimension increases. Although the closed‑form expression for k‑SMI is intractable, Proposition 4.2 provides its integral representation, from which one can infer the same saturation behavior we observe empirically. Moreover, max-SMI remains subject to the same redundancy bias we identify for SMI, while optimal SMI interpolates between max-SMI and vanilla SMI, and thus inevitably inherits both their advantages and limitations.

   We deliberately center our attention on SMI and k‑SMI, as these variants are most pervasive in practice due to their simple implementation (no auxiliary optimization is required). By contrast, max‑SMI and optimal SMI demand using auxiliary NNs, placing them in a higher complexity class.

2. It is indeed true that when comparing MI and SMI, large differences between these two quantities are not of significant interest, and one should focus on SMI's ability to capture the correct ranking of dependencies. We explicitly state this on lines 220–223:

   SMI is a distinct measure of statistical dependance, and should not be viewed as an approximation of MI. Instead, our analysis focuses on the relationship between the two measures: since MI captures the true degree of statistical dependence, opposing trends in MI and SMI reveal problems with the latter quantity.

   That being said, our analysis shows that SMI is unable to provide a correct ranking of dependencies. Consider the two examples from Lemma 4.1 and Corollary 4.5. In the first case, MI is $d$ times higher than in the second case, while SMI is, in contrast, strictly smaller. Therefore, SMI incorrectly attributes stronger dependency to the case where, in fact, the dependency is weaker.

Questions:

1. While we did not apply explicit normalization or other preprocessing "tricks" in our current experiments, we do discuss the impact of input scaling on SMI in Section 6.1 (see Table 1), where we contrast the sensitivity of SMI versus MI under two different normalization schemes. Our experiments reveal that SMI is excessively sensitive to the choice of normalization, suggesting that no single preprocessing method can be relied upon, since it can substantially change the measured dependence.

We hope our response addresses your comments clearly, and we are happy to provide further clarification if needed.

Thank you again for your thoughtful review and helpful insights.