Neural Mutual Information Estimation with Vector Copulas

Thank you for your very insightful comments and the constructive criticisms. We believe many criticisms (for instance, no high-MI/high-dimensionality experiments, improper evaluation) were due to some misunderstandings, which we clarify below. Many of the requested comparisons/analyses can also be found in the provided appendix. Meanwhile, we have included new results and theories, as well as properly discussed related works to fully address your concerns.

1a. Comparison with STOA Discriminative Methods

We address your concerns on baselines from two aspects: (a) clarifying our baseline choice and (b) futher comparing with the suggested baselines.

• (MRE is a competitive baseline) We fully agree that $f$-DIME and SMILE are STOA estimators, as we will note in our paper. That said, MRE is also highly competitive: it is verified on high-dimensional non-Gaussian data (80D) and high MI regimes (20+nats). Importantly, MRE shares key similarities with GAN-DIME---both methods estimate the ratio via classifier training and avoid computing the partition function. In practice, we found them almost equally strong, so we mainly compare with MRE for space efficiency.
• (Further comparison to SMILE) Following your suggestion, we have also comapred to SMILE; see the table below. The results highlight the advantanges of our method. A more complete comparison will be included in the manuscript.

Table 1. Comparing SMILE vs VCE with 64D data. Results are collected from 8 independent runs. Code available in the provided code repo.

If necessary, we are fully happy to replace MRE with $f$-DIME or SMILE in our main text. We wish it understandable that it was unrealistic to include every baseline due to space limit.

1b. Issues with Benchmark & Evaluation Protocol

We clarify that our benchmark and protocols highly overlap with SOTA benchmarks [44, 50, 51] and recent estimators (and may even be richer). See Table 2 for a comparsion.

Table 2. Comparing the benchmarks and evaluation protocols in different works.

$^†$We do not employ the self-consistency (SC) test from [29], as all our test cases (including images and texts) have known and controllable ground truth (GT), requring not using this test.

2. Bias-and-variance Analysis

• (Empirical analysis) Following your valuable suggestion, we have now included an empirical variance analysis of VCE. See Table 3 for an example.

Table 3. Empirical variance of VCE with 64D data w.r.t sample size $n$. Reported values are the std across 16 randomly sampled batches.

• (Theoretical analysis). We also derive a new proposition analyzing how the number of mixture components $k$ governs the bias-variance trade-off in vector copula modeling:

(Proposition 5) For large sample size $n$, the expected KL divergence between the true vector copula $c$ and its $k$-component vector Gaussian copula mixture estimate $\hat{c}_k$ satisfies

$\mathbb{E} \left[ \mathrm{KL}[c || \hat{c}_k] \right] \leq C_1 \cdot k^{-\frac{2\alpha}{d}} + C_2 \cdot \frac{k d \log (C_3 n)}{n},$

where $\alpha, C_1, C_2, C_3 > 0$ are constants depending on the true vector copula solely, $d$ is the data dimension and $n$ is the sample size. Here, the 1st term on the RHS quantifies the bias, which decreases as $k$ grows and is irrelevant to the sample size $n$. The 2nd term on the RHS corresponds to the variance, which increases as $k$ grows and decreases as $n$ grows.

3. Neural Parameterization vs Model-based Parameterization

Thank you for highlighting this comparison --- we are on the same page! Figure 1 in Appendix B2 precisesly perform this comparison. In the figure, we compare VCE (model-based parameterization) and VCE’ (neural parameterization) in diverse settings. The results clearly show the advantange of model-based parameterization in the considered settings.

We fully understand that the reviewer is skeptical regarding the applicability of classic methods in high-dimensional space. However, please note that we are not applying classic methods directly in the original space, but rather a transformed space learned by flow models. This transformation bridges the gap, making classic method viable in this space.

4. High MI and High Dimensionality (d>50) Scenarios

We clarify that our original submission actually include both these cases (though they might not be so easy to notice due to our presentation); see Table 4 below for a summary. Our method demonstrates either significant advantange or highly competitive performance in both cases.

Table 4. MI and data dimesnionalities considered in this work.

*For compressible data, we leverage injective flow (implemented as autoencoder + latent flow). The same autoencoder is used for all other estimators for fairness.

5. Comparison to Related Copula Methods

• Additional paragraph comparing our work with [R1, R2]

“Copula has been widely used for MI estimate [R1, R2, other works]. Existing methods primarily focus on classic copulas, where the copula transformation is applied independently to each univariate marginal. This strategy has been shown to improve accuracy and reduce variance [R2]. We go one step further by using vector copulas, where the transformation jointly considers all dimensions of the multivariate marginals. This can be seen as a generalization of classic copula transformation, where we not only consider MI's invariance to element-wise bijections but also to any bijective function. Another key difference lies in that existing works [R1, R2, …] treats PMI as a density ratio, whereas our work treats PMI as a density.“

• Texts clarifying the difference between our work and [40]

“Similar to our work, prior work [40] also interprets MI as the differential entropy of copula. However, building upon classic copulas, their result only holds for scalar marginals, and generalizing their result to vector marginals require non-trivial formulation and derivation. Our work in contrast holds for vector marginals with any dimensionalities by leveraging vector copulas.”

We promise that all these discussions will be included in the final version. Thank you once gain for highlighting this important comparison.

6. Double Step Approach Validation

We are on the same page again --- the ablation study Joint Learning vs Separate Learning in Appendix B2 precisely performs this quantitative analysis. The results clearly demonstrate the benefits of the double step approach over single step.

Note that even if the learned vector ranks are imperfect, double step learning can still be useful. This is because the first step will not alter the underlying MI. In this regard, computing vector ranks can be viewed as an informed data-preprocessing step to improve MI estimation --- akin to the scalar rank transformation used in the CODINE paper [R2].

Summary: VCE is a robust and accurate MI estimator across diverse settings, including high-MI regimes (e.g. 14nats), high-dimensional data (e.g., 64D incompressible data, 512D text embeddings), long tails, non-Gaussian dependencies, heteogeneous marginals, etc. However, it may be is less suitable for incompressible data with very high dimensionality (>128D).

Dear Authors,

Thanks a lot for having clarified some of the initial doubts I had while reading the paper. As a matter of fact, some of these were indeed already solved in the paper and in the appendix. I'm confident the new version will leave no room for doubts.

As all my concerns are now resolved, I'm willing to raise my score to 4. Looking forward to reading the revised version of this paper.

Dear reviewer 44kJ,

Thank you once again for your thoughtful review and for signing the Mandatory Acknowledgement. Your inputs have been highly valuable. We would be grateful to learn from you in details whether our rebuttal has addressed your concerns, particularly those stemming from potential misunderstandings (e.g. no high MI/high-dim experiments, benchmark choices) and insufficient prior work discussion. This would greatly help us improve our work regardless of the final decision.

Best regards, Authors

Thank you for your time and your work in reviewing our submission! Although being concise, your feedback is still valuable in improving our work. We truly appreciate your recognition of our work's merits, including our extensive evaluation, the reproducibility, and the overall strong performance. Below, we address your concerns.

• Figures too small: We sincerely apologize for this. In the revised version, we have (a) resized all figures and (b) increased the font size to improve clarity.
• Appendix missing: The appendix is actually provided as a separate file via the ‘Supplementary Materials’ link above. We have now merged it with the main text for better clarity.
• Table 2 may be redundant: This could be true, but Table 2 may help readers to get a sense of the text dataset, as it is a relatively new dataset to the MI community.

We hope that we have addressed your concerns. Thank you again for reviewing our work. Please don’t hesitate to let us know if there are any additional comments or suggestions!

Thank you for your high-quality review and the detailed feedbacks! We are deeply grateful for your recognition of our work’s theoretical maturity, potential impact in creating a focus shift in MI community and our comprehensive evaluation. Below, we address your concerns, particular those regarding the quality of the obtained vector ranks and related works.

W1. Sceptical about the quality of the obtained vector ranks

This is a valid concern. We address it from two aspects:

• (Guaranteeing uniformity) Each univariate rank $\mathbf{\hat{u}}_d$ is guaranteed to be perfectly uniform in our method. Specifically, when computing the vector ranks, we applied element-wised empirical rank rather than Gaussian CDF; see the provided code. The description in the manuscript was indeed inaccurate, and we sincerely apologise for this inaccuracy.
• (Testing independence) Following the reviewer’s insightful suggestion, we have performed a diagnostic to validate whether dimensions $\mathbf{\hat{u}}_i$, $\mathbf{\hat{u}}_j$ are independent. Tables 1 and 2 summarize the results. It is clear that $\mathbf{\hat{u}}_i$, $\mathbf{\hat{u}}_j$ are indeed near-independent in moderate-to-high dimensional settings. We will include this diagnostic in the revised manuscript.

Table 1. (Student-d distribution case). Inspecting the non-diagonal elements $\Sigma_{ij}$ in the covariance matrix $\Sigma$ of the estimated vector rank $\mathbf{\hat{u}}$.

Table 2. (Spiral transformation case). Inspecting the non-diagonal elements $\Sigma_{ij}$ in the covariance matrix $\Sigma$ of the estimated vector rank $\mathbf{\hat{u}}$.

While the above results indicate good vector rank quality (perfect element-wise uniformity + low correlation), quality degration may still occur. The above test serves as a highly useful diagnostic for quality assessment. We sincerely thank the reviewer for prompting this checking, which has been instrumental in improving the rigor and practicality of our work.

W2. Joint learning of vector copulas is not a well-posed problem

We are on the same page! Our prior mention of joint learning was merely for completeness, not to recommend it. In fact, separate learning is a key design in our method, as motivated by l120–l126 and justified by the ablation study “Joint Learning vs. Separate Learning” in Appendix B. The reviewer's reference further supports this design.

W3. How to obtain meaningfully different mixture components $c_k$ in the vector copula

The answer is simple: just randomly initialize the parameters of each mixture component, following standard practice in Gaussian mixture modeling. This is analogous to how neural networks are initialized to avoid homogeneous weight configurations. If the true dependence is non-Gaussian, MLE should learn diverse components to attain a high likelihood.

W4. Relevant previous works missing

Thank you very much for pointing us to these highly relevant works, which we have now discussed properly in the paper:

• (Acknowledging [A]). When introducing VCE’ (reference-based parameterization of vector copula), we cite [A] to explicitly acknowledge that similar method has been explored for classic copula. We have also related our Section 5 to the analysis in [B], thereby drawing a connection between the two works.
• (Discussing [B]). We have now included a new paragraph discussing existing copula-based MI estimators, where both [A][B] and other relevant works are included.

[A] Your copula is a classifier in disguise: classification-based copula density estimation, AISTATS 2025

[B] Towards a universal representation of statistical dependence, 2302.08151.

W5. Confusion on Proposition 4

We feel that we are on the same page --- Proposition 4 indeed suggests that a vector Gaussian copula model can perfectly represent any independent distribution $p'(\mathbf{x}, \mathbf{y}) = p(\mathbf{x})p(\mathbf{y})$.

Questions

• (The quality of the obtained vector ranks?) Thank you for this great suggestion. We have now included a diagnostic for assessing the quality of the learned vector rank $\hat{\mathbf{u}}$; please see our response to W1 above. More advanced tests can also be used, but we consider it as future works.

• (Other ways for obtaining vector ranks?) The linear programming approach from the original vector copula paper can also be used, but we found their method computationally expensive for even moderate dimensional variables (e.g. 30D), so we use the flow-based approach instead.

Curiosity

Is vector Gaussian copula mixture a simplification of vines?

This is an inspiring ask! It could be, but a careful check is needed, as we work with high-dimensional vector Gaussian copula rather than the commonly used bivariate copula in vines.

Do you test for or show in one of your examples that $\mathcal{N}$-MIENF / DINE-Gaussian indeed learns a vector Gaussian copula?

We have not shown this directly; however, the experiments in 6.1 may have implicitly verified this, as $\mathcal{N}$-MIENF only works well in cases with Gaussian dependence (see Figs 2c, 2d, 2e).

Could the better performance of neural estimators in Sec 6.2 also due to that they do not need to be normalised?

Yes, particularly considering its deep connection to model flexibility and capacity. However, please don't forget that this flexibility comes at a price (no control over complexity).

Other issues. Issues such as typos and notation issues have been fixed in the revised manuscript. We sincerely thank the reviewer for the very detailed checking and correction.

Please accept our heartfelt gratitude, not only for raising the score to a 5, but also for the very enlightening and rewarding discussion. We have learned a lot from it.

Best regards, Authors

Thank you very much for your prompt response! Please find below our detailed follow-up:

• W1: We promise that a comprehensive diagnostic will be added to the revised manuscript, covering diverse patterns, varying dimensionalities and failure cases. The full distribution of the non-diagonal elements in $\Sigma$ will be shown (see the examples under /ablations/vector_ranks_quality in our repo). Tables 3 & 4 below show more examples.

• W2: Thank you for the suggestion! We have replaced the original footnote with the note you proposed, and we now cite the TACtis-2 paper as additional support.

• W4: All promised modifications will be included in the final version — you have our word. Please don’t hesitate to share any further relevant literature.

• W5: Good question! While Proposition 3 tells that a VGC is a reasonable approximation up to second order, it does not imply that higher order dependencies can always be ignored. Proposition 4 goes further by suggesting that such high-order dependencies might be negligible for weakly dependent RVs. Our empirical results is consistent with this theory.

Table 3. (Mixture of Triangles). Inspecting the non-diagonal elements $\Sigma_{ij}$ in the covariance matrix $\Sigma$ of the estimated vector rank $\mathbf{\hat{u}}$.

Table 4. (Compressed BERT embeddings). Inspecting the non-diagonal elements $\Sigma_{ij}$ in the covariance matrix $\Sigma$ of the vector rank $\mathbf{\hat{u}}$. The case d=64 is a failure case$^*$.

*Interestingly, we find that even if the vector ranks are not perfectly learned (e.g. d=64 in Table 4 above), our estimator still yields a reasonable estimate. This may be due to that all univariate ranks $\mathbf{\hat{u}}_d$ are perfectly uniform, so even if $\mathbf{\hat{u}}_i, \mathbf{\hat{u}}_j$ occationally exhibit mild dependence, the overall entropy $|H( \mathbf{\hat{u}}_d)|$ remains low, leading to a small bias in Proposition 2.

We sincerely thank the reviewer again for the constructive feedback and the very enlightening discussion, which have been invaluable in improving the quality of our work.

Thank you for your constructive feedback and the positive assessment! Your recognition that our work provides a novel perspective of PMI, enables important re-interpretion of prior works, and have good theoretical contributions are very encouraging. Below, we address your constructive criticisms regarding empirical performance and presentation.

Stronger Empirical Performance

We fully agree on the importance of strong empirical performance. In fact, our estimator does demonstrate excellent performance across a wide range of settings, though this is not easy to notice due to readability issues in Figures 1 and 2 (which we have now fixed). Below, we highlight scenarios where our method shines:

Table 1. Scenarios where our method is a clear winner. Cases like A, C and D are particularly challenging and are pain points in existing estimators.

(Additionally, our method also gracefully scales to image and text data, either achieving the best performance or coming very close to the top performer in these cases.)

Notations & Presentation

We sincerely thank the reviewer for pointing out the issues with notation and presentation. We have made substantial improvements in the revised version, including:

• (Figures 1 and 2) We have make them clean and more readible by (a) increasing the font and figure sizes; (b) avoiding clutter by moving some baseline to the appendix.
• (Theoretical proof pointer) Following your suggestion, we have now pointed the reader to the relevant proof in the appendix for each theoretical result.
• (Notations clarity) We have formally defined the push-forward operation (the symbol “#”) in Section 2 to improve clarity.

Thank you for the prompt acknowledgement! Once again, we sincerely appreciate the time you’ve taken to evaluate our work and your recognition of our work.