Response to Reviewer JcVR

We greatly appreciate the reviewer’s thorough evaluation and insightful feedback. Your comments have been invaluable in helping us improve the clarity and rigor of our work. We address each of your points in detail below.

Response to Q1

We thank the reviewer for this important comment. Due to page limits, we placed the Related Work section in the appendix, but we will make this clearer in the revised version. We also appreciate the observation that “dimensional collapse has been studied extensively in representation learning and self-supervised learning literature.” Accordingly, we now include more regularizers inspired by self-supervised methods (see our later response).

We would like to clarify, however, that the collapse we study in VQVAEs differs from that in self-supervised settings: while the representational collapse of self-supervised arises from forcing invariant output, the collapse in VQVAEs is due to the quantization mechanism, which are inherently different.

Response to Q2

We thank the reviewer for pointing out the limited scope of the comparative analysis. To address this, we extended our study beyond rank regularization to include several established approaches, such as Barlow Twins and VICReg from the self-supervised learning literature. Though originally designed for contrasting augmented views, these methods can be adapted to our setting for avoiding representational collapse. We also present a custom regularizer (SingHinge) designed to encourage singular values of the feature covariance matrix to be above a certain threshold. Detailed implementations of these regularizers are provided later in this response.

Table 1: CIFAR10 performance with different regularizers

(All regularizers were applied on the unquantized latents. Due to the time limit, we only tested the CNN architecture with $f=4$. Val Loss reports validation reconstruction loss only, excluding the regularization term. SingHinge($h$) applies a hinge loss on singular values with threshold $h$ (e.g., 0.01 or 0.1). Regularization weights were chosen to yield performance comparable to the no-regularization baseline.)

From these results, we observe a consistent pattern: stronger regularization increases the effective dimensionality but also leads to performance degradation once the dimension grows beyond the baseline level (≈9). Importantly, this trend appears robust across diverse regularization strategies, suggesting that the limitation is not specific to a single method or optimization artifact, but instead reflects a fundamental property of the VQ quantization bottleneck.

Response to Q3

We appreciate the reviewer’s suggestion to strengthen our evaluation with larger-scale datasets. We would like to clarify that Section 5.2 already includes ImageNet-1k experiments for our method and RQVAE. To further address the reviewer’s concern, we extended the evaluation to include vanilla VQVAEs and additional regularizers on ImageNet-1k:

Table 2: ImageNet-1k performance with different methods

Consistent with our CIFAR10 findings, we observe that increasing the strength of Barlow Twins, VICReg, or SingHinge raises the effective dimensionality but eventually degrades validation performance once dimensionality exceeds the baseline regime. This trend suggests that the trade-off between dimensional usage and reconstruction quality is robust across both small-scale and larger-scale datasets.

Please note that, due to time constraints, the results presented here are from training runs stopped early (21k steps), before full convergence. At this stage, the effective dimensions had plateaued, making it possible to compare the influence of regularizers. Differences from Section 5.2 also include the use of smaller parameter sizes. We plan to provide full-convergence results in the camera-ready version.

Response to Q4

We thank the reviewer for raising the important point about the computational cost of DCVQ. We emphasize that the computational overhead of DCVQ is minimal, as the processing of each subspace is parallelized. To quantify this, we report the actual training times of DCVQ compared to vanilla VQVAE on CIFAR10 using a CNN architecture with $f=16$, trained for 100 epochs on an A100 GPU:

From these results, we observe that the training time of DCVQ is essentially the same as that of vanilla VQVAE when the total dimensionality is matched. This indicates that the additional quantizations in DCVQ introduce negligible overhead.

Implementation Details for Regularizers

For Barlow Twins and VICReg, which were originally designed for contrastive learning with augmented view pairs, data augmentation is not needed in our rank-promotion setting. Instead, we form input pairs by duplicating the same latent vector. All other hyperparameters follow the original codebase.

Barlow Twins Loss

Given normalized latent vectors $x, y \in \mathbb{R}^{B \times D}$, where $B$ denotes the total number of patches across GPUs and $D$ the feature dimension, and letting $\text{BN}$ denote 1D batch normalization, the Barlow Twins loss is defined as:

VICReg Loss

Given $x, y, B, D$ defined similarly, the VICReg loss combines variance and covariance regularization:

The variance term:

where $\sigma$ denotes the standard deviation.

The covariance term:

with $\text{Cov}(x) = \tfrac{1}{B-1}(x - \mu(x))^\top(x - \mu(x))$, where $\mu(x)$ is the batch mean.

The total VICReg loss is then:

SingHinge

For latent representations $z \in \mathbb{R}^{B \times D}$, let $s_i$ denote the singular values of $\text{Cov}(z)$. With a threshold $h$, the SingHinge loss is defined as

References:

[1] Barlow Twins: Self-Supervised Learning via Redundancy Reduction

[2] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Response to Reviewer 21RK

We sincerely thank the reviewer for the thoughtful review and for the positive assessment of our work. We greatly appreciate your constructive feedback, and we have carefully considered each of your comments in detail. Below we provide clarifications and additional context.

Response to "I had hoped the authors would uncover the fundamental cause of dimension collapse"

We appreciate this insightful question, as understanding the fundamental cause of dimension collapse is indeed central to advancing this line of work. While a full theoretical characterization remains an open challenge, our experiments provide evidence for a plausible underlying mechanism.

As discussed in the paper, the quantizer is the first module to collapse. We attribute this not to incidental factors but to an inherent bias of the quantization process. Specifically, VQ‑VAE’s quantization can be viewed as an online K‑Means procedure. A synthetic experiment can illustrate that K‑Means systematically reduces the effective dimension of the data: the learned centroids bias towards with the high‑variance directions of the data covariance matrix. Concretely, the ratio between centroid eigenvalues and data eigenvalues stays close to 1 for the largest eigenvalues but drops significantly for smaller ones. Thus, high‑variance components are well preserved, whereas low‑variance components are suppressed, leading to a reduced effective dimension in the codebook compared to the original data.

The behavior could be illustrated by the snippet below. The experiment constructs synthetic data with a controlled covariance spectrum, applies K‑Means clustering, and compares the eigenvalue decay before and after quantization. The results consistently show that smaller eigenvalues shrink more severely, confirming the bias towards high‑variance directions.

Our findings are also supported by prior theoretical work linking K‑Means and PCA, including [1] Spectral Relaxation for K‑Means Clustering and [2] K‑Means Clustering via Principal Component Analysis. These works suggest that clustering inherently emphasizes dominant principal components, providing a theoretical lens through which to interpret the observed rank vanishing.

While a rigorous proof remains an exciting direction for future research, we believe that the empirical evidence presented here, together with prior literature, strongly indicates that dimension collapse arises as a structural property of quantization.

import torch
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

def gen_data(N, D, p):
    eigvals = (10 ** torch.randn(D)).pow(p).sort(descending=True)[0]
    U, _, Vt = torch.linalg.svd(torch.randn(N, D), full_matrices=False)
    X = U @ torch.diag(torch.sqrt((N - 1) * eigvals)) @ Vt
    return X, eigvals

def centroid_eigvals(X, n_clusters=512):
    kmeans = KMeans(n_clusters=n_clusters, random_state=0, n_init=10)
    kmeans.fit(X.numpy())
    cov = torch.cov(torch.tensor(kmeans.cluster_centers_).T)
    return torch.linalg.svd(cov)[1]

torch.manual_seed(0)

N, D, K = 4096, 64, 512
X, target_eigs = gen_data(N, D, p=0.3)
centroid_eigs = centroid_eigvals(X, n_clusters=K)
ratio = centroid_eigs / target_eigs

plt.plot(ratio.numpy(), 'o-')
plt.axhline(1.0)
plt.xlabel('Eigenvalue Index (Largest to Smallest)')
plt.ylabel('Centroid / Data Eigenvalue')
plt.yscale('log')
plt.show()

Response to "The proposed Divide-and-Conquer method still relies on low-dimensional codebook subspaces ... lacking a truly novel solution."

We thank the reviewer for this comment. We would like to clarify that Improved VQGAN does not perform dimensionality reduction; instead, it aggregates high-dimensional codebook vectors, which unfortunately still leads to collapsed representations. Our proposed DCVQ is explicitly designed to mitigate this issue, enabling more effective utilization of the latent space.

We also wish to emphasize that, beyond proposing a method, our work makes a contribution by identifying and systematically analyzing the collapse phenomenon itself — an aspect that, to our knowledge, has not been studied in depth in prior literature.

Repsonse to Q1: "It is necessary to include comparisons with VQGAN-FC ..., while also controlling for the same amount of information bits in the codebooks."

We greatly appreciate this suggestion. While we did not explicitly label it as “VQGAN‑FC,” we believe the requested comparison is already covered in our experiments. Specifically, in §3.2.1 we introduced a projection layer after the encoder and a corresponding inverse projection before the decoder to vary the background dimension while keeping the encoder architecture unchanged. This setup is equivalent to the Factorized Codes approach.

In Table 1, we report results across a wide range of background dimensions (3, 4, 6, 8, 16, 32, 64, 128, 256), directly testing different levels of factorization. Furthermore, Figure 5 shows how varying the background dimension affects the effective dimensionality, thereby addressing the reviewer’s concern.

Regarding normalization, we note that all models in our experiments apply L2 normalization before quantization (as described in §3.2.1). The only exception is in §5, where Residual VQ does not support L2 normalization; in this case, we disabled it for our method as well to ensure fairness.

Finally, we ensured comparability by controlling the total information content: the number of tokens and codebook sizes were kept consistent across methods, ensuring that the effective information capacity was comparable.

References:

[1] Spectral Relaxation for K-means Clustering

[2] K-means Clustering via Principal Component Analysis

Response to Reviewer TLST

We are truly grateful to the reviewer for taking the time to carefully read our work and provide such constructive suggestions. Your feedback has been instrumental in refining our manuscript. Below, we respond to each of your comments and questions.

Response to Weakness 1

We omitted detailed comparisons earlier because $f$ strongly affects validation loss, making figures hard to interpret. Table 1 now reports the values from Figure 7, grouped by scale factor $f$. Results show that effective dimension is not systematically influenced by $f$, while validation loss is. Under KoLeo, effective dimension varies with $f$, but without a consistent trend—likely due to regularization-induced instability.

Table 1. Comparison of different $f$

Response to Weakness 2

We thank the reviewer for this valuable comment. We agree that evaluating the generation procedure and sample quality is essential. These experiments are ongoing, and we plan to include results soon.

Response to Weakness 3

We thank the reviewer for the detailed feedback on our figures and tables.

1. Fig. 4: We will improve circle size visibility in the revision.
2. Fig. 5: All points are plotted with the same color; the perceived differences arise from overlapping points. We will revise the plotting style to eliminate this artifact.
3. Notation / Table 1: We apologize for the oversight. The effective dimension $d_eff$ was denoted as Effective Dim in the figures. We will revise 2.1 to make it clearer.
4. Fig. 7: We will remove line styles, as colors suffice.
5. $f=4$ (Fig. 9): Omitted due to space, but we will add it.
6. Fig. 9 (10?) (RQVAE parameters): The varied parameter was the number of quantizers; we will revise the caption/description. We also plan to include the standard VQVAE as a baseline.
7. Table 2:  (a) We use correlations instead of regression coefficients since hyperparameters vary in scale (e.g., EMA decay vs. codebook size), making regression coefficients non‑comparable; correlations are scale‑independent.  (b) We agree the binary “Rotation Trick” should be excluded and will present correlations separately for enabled/disabled cases. Please check the updated Table 2.1 and Table 2.2. Conclusions remain unchanged.

Table 2.1 Correlation coefficients when Rotation Trick is enabled

Table 2.2 Correlation coefficients when Rotation Trick is disabled

Response to Weakness 4

1. The definitions of the rotation trick and dead‑code restart threshold are provided in Related Work (Appendix A). We will expand on these explanations in the revision.
2. Training details are included in Appendix C.

Response to Questions

1. The projection layer following the encoder is a trainable linear layer. This allows us to map the latents into arbitrary background dimensions. Similar tricks could be found in Improved VQGAN (RQVAE).
2. For RQVAE, the final representation is obtained by summing multiple codes, while the representational space is determined by the set of codes prior to summation. We concatenate all code vectors into a joint matrix and compute its effective dimension. In contrast, our method defines the space as a direct sum of subspaces, so its effective dimension equals the sum of the effective dimensions of the individual subspaces.
3. In Figure 9, effective dimensions of around 100 correspond to the DCVQ method. The figure illustrates how increasing the number of quantizers raises the effective dimension. For example, the notation $16 \times 8d$ indicates 16 quantizers, each with dimensionality $8d$.
4. Our hypothesis is that quantization biases towards high‑variance directions. Since quantization can be viewed as a variant of online K‑Means, we can illustrate this phenomenon in a controlled K‑Means experiment. Specifically, we generate synthetic data with a covariance matrix of desired eigenvalues, apply K‑Means clustering, and then examine the eigenvalues of the covariance of the cluster centroids. We find that the smaller eigenvalues shrink more severely after clustering, while the larger ones are preserved. As a result, the effective dimension of the centroids (i.e., the codebook) is reduced compared to that of the original data. The following snippet demonstrates this behavior. To develop a theoretical explanation for the rank‑vanishing behavior observed in K‑Means, one could connect the observations to existing frameworks, such as (1) Spectral Relaxation for K‑Means Clustering and (2) K‑Means Clustering via Principal Component Analysis. These studies suggest that K‑Means exhibits PCA‑like behavior, preferentially capturing the principal components of the data.
5. We assume that the effective dimension is more like an artifact of the quantization process, since no such collapse is observed in standard autoencoders.

import torch
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

def gen_data(N, D, p):
    eigvals = (10 ** torch.randn(D)).pow(p).sort(descending=True)[0]
    U, _, Vt = torch.linalg.svd(torch.randn(N, D), full_matrices=False)
    X = U @ torch.diag(torch.sqrt((N - 1) * eigvals)) @ Vt
    return X, eigvals

def centroid_eigvals(X, n_clusters=512):
    kmeans = KMeans(n_clusters=n_clusters, random_state=0, n_init=10)
    kmeans.fit(X.numpy())
    cov = torch.cov(torch.tensor(kmeans.cluster_centers_).T)
    return torch.linalg.svd(cov)[1]

torch.manual_seed(0)

N, D, K = 4096, 64, 512
X, target_eigs = gen_data(N, D, p=0.3)
centroid_eigs = centroid_eigvals(X, n_clusters=K)
ratio = centroid_eigs / target_eigs

plt.plot(ratio.numpy(), 'o-')
plt.axhline(1.0)
plt.xlabel('Eigenvalue Index (Largest to Smallest)')
plt.ylabel('Centroid / Data Eigenvalue')
plt.yscale('log')
plt.show()

Thank you for your thoughtful and clarifying response. I appreciate that many of my concerns have been addressed, and I found your explanation regarding the quantization bias along the principal component direction particularly interesting. Highlighting this point in Section 3.3 would, in my view, add valuable depth to your discussion.

While it is understandable that time constraints during the review period made it difficult to conduct additional generation experiments, I believe exploring the relationship between effective dimensionality and generation quality in future work could provide further important insights. With respect to Table 2, although correlation coefficients are certainly useful, I wonder if reporting standardized regression coefficients might better capture the individual effect of each variable while controlling for others, thus more directly serving your intended analysis.

Lastly, the impact of $f$ (the degree of data compression) does not appear to be a central focus in the current manuscript. Given that one might intuitively expect a connection between $f$ and the effective dimension, I believe that explicitly discussing your empirical observation of no such relationship—and offering possible interpretations or hypotheses, as you did for the quantization bias—would further strengthen the contribution of your paper.

Accordingly, if the authors adequately address the issues raised in this review—such as reorganizing figures and tables and providing thorough interpretations regarding quantization bias and the encoder scale factor $f$—I believe this paper merits a higher evaluation. For these reasons, I will raise my score to 5.

Dear Reviewer TLST,

I hope you are doing well.

We are truly grateful for your constructive feedback and your kind note indicating that you would consider raising your score to 5 after our rebuttal. Your suggestions on clarifying the quantization bias discussion and other points have been invaluable to improving our paper.

As the rebuttal period is coming to an end, we just wanted to kindly check whether you might have had the chance to update your score and complete the Mandatory Acknowledgement in the system. We fully understand everyone’s busy schedule, and we greatly appreciate the time and effort you’ve already dedicated to reviewing our work.

Thank you again for your thoughtful engagement and support.

Best regards,

The authors

Response to Reviewer enVD

We greatly appreciate the reviewer’s thorough evaluation and insightful feedback. We address each of your points in detail below.

Response to Q1

We acknowledge the concern that KoLeo’s performance drop may not reflect true higher-rank incompatibility. To test this hypothesis, we conducted experiments with multiple alternative rank-promoting methods: Barlow Twins, VICReg, SingHinge (our hinge loss on singular values of covariance), and RankHinge (as suggested by the reviewer). All regularizers were applied on the unquantized latents. Due to the time limit, we only tested the CNN architecture with $f=4$. Detailed implementation is provided later in this response.

Table 1: CIFAR10 performance with different regularizers

(Val Loss reports validation reconstruction loss only, excluding the regularization term. SingHinge($h$) applies a hinge loss on singular values with threshold $h$ (e.g., 0.01 or 0.1). RankHinge($r$) applies a hinge loss on feature rank with threshold $r$ (e.g., 64 or 128). Regularization weights were chosen to yield performance comparable to the no-regularization baseline.)

From these results, we find a consistent trend: increasing the strength of KoLeo, Barlow Twins, VICReg, or SingHinge increases the effective dimension but also degrades performance once it grows beyond the baseline level (≈9). This pattern holds across very different gradient structures, suggesting the effect is not an artifact of KoLeo but a fundamental property of the VQ quantization bottleneck.

RankHinge is a special case. While the codebook effective dimension remains flat (~9) regardless of weight, we want to emphasize that this is not due to insufficient hyperparameter tuning: the unquantized latents show clear growth in dimensionality, reaching levels similar to KoLeo (shown below). Even with regularization weights up to 1000, the effective dimension stays at 9 while validation loss worsens (not shown), underscoring a regularizer limitation rather than a tuning issue.

Table 2: RankHinge regularization effective dimension vs. unquantized effective dimension

Response to "it would be notably strengthened with more modern examples (e.g. ImageNet)"

We appreciate the reviewer’s suggestion to include more modern datasets. We wanted to clarify that Section 5.2 includes experiments on ImageNet for our method and RQVAE. Here, we add comparison with vanilla VQVAEs and other regularizers on ImageNet:

Table 3: ImageNet-1k performance with different methods

Similar to CIFAR10, we observe that increasing the regularization strength leads to higher effective dimensions but also worse validation loss. RankHinge maintains a low effective dimension while showing no performance degradation.

Please notice that the results presented here are of a different setting compared to Section 5.2. Here, we didn't train the models to converge due to time constraints. Instead, we stopped training after 21k steps, at which point the effective dimensions had stabilized. Another difference is the use of a smaller parameter size. We plan to include full convergence results in the camera-ready version.

Response to Q2

In the above tables, we explored a diverse set of regularization strategies, which can be grouped into three categories:

• Embedding spreading (e.g., KoLeo): Encourages the embeddings to be spread out in the latent space.
• Redundancy reduction methods (e.g., Barlow Twins, VICReg): These encourage feature decorrelation and variance, commonly used in self-supervised learning.
• Covariance spectrum shaping (e.g., SingHinge): These directly regularize the singular values of the feature covariance matrix to promote higher rank.
• Rank thresholding (e.g., RankHinge): Inspired by the reviewer’s suggestion, this penalizes low-rank representations only when effective dimension falls below a target threshold.

Response to Q3

We observe that using more subspaces generally helps preserve high-frequency details in the reconstructions (e.g., hair). However, increasing the dimensionality of each subspace beyond our recommended 8 dimensions often harms these details. Due to rebuttal constraints, we are unable to upload visual examples at this stage, but we will include detailed qualitative comparisons and reconstructions in the camera-ready version.

Implementation Details for Regularizers

For Barlow Twins and VICReg, which were originally designed for contrastive learning with augmented view pairs, data augmentation is not needed in our rank-promotion setting. Instead, we form input pairs by duplicating the same latent vector. All other hyperparameters follow the original codebase.

Barlow Twins Loss

Given normalized latent vectors $x, y \in \mathbb{R}^{B \times D}$, where $B$ denotes the total number of patches across GPUs and $D$ the feature dimension, and letting $\text{BN}$ denote 1D batch normalization, the Barlow Twins loss is defined as:

VICReg Loss

Given $x, y, B, D$ defined similarly, the VICReg loss combines variance and covariance regularization:

The variance term:

where $\sigma$ denotes the standard deviation.

The covariance term:

with $\text{Cov}(x) = \tfrac{1}{B-1}(x - \mu(x))^\top(x - \mu(x))$, where $\mu(x)$ is the batch mean.

The total VICReg loss is then:

SingHinge

For latent representations $z \in \mathbb{R}^{B \times D}$, let $s_i$ denote the singular values of $\text{Cov}(z)$. With a threshold $h$, the SingHinge loss is defined as

RankHinge

The intuition behind RankHinge is to ensure that at least $r$ singular values are necessary to capture 99% of the variance. To achieve this, we penalize the case where the first $r-1$ singular values already explain $\geq 99\%$ of the variance. Formally, let

denote the cumulative variance explained by the top $r-1$ singular values. The RankHinge loss is then defined as

References:

[1] Barlow Twins: Self-Supervised Learning via Redundancy Reduction

[2] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

Thank you again for taking the time to review our work. Since the discussion phase is nearing its end, we wanted to check whether the additional extension and ablation experiments we included have helped to address your concerns. Please let us know if any further clarification would be helpful.

Dear Reviewer enVD,

We understand you may have a busy schedule, and we truly appreciate the time you’ve taken to review our submission. As the rebuttal period is closing in the next 24 hours, we just wanted to check in—if you have any further comments or questions, we’d be eager to respond if needed.

Thank you again for your time and consideration.

Best regards,

The authors