W1 (Mutual information)

Many thanks for this advice and suggestion. Following Reviewer SF1W’s feedback, we have included the definition of mutual information in Section 4.1. That is, we added the definition “mutual information $I$ signifies the amount of information obtained about one random variable by observing another.” and its mathematical definition to Appendix B, where we use the definition to derive Eq. (13) from Eq. (12). Furthermore, in Section 5.3.1, which focuses on the mutual information experiment, we now include a pointer to Appendix D.2.6 for detailed information on the estimation methods used.

W1 (Appendix B), Q1

We firstly appreciate the Reviewer SF1W’s pointing out that the proof in Appendix B is contributing. As Reviewer SF1W’s feedback, similar statements for $I(\boldsymbol{h}, r)$ cannot be derived. Appendix B proves that minimizing $L_\mathrm{dict}$ increases the lower bound on $I(\boldsymbol{h},y)$. With the inequality used in [2], $I(\boldsymbol{h}, r) <= I(\boldsymbol{h}, \boldsymbol{x}) - I(\boldsymbol{h}, y)$, $I(\boldsymbol{h}, r)$ will increase if $I(\boldsymbol{h},\boldsymbol{x})$ increase more than $I(\boldsymbol{h},y)$ increases. Likewise, $I(\boldsymbol{h}, r)$ will decrease if $I(\boldsymbol{h},y)$ increase more than $I(\boldsymbol{h},\boldsymbol{x})$ increases. Thus, to make a claim on $I(\boldsymbol{h},r)$, we should make a deductive claim on how $L_\mathrm{dict}$ affects $I(\boldsymbol{h},\boldsymbol{x})$.

Theoretically, making such a claim appears implausible, given that $L_{dict}$ pertains solely to the relationship between $\boldsymbol{t}_z$ and $\boldsymbol{h}$, and does not extend to the relationship between $\boldsymbol{h}$ and $\boldsymbol{x}$. Instead, Figure 4 empirically demonstrates a trend that $I(\boldsymbol{h}, \boldsymbol{x})$ progressively decreases as layers progress, resulting in the decline of $I(\boldsymbol{h},r)$.

In other words, we focused on the empirical aspect of $I(\boldsymbol{h}, r)$, demonstrating that our loss function is more effective in reducing $I(\boldsymbol{h}, r)$ compared to conventional contrastive loss in the absence of auxiliary networks $f_\phi$. Although the proof in Appendix B is vital to our paper, given the considerations mentioned earlier, please consider that we are unable to relocate it due to its considerable length.

However, we already know $I(r, y)=0$ by the definition of task irrelevant variable $r$: "any random variable that affects the observed data $x$, but is not informative to the task we are trying to solve" such that $I(r, y) = 0$ [1,2]. Intuitively speaking, observing task irrelevant variable $r$ gives no insight about task-relevant variable $y$, and vice versa.

W2

Thank Reviewer SF1W for highlighting the issue with the variable $z$. To reduce confusion, we have revised the notation for auxiliary network output from $\boldsymbol{z}=f_\phi(\boldsymbol{h})$ to $\boldsymbol{a}=f_\phi(\boldsymbol{h})$. Additionally, to ensure consistency, we have unified the class index as 'z' in Eq (2) and Appendix B, and changed $y\in\mathbb{N}$ to $y\in\{1,...,Z\}$ on page 3.

Q2

Many thanks for this interesting suggestion. The outcomes from your suggestion are indeed meaningful. As depicted in Figure 15, the t-SNE visualization of label embeddings post-training on CIFAR-100 (with 20 super-labels and 100 sub-labels) illustrates a noteworthy pattern. Labels within the same super-label group tend to cluster together. Furthermore, labels from different super-label groups are observed to be in close proximity if they share semantic similarities (e.g., the embedding for 'forest' is closer to that of 'trees' compared to other embeddings from the 'large outdoor scene' category.)”. We have incorporated a discussion of these findings into Section 5.3 of our paper.

Minors

Thanks for pointing out our mistakes on citations. We have revised the citation.

[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1947–1980, 2018.

[2] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. In International Conference on Learning Representations, 2020

In Relation to Neural Collapse

We extend our deepest thanks to Reviewer 3KwZ for insightful recommendations of significant literature. Our method aligns closely with the neural collapse phenomenon, particularly NCC or NC4 [1]. As [2,3,4] suggest, neural collapse also emerges in the intermediate layers. By minimizing $\mathbb{E}[L_{dict}(\boldsymbol{h},\boldsymbol{D}_Z)]$, our approach actively encourages the alignment between the average of local features with label $z$ and label embedding vectors $\boldsymbol{t}_z$, akin to the motivation of local losses used in [2,5].

However, our work is different from these studies in its exploration of locally decoupled contexts. Existing works [2,3,4,5] focus on local features within an end-to-end BP framework, whereas we investigate these features in scenarios where each layer or module functions independently. In this perspective, every local feature can be considered as the “final feature before the classifier”, which is where neural collapse emerges. In fact, as detailed in Appendix C, our methodology is capable of making layer-wise predictions. Figure 9 in the revised script demonstrates an increase in accuracy as we progress through the layers, resonating with observations made in [3].

This unique perspective allows us to uncover and understand the effects of neural collapse in a different and possibly more nuanced computational environment. We've drawn connections to neural collapse in our revised manuscript in Appendix F, even though it wasn't the central focus of our original paper, to elucidate the relevance of our findings within the broader landscape of neural network research.

W1

We thank Reviewer 3KwZ for the suggestion. Because we are more familiar with MLP-Mixer than RegNet, we test $L_\mathrm{dict}$ with ResNet-32, ViT, and MLP-Mixer in Appendix E. These architectures are composed of multiple modules, with each one acting as a functional unit. For example, in ViT, the multi-head attention (MHA) and MLP cooperate as a single unit (module); MHA is responsible for capturing attention, and the MLP is tasked with processing it. We maintain that these modules are integral to their respective architectures and should remain coupled, as decoupling them would disrupt their designed functionality. Consequently, in our experimental setup, we applied $L_{dict}$ at a module level, incorporating module-wise BP. However, we did not employ module-wise auxiliary networks used in LL. Moreover, we compare our results with $L_\mathrm{contrast}$ (as defined in Eq (1)), which does use such networks.

The results, which are compiled in Table 8, indicate that our approach is not only more parameter and memory efficient but also yields superior results compared to $L_\mathrm{contrast}$ across all tested architectures.

Q1 (One embedding per class for all layers).

We express our sincere appreciation for Reviewer 3KwZ’s valuable suggestions. Following the advice, we have applied and evaluated the layer-wise dictionary $\boldsymbol{D}^Z_l$ across datasets.

The results indicate that the layer-wise approach outperforms a single dictionary overall. Moreover, Reviewer 3KwZ’s recommendation presents an effective method for parallel training contexts where each layer could be processed on a separate GPU.

Q1 (Replacing average pooling with MLP)

We again convey our deep gratitude for Reviewer 3KwZ’s insightful recommendations concerning alternatives to pooling. In an experiment based on this suggestion, we mapped $\boldsymbol{t}_z$ to $\boldsymbol{t}_z^l$ using a layer-wise fully connected layer $f_P$, which act as the auxiliary networks for $\boldsymbol{t}_z$. The results, as displayed in Table 9, were unexpected: $f_P$ led to a decline in performance for both $\boldsymbol{D}^Z$ and $\boldsymbol{D}^Z_l$. Thanks again to Reviewer 3KwZ’s suggestion. This experiment provides an ablation study on $pool_l$.

[1] Papyan, V., Han, X. Y., & Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40), 24652-24663. https://doi.org/10.1073/pnas.2015509117

[2] Ben-Shaul, I. & Dekel, S.. (2022). Nearest Class-Center Simplification through Intermediate Layers. <i>Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022</i>, in <i>Proceedings of Machine Learning Research</i> 196:37-47

[3] Ben-Shaul, I., Shwartz-Ziv, R., Galanti, T., Dekel, S., & LeCun, Y. Reverse Engineering Self-Supervised Learning. In Proceedings of the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023).

[4] Rangamani, A., Lindegaard, M., Galanti, T. & Poggio, T.A.. (2023). Feature learning in deep classifiers through Intermediate Neural Collapse. <i>Proceedings of the 40th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 202:28729-28745 Available from https://proceedings.mlr.press/v202/rangamani23a.html.

[5] Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In NeurIPS, 2018.

Q2

We express our deepest gratitude for Reviewer 3KwZ’s valuable insights. Following this suggestion, we have conducted an experiment to explore the concept of multiple embeddings per class, specifically in a scenario where we can more precisely discern variations within each class. Given that CIFAR-100 is organized into 20 super-classes, each comprising 5 sub-labels, we know exactly how each super-class label (super-label) varies within the 5 sub-labels, thereby obtaining 5 sub-labels per super-label. We tested models trained on CIFAR-100 to make inferences on 20 super-labels, using two methods, 1) assigning the super-label based on the closest proximity of the local feature $\boldsymbol{h}$ to any sub-label embeddings within the super-label (“Super”); 2) averaging the sub-label embeddings to create a unified super-label embedding, which is then used to predict the closest super-label during inference (“Mean”).

The results, presented in Table 7 in the revised script, reveal that the "Super" method outperforms the others, while the "Mean" method does not improve upon the baseline approach (predicting sub-label's super-label as the prediction). We speculate that this is due to the presence of outlier sub-labels. For instance, as demonstrated in Figure 7 in the revised script, the "forest" sub-label is more closely aligned with the "trees" super-label rather than its own "large natural outdoor scene" group, skewing the average embedding for the super-label and causing a bias. Additionally, our results indicate that the model learns semantic relationships more effectively than dataset-defined taxonomies. For example, "chimpanzee" is categorized under "large omnivores and herbivores" alongside "elephant, cattle, camel, and kangaroo," yet it more intuitively aligns with "people," as illustrated in Figure 7.

Clarification on K

Thank Reviewer UJZ4 for the advice regarding $K$. To address this, we want to first clarify that $K$ in our study differs from "$K$" in InfoPro [1]. "$K$" in [1] refers to the number of local modules, each containing multiple layers, whereas $K$ in our paper signifies the number of feature vectors. At the l-th layer, a feature map $\boldsymbol{h}_{map}^l \in \mathbb{R}^{C_h^l\times H_l\times W_l}$ holds $K_l$ (i.e., $H_l\times W_l$) feature vectors, each of dimension $C_h^l$. We adopted $K$ for two main reasons: i) to standardize the format of local features between fully connected (FC) and convolutional (Conv) layers; and ii) to establish the notation for averaging feature vectors over $K$.

For FC layers, local output is a flat vector $\boldsymbol{h_{flat}}^l \in \mathbb{R}^{C_{flat}^l}$. We can unify the expression for $\boldsymbol{h_{flat}}$ and $\boldsymbol{h_{map}}$, by considering $\boldsymbol{h_{flat}}$ as $\boldsymbol{h} \in \mathbb{R}^{C_h^l\times K_l}$, thereby setting $C_{flat}^l = C_h^lK_l$. Furthermore, our findings suggest that for FC layers, averaging over $K_l>1$ yields better results than using the flat vector $K_l=1$, as evidenced in Appendix K.

Regarding $C_D<C_h^l$ and $pool_l$

For a model with L layers, we make sure $C_D = \max\limits_lC_h^l$. When $C_D>C_h^l$, we use pooling to map $\boldsymbol{t}_z \in \mathbb{R}^{C_D}$ to $\boldsymbol{t}_z^l \in \mathbb{R}^{C_h^l}$.

For $pool_l$, we use the average pooling with a padding of 0, a stride of $\lfloor\frac{C_D}{C^l_h}\rfloor$, and kernels of size $C_D - (C^l_h-1) \times \mathrm{stride}$ (torch.nn.AdaptiveAvgPool1d). We employed pooling as it is a simple, but effective method to reduce vector dimensions without resorting to BP. Appendix G reveals that pooling not only simplifies the process but also outperforms using a FC layer $f_P^l$ to map $\boldsymbol{t}_z$ to $\boldsymbol{t}^l_z$.

Regarding Parallel Training

Thank the Reviewer UJZ4 for pointing out the parallel training issue. We agree with Reviewer UJZ4 on the need to address the parallel training scenario. The version presented in the script updates the label embeddings sequentially offering a benefit in terms of memory efficiency (as the allocated memory is freed after each forward pass of the module). Additionally, layer-wise gradients for $\boldsymbol{t}_z$ can be synchronized by averaging across all layers, just as DistributedDataParallel in Pytorch averages batch-wise gradients across devices to synchronize (allowing parallel training). However, we admit that updating from the average gradients may cause training instability because label embeddings receive concurrent error signals from all layers. To address this issue, Reviewer UJZ4 ‘s suggestion (fixed embeddings) can be a solution. However, our experiment illustrated in Figure 5 already exhibits that updating $\boldsymbol{t}_z$ is better. Thus, we propose two alternative strategies more suitable for parallel training: (i) updating the label embeddings only at the last intermediate layer (DC-FL-O), and (ii) employing independent label embeddings for each layer (DC-FL-LD). Both versions still demonstrate competitive performance and memory/parameter efficiency.

We have added this in Tables 3 and 4 in the revised script.

Regarding $\boldsymbol{h}_n^k$

We agree your advice that $\frac{1}{K}\sum^K_{k=1}\langle\boldsymbol{h}_n^k, \boldsymbol{t} \rangle$ is mathematically equivalent to $\langle \boldsymbol{\bar{h}}_n, \boldsymbol{t}\rangle$, where $\boldsymbol{\bar{h}}_n=\frac{1}{K}\sum^K_{k=1}\boldsymbol{h}_n^k$. In fact, we used the simplified expression (what Reviewer UJZ4 suggested) for the mathematical proof in Appendix B. However, please consider that ‘averaging feature vectors first’ could degrade performance due to more severe loss in floating-point precision, which is typically more pronounced when handling smaller numbers. Averaging the dot products, which generally result in higher values, is less prone to precision loss. Conversely, averaging the feature vectors before the dot product can lead to greater precision loss, given that the values in feature vectors are usually smaller. Our intention was to accurately represent the implemented version of the loss. However, considering the simplicity of the expression Reviewer UJZ4 highlighted, we have revised the notation in Eqn. (2) and included a footnote to address the potential loss of floating-point precision.

Regarding InfoPro, auxiliary networks

We are truly grateful to Reviewer UJZ4 for insights and understanding. While Reviewer UJZ4 has overall grasped our work well, we would like to clarify one particular aspect: we did not utilize auxiliary networks $f_\phi$ in our approach. As Reviewer UJZ4 pointed out, LL uses module-wise auxiliary networks and hence has module-level BP to update auxiliary networks. Yet, FL does not use auxiliary networks, being free from BP. Therefore, we can say that DC-FL is a FL framework. Our motivating experiments (in Section 3) starts from what happens if there is no auxiliary network (i.e., FL) in InfoPro [1], comparing $L_\mathrm{feat}$ (auxiliary network-free InfoPro) to $L_{\mathrm{contrast}}$ (InfoPro). This observation (Figure 1) motivated the need for a novel solution that is entirely independent of $f_\phi$ while still delivering competitive performance.

With the above-mentioned respective, though our proposed objective is quite similar to that of InfoPro in that both use contrastive loss, they are different from each other (LL vs. FL). Nonetheless, sincerely consenting with Reviewer UJZ4, we conducted a further comparison with InfoPro.

We have included the results in the following Table 3,4,8 in the revised script. Our method continues to show competitive performance against InfoPro (LL), while being more memory and parameter efficient. These additional findings have been incorporated into the revised manuscript.

[1] Wang et al., Revisiting locally supervised learning: an alternative to end-to-end training, ICLR 2021.