Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective

Q1: The paper suffers from major presentation issues. In particular, a lot of notation/formulation errors lead to unclear results.

A1: Thanks for pointing out these typos and errors! We have made the following corrections in the revised version:

• In Line 74, we have adjusted the domain as $\phi(x):R^{in\times N}\rightarrow R^{D\times N}$.
• We changed $\frac{E_WE_H}{N-1}$ on Line 549 to $-\frac{E_WE_H}{K-1}$, and we also adjusted the conclusion in NC2 from $k$ to $K$.
• In Line 169, we replace $i \in \tau(i)$ with $i\in \tau (k)$，where $\tau(k)$ denotes the samples belonging to the $k$-th class.
• We have adjusted the y-axis of Figure 3(a) for NC1 and changed it to a log scale for NC1.

Q2: The author's claim about DEQ reaching lower loss when compared to explicit layers is not validated in the experiments. Especially, the (train) cross-entropy loss is not illustrated.

A2: Thanks for pointing this out! Since the specific value of the loss does not have practical significance in classification, we primarily use accuracy to compare the performance of explicit and implicit layers. In the revised version, we will include the presentation of loss values.

Q3: Based on the results in Figure 3, it is unclear when the ResNet/DEQ reaches the terminal phase of training.

A3: As discussed in Section 5.1, this early-stop setting was intentional. This is because DEQ models exhibit instability, which becomes particularly pronounced as training progresses, with some samples struggling to converge to a fixed point and requiring more iterations. The previous paper raised this issue (section 3.1 in [1]) and addressed it using early stopping. Here, we follow this standard technique in DEQ.

Q4: The paper states that DEQ models can be memory efficient, but do not provide any numerical results on the computational benefits of employing them.

A4: Memory efficiency is the fundemental properities of DEQ. Extensive experiments have conducted on this aspect when it is proposed. According to table 1-3 in [2], for example, Transformer require 4.8 GB and 6.8 GB memory, while its corresponding DEQ version only requires 1.1 GB. This is because the core of DEQ lies in parameter weight sharing, which saves a large number of parameters. It is not the focus of our paper, so we did not include experiments on this. We primarily explain the performance of DEQ and compare it with explicit neural networks in classification problems.

Q5: The experimental results rely on training with a single random seed which is an incomplete measure of good/bad performance.

A5: We would like to argue that this is not true. We did not use just one seed; instead, we used different random seeds and computed the average and standard deviation of accuracy, which are shown in the Table 1-4.

Q6: The experimental setup with ResNet and the DEQ is unclear. Which part of the ResNet is being used as the image encoder? As of now, it seems like only the last ResNet block is reformulated as the DEQ, and all the previous layers are used as the image encoder. Is this observation correct? If that is the case, then what is the computational overhead of the DEQ layer?

A6: We are afraid that this is not true. In our experiment, we use the layers from the last stage of ResNet, specifically, the last two residual blocks, which comprise four neural network layers collectively structured as DEQ. This will reduce the number of parameter during computation. We apologize for any confusion in the previous statement and have clarified this in our revised paper.

Q7: Based on this setup of replacing only one ResNet block with the DEQ, how does this approach compare to simply fixing the last classifier layer of ResNet as a simplex ETF to address the class-imbalance issue?

A7: Comparing DEQ to simplex ETF on the performance of imbalance learning is not appropriate. DEQ is a broader network representation and is not specifically designed to address imbalanced learning. It can incorporate any network structure as long as there are parameters to be learned. Thus, the last layer with a simplex ETF can also be converted into the DEQ formulation.

Q8: What happens if we replace more than one block of the ResNet with a DEQ layer?

A8: Currently in our paper, we have already combined multiple consecutive residual blocks into DEQ. A distinctive feature of DEQ is its capability to convert arbitrary neural networks into the form of an implicit network regardless of length.

Q9: How sensitive is the DEQ layer to learning rate and batch size?

A9: In our experiments, DEQ is not sensitive to batch size, but it is relatively more affected by the learning rate. A larger learning rate causes fluctuations in DEQ training to occur earlier. Due to the limited theoretical research on DEQ, these issues have not been thoroughly addressed and studied. Currently, [3,4] have explored different learning rate settings and compared their outcomes. [5] has proven that with sufficiently small learning rates, the loss function can converge at a linear rate. However, though we believe these are not within the scope of our paper, we will add the analysis into the related work part of our revised version.

ref:

[1] Stabilizing Equilibrium Models by Jacobian Regularization, icml 2021

[2] Deep Equilibrium Models, nips 2019

[3] Deep equilibrium networks are sensitive to initialization statistics, icml 2022

[4] Positive Concave Deep Equilibrium Models, icml 2024

[5] Global Convergence Rate of Deep Equilibrium Models with General Activations, arxiv: 2302.05797

Q1: My major concern of this paper is its contribution and significance. Since the discovery of minority collapse, there has been tons of literature on how to mitigate the minority collapse. Therefore, my opinion is that neural collapse analysis can provide limited insight about the advantage of DEQ over standard NN. I would suggest the authors compare the DEQ and standard NN with some standard techniques such as reweighting, and confirm that the advantage indeed have practical influence.

A1: Thanks for your comment! We would like to clarify that our goal was not to use DEQ to mitigate minority class collapse but rather to leverage NC analysis to explain the performance of DEQ. We had employed reweighted CE to test both explicit NN and DEQ, observing that their accuracies both improved. Importantly, their difference remained similar. We shows the case of Cifar-10, $K_A=3$ and $R=10$, the results are as follows:

Additionally, our work primarily focuses on theoretical aspects since current DEQ research heavily relies on empirical evidence for its effectiveness and performance and lacks theoretical grounding.

Furthermore, we would like to argue that NC do not brings limited insight to DEQ. A significant similarity between DEQ and NC research is their model-agnostic nature. By using NC tools, we expand on the internal computational processes of both explicit and implicit structures, providing explanation of why DEQ might outperform explicit NN under certain conditions, i.e., we analyze and compare the differences between $W_{DEQ}$ and $W_{EX}$ in their application. Our findings demonstrate that the extracted features of DEQ align more closely with the vertices of a simplex ETF and exhibit better alignment with classifier weights under specific conditions.

We quantify the benefits of DEQ’s multiple forward iterations, offering a valuable theoretical supplement to the existing research. Thus, we believe our paper contributes to the community by addressing a theoretical gap and providing a valuable addition to the existing research.

Q2: The experimental results are limited to ResNet-18 only, I would encourage the authors to perform more extensive evaluation on various architectures to support the finding.

A2: Thanks for your advice! We conducted a new experiment using MobileNet as the network backbone. The results are as follows:

$K_A=3$, Cifar-10

Due to character limits, we omitted the $\pm$ and variance term and selected a specific set of experiments to showcase. Additionally, we incorporated more results into our revised paper. The conclusion we found that is, the differences between Explicit NN and DEQ consistently persisted across different backbones.

Q3: The proof needs to be better organized. It will be helpful if the authors can provide an outline of proof in the appendix, restate the main result and break it into several propositions, and add some high level intuition of why DEQ outperforms standard NN.

A3: Our proof sketch divides into two cases: in the balanced scenario, we utilize inequalities to iteratively establish conditions under which each equation holds, thereby deriving properties of NC. In the imbalanced case, we analyze conditions under which the majority and minority classes achieve their minima, and integrate these insights to compare the performance of Explicit NN and DEQ within the NC framework.

We have re-orginzed our prove and made the following modification:

• We add a table of contents for our appendix.
• In the proof of balanced learning, we have rewritten Case 1 (explicit NN) in Line 528 and Case 2 (DEQ) in Line 552, emphasizing the three properties of NC for comparison. We explain that DEQ performs slightly better than explicit NN because of its lower bound on the loss function (theorem B.3).
• In the proof of imbalanced learning, we have added more detailed explanations about why DEQ performs better under mild conditions, which is mainly because the forward fixed-point iteration increases the number of learning iterations for samples in the minority class.

Q4: Neural Collapse solutions do not have full rank in general, would that bring issues when writing down the inverse of gradient in equation (3)?

A4: We don’t think so. Equation (3) describes the forward solving process of DEQ, where $x$ is the input to the DEQ layer, and $z^\star$ the output of this layer through fixed-point iteration. The rank in (3) differs with that in NC. Besides, the matrix $B^{-1}$ is not directly obtained through inversion in practice, but rather approximated using BFGS.

Q5: In your theoretical analysis, it was assumed that the DEQ is linear with a specific expansion form. The authors should highlight this with a hold out assumption and discuss when it can be satisfied.

A5: Thanks for your suggestion! We would like to clarify that the assumption of Linear DEQ, as used and discussed in [1], is a common approach in DEQ studies. This assumption posits that a linear function is used during the fixed-point iteration to compute the equilibrium point $z^\star$. We will add this assumption more clearly in the final version.

Q6: In particular, what is $W^i_{DEQ}x$ in line 162? Are they being optimized in the programming? In the current form it seems to reduce to simply a linear function $Wx$.

A6: $W_{DEQ}$ represents the parameter being optimized, and $W_{DEQ}^i$ refers to its $i$-th power. Its presence arises from the necessity of repeated iterations during the forward solving process. It cannot be seen as a simple linear function $Wx$.

ref: [1] Deep equilibrium networks are sensitive to initialization statistics, icml 2022

Q1: The analysis is limited to simple imbalanced scenarios and DEQ models, restricting the generalizability of the findings to more complex real-world situations.

A1: Thanks for your comment! Currently, our work primarily focuses on theoretical aspects, and discussing real-world issues will be a part of our future work. However, our research inherently addresses the model's generalization capability.

Through our theoretical proof, we have established that DEQ's NC2 and NC3 demonstrate superior properties than explicit neural network under mild conditions. Specifically, we discovered that features extracted by DEQ exhibit closer alignment with the vertices of a simplex ETF and show better conformity with classifier weights under specific conditions. As a result, feature separability of DEQ is enhanced compared to explicit layers under some mild conditions, which can potentially lead to stronger generalization performance in downstream tasks.

Q2: While the paper provides valuable theoretical insights and experimental validation in section 5, there may be a need for more in-depth quantitative analysis like statistical analysis to further support the claims and conclusions drawn.

A2: Thanks for your advice. We have plotted the values of NC1 - NC3 in Figure 3 to Figure 5 and their statistical representations are included in Appendix Section A. The NC cases shown in Figure 3(a) and Figure 3(b) precisely support the conclusions in section 3 and section 4 respectively. Additionally, we have also calculated the mean and standard deviation of training accuracy, which are presented in the Table 2 - Table 4.

Q3: The lack of broader experiment for $K_A\neq 3$ in Table 3 may hinder a comprehensive comparison of DEQ with Explicit NN.

A3: We did conduct the experiments with $K_A$ values of 5 and 7, and the results were presented in the appendix (page 26).

Q1: The smaller lower bound of loss function may cause overfitting, which may affect the performance in the test phase. Add discussions on this part can be helpful.

A1: Thanks for your valuable suggestion! We agree with you that lower bound of loss function can lead to overfitting.

We would like to claim that we did employ some manual setup methods to avoid overfitting. For example, we implemented an early stopping mechanism (sec 5.1) to prevent instability in DEQ and halt the training process before the model overfits the training data, thereby reducing the risk of overfitting.

Additionally, the solving process of DEQ can be configured with specific thresholds to terminate at appropriate points, and a manual iteration limit can also be set.

Furthermore, in our experiments, we also employed the Jacobian regularizer proposed in [1], which penalizes overly complex models and discourages the model from fitting the training data too closely.

We have added the detailed discussion into our revised paper.

Q2: Maybe add some experiments on dataset other than Cifar can make the conclusion more persuasive.

A2: Thanks for your comment. Following the settings in previous DEQ papers, we have also included results on additional datasets, such as SVHN and MNIST, the results are shown here:

SVHN, $K_A=3$:

SVHN, $K_A=5$:

SVHN, $K_A=7$:

And the trend also resembles as Cifar dataset. In the MNIST dataset, the results of DEQ and Explicit NN are quite similar due to the dataset's simplicity, so we do not present them here.

ref：

[1] Stabilizing Equilibrium Models by Jacobian Regularization, icml 2021