Enhancing Accuracy and Parameter Efficiency of Neural Representations for Network Parameterization

PART 2/2

Intuition in Smoothness Analysis Thank you for the suggestion to analyze generalization explicitly. We evaluated the generalization gap, defined as the absolute difference between train loss and test loss (measured by cross-entropy loss). As shown in the table, our method demonstrates a generally decreasing trend in the generalization gap across progressive rounds. For example, starting from the original network with a gap of $1.28485$, the gap decreases to $1.24161$ by Round 5.

We can further verify the generalization of our predicted weights using downstream tasks, i.e., transfer learning to a new dataset, CINIC-10. We compared our predicted weights with original weights under two scenarios: 1) Fine-tuning all layers: adapts the model completely to the new task, 2) Fine-tuning only the linear layer: highlights the generalizability of convolutional weights. The results show that our predicted weights (Hidden 360, CR < 1) consistently outperform the baseline pre-trained weights in both scenarios, achieving higher accuracy. Additionally, using predicted weights from progressive reconstruction (Hidden 680, CR > 1) further improves generalization slightly. Overall, these results confirm that the proposed method generalizes well to downstream tasks.

Figure 1, L64, L251, L264-266, bolds in Table 1 Thank you for your suggestion and for pointing out the typos. We will revise our manuscript accordingly. Regarding lines 264-266, as our predictor focuses on predicting the weights in the convolutional layers, we did not use the term "decision layer" directly. Instead, we referred to a later layer closer to the decision layer, as it contributes more significantly to the decision-making process than early layers. We will update lines 260-266 as follows:

Interestingly, we find that this two-stage optimization leads to significant differences in the early layers of the network, while still matching the later layers. This is intuitive, as it is well known that the decision rules typically emerge in the later layers of a deep network. While the larger differences in the early layers may seemingly compromise reconstruction fidelity, this separate training strategy facilitates more effective integration of distillation into the network parameterization, as evidenced by significant improvements (red bars in Figure 4(c)).

Reconstruction cost vs. inference cost As expected, the reconstruction cost will be significantly higher than the single inference cost. However, the intention is not to do the reconstruction on the fly for a given instance but rather reconstruct first before doing a large number of inferences, therefore, amortizing the cost of the reconstruction. More importantly, our primary aim is to investigate the trade-off and improve upon the state-of-the-art weight prediction methods, e.g., NeRN, which itself is an early work of a rather new field (weight prediction), so the goal is more oriented toward academic exploration rather than practical deployment.

Other layer types Thank you for your insightful comments and references. While fully connected layers can indeed be interpreted as 1×1 convolutional layers, making them compatible with the predictor network, we have not explored this implementation in our study for the sake of simplicity. We acknowledge that applying smoothing strategies or training an implicit neural representation on fully connected layers could offer interesting insights.

Train from scratch with weight smoothing To simplify the task, NeRN initially explored a regularization-based approach to promote weight smoothness by explicitly adding a loss term during training of the original network. While this approach effectively encourages smoothness in the network, it often leads to slightly inferior performance on the original task. Furthermore, increasing the regularization factor significantly degrades the original network's performance.

PART 1/2

Limited performance / KD baseline / High-Performing Teacher To address the concerns listed in the title, we designed two experiments to highlight the advantages of our proposed training strategies, which cannot be achieved with the baseline NeRN. As suggested, we directly compared our predictor network with a distilled network trained using conventional KD. In this setup, a student network is trained from scratch using ResNet50 teacher guidance. The results show that our decoupled training achieves an accuracy of 73.95%, outperforming the conventional KD approach, 73.60%. However, the advantages of the proposed method are not limited to this performance gain; it also allows for further iterative improvement in each case.

First, we applied the progressive-reconstruction process by targeting the distilled network (73.60%) and proceeded to the second round. The first round of progressive-reconstruction resulted in improved performance beyond the target accuracy, and the second round achieved further enhancement. These results demonstrate that our method is not constrained by the initial training techniques and can effectively refine an already optimized baseline.

We also investigated whether progressive-reconstruction can enhance our best-performing model (achieved through decoupled training with a high-performing teacher, as shown in Table 4). The results confirm that the first round improves upon the target accuracy (73.95%), and the second round achieves additional gains. These results also emphasize the effectiveness and flexibility of the proposed recipe in achieving further performance improvements.

Line 82-84, limitation on data usage It is correct that both baseline NeRN and our decoupled training need the original task's data (through the iterative refinement with large predictor use only reconstruction loss so access to the original data is not required). Despite that fact, we conducted an experiment that addresses the challenge of operating in a completely data-free environment. We employ uniformly sampled noise as input data for both methods, and our results demonstrate its effectiveness compared to the baseline NeRN even in the absence of meaningful data.

Significantly degraded performance of quantized predictor We use the term "orthogonal" to indicate that quantization can be directly applied to the predictor network if further benefits are desired. While quantizing the predictor leads to a significant degradation in performance (e.g., from 70.84% to 51.72% on CIFAR100), this outcome is explainable. The degradation is not necessarily due to the brittleness of predictor, but rather it is because the predictor is the weight generator. Even small changes in the predictor's outcome (weights for reconstructed network), such as those introduced by quantization, can significantly impact the performance of the reconstructed model. In contrast, quantizing the original network (e.g., ResNet) has less impact because it operates on a fixed structure rather than dynamically generating weights. Despite the degradation observed in the quantized predictor, our method (70.84%), using a predictor of the same size as the quantized ResNet56, achieves better performance than its counterpart (69.65%).

We also want to clarify that the primary objectives of our work differ from quantization methods. While improved model compression is one of the benefits offered by our approach, the core contribution lies in effectively performing NeRN-style reparameterization through our novel progressive-training and decoupled-training strategies.

Thank you for your response to my comments. However, I believe my concerns remain:

(i) The revised introduction and the abstract still highlight the performance gain from weight reconstruction, although this gain is minimal and barely exists. This is also seen in the added results of the rebuttal, where the accuracy increases by less than 1% on CIFAR10 after multiple iterations. This is also seen in the generalization experiments where the gain is less than 0.3% on transfer learning. Moreover, the rebuttal did not address the ImageNet case which displays an opposite trend (on a larger and more challenging dataset). To summarize, I believe the accuracy gain claim is not well supported, and the writing should reflect that.

(ii) The higher performing teacher still seems unfair. Distilling the teacher still seems like a better option.

I choose to keep my score.

Presentation We appreciate the encouragement to provide a stronger motivation for our research and to highlight the appeal of the original NeRN method. In response, we have revised our introduction (lines 32-40), added a clearer explanation of the multi-objective loss function (lines 44-46), and removed ambiguous terms (lines 47-49).

Generalization to Transformers While our current work focuses on effectively performing NeRN's reparametrization on CNNs, we recognize the importance and value of extending NeRN to other architectures. Unfortunately, due to time constraints, we were unable to perform the extensive research to extend NeRN's application to transformers in this submission. However, we have carefully considered the challenges and requirements and will briefly discuss them here.

Generalizing NeRN to transformers would require significant modifications to its current setup, primarily in designing a suitable coordinate system and addressing the larger size and complexity of weight matrices in transformers. For example, the coordinate system would need to capture aspects such as layer index, head index, weight type (query, key, value), and block indices for submatrices (if decomposed weight matrices into smaller submatrices). These additions would maintain NeRN's core principle of functional weight representation but would introduce new challenges during training due to the complex structure.

Additionally, although NeRN does not impose architecture-specific assumptions, it relies on INR, which typically assumes that input coordinates represent a smooth and continuous space and that the learned function exhibits some level of smoothness. However, in transformers, the coordinates are discrete, and the corresponding target weight lacks continuity, making convergence and accuracy more challenging. NeRN addresses this for CNNs by introducing permutation-based smoothness, which promotes kernel smoothness by reordering pre-trained weights without altering their values. However, in transformers, the increased size and complexity of both the coordinate system and weight matrices would require further innovations to simplify the prediction task and ensure smooth learning.

Beside NeRN there are other approaches that aim to predict transformer weights, such as diffusion-based weight prediction methods. These can potentially be adopted as the predictor network to replace NeRN, whereas our proposed iterative weight reconstruction could potentially still be applied.

Smaller Issues

• Adding more citations: We added a short discussion of semantic representations to the related works section, including references to the suggested literature.

  Semantic representations of neural networks encode meaningful, interpretable features aligned with human-understandable concepts (references). However, our implicit representations encode information in a distributed and flexible manner, capturing complex patterns and relationships.

• Inception-like: We acknowledge the confusion and have changed the term to "recursive" for clarity.

• Typos and naming (line 92, line 252, NeRN): We have revised these typos and replaced 'baseline' with 'NeRN'.

• Lines 264-266 are not very clear: We have revised lines 264-266 in the updated manuscript to improve clarity.

• Clarity section 3.3 and Figure 5: Both arrows in Figure 5 should indeed be red, as they represent the decoupled training process described in Section 3.3. Specifically, Table 3 shows results for a parameter-efficient predictor (CR<1) with logit distillation, while Table 4 shows results for a larger predictor (CR>1) using the same method. The blue arrow represents progressive reconstruction, which can further refine a decoupled model by targeting this high-performing model. To clarify, we have provided additional results in the following table. We hope this explanation and added results clarify the use of colors in the figure.

• To evaluate whether progressive reconstruction enhances smaller NeRNs, we used a compact NeRN model with 70.84% accuracy ("Ours" in Table 2) as the target network. As shown in the results, applying a second round of progressive reconstruction (Hidden 680, CR>1) improved performance beyond the target accuracy (70.84%) and brought it closer to the original accuracy (71.37%). This suggests that smaller NeRNs can benefit from progressive reconstruction without directly targeting the original weights.

Downstream tasks To evaluate the generalization of our predicted weights, we designed a transfer learning experiment comparing the original weights (baseline pretrained on CIFAR-100) and our predicted weights (trained on CIFAR-100) when applied to the CINIC-10 dataset. We tested two scenarios: 1) Fine-tuning all layers to fully adapt the model to the new task, and 2) Fine-tuning only the linear layer to highlight the generalizability of convolutional weights. In both cases, models were fine-tuned for 10 epochs, while the 'From Scratch' model was trained for 150 epochs. The results show that our predicted weights (Hidden 360, CR < 1) consistently outperform the original weights in both scenarios, achieving higher accuracy. Additionally, using predicted weights from progressive reconstruction (Round 5, Hidden 680, CR > 1) further improves generalization slightly. Overall, these results confirm that the proposed method generalizes well to downstream tasks.

Progressive enhancement of high-performing networks We are not entirely sure about the interpretation of the question. Please provide clarification if you need more information. We think you are asking whether the progressive-reconstruction can generalize to a higher-performing network built upon the original network, such as those enhanced through techniques like knowledge distillation. To address this, we designed the following experiments: Instead of applying progressive-reconstruction to the original network (71.37%), we used a distilled network (73.60%) where knowledge distillation with a high-performing teacher was used while training the original network. This experiment is crucial because it demonstrates that our method is not constrained by the training techniques. The results confirm that progressive-reconstruction, when applied to the distilled network, also achieves better performance than the target accuracy (73.60%). This aligns with the findings reported in Table 1, which illustrates that our method effectively improves performance, even when targeting a more optimized baseline.

Extend beyond CNN-based architecture While our current work focuses on effectively performing NeRN's reparameterization on CNNs, we recognize the importance and value of extending NeRN to other architectures. Unfortunately, due to time constraints, we were unable to show an example of NeRN's application to transformer architecture in this submission. However, we have carefully considered the challenges and requirements.

Generalizing NeRN to transformers would require significant modifications to its current setup, primarily in designing a suitable coordinate system and addressing the larger size and complexity of weight matrices in transformers. For example, the coordinate system would need to capture aspects such as layer index, head index, weight type (query, key, value), and block indices for submatrices (if decomposed weight matrices into smaller submatrices). These additions would maintain NeRN's core principle of functional weight representation but would introduce new challenges during training due to the complex structure.

Additionally, although NeRN does not impose architecture-specific assumptions, it relies on INR, which typically assumes that the input coordinate system represents a smooth and continuous space and that the learned function exhibits some level of smoothness. However, in transformers, the coordinates are discrete, and the corresponding target weight lacks continuity, making convergence and accuracy more challenging. NeRN addresses this for CNNs by introducing permutation-based smoothness, which promotes kernel smoothness by reordering pre-trained weights without altering their values. However, in transformers, the increased size and complexity of both the coordinate system and weight matrices would require further innovations to simplify the prediction task and ensure smooth learning.

Beside NeRN there are other approaches that aim to predict transformer weights, such as diffusion based weight prediction methods. These can potentially be adopted as the predictor network to replace NeRN, whereas our proposed iterative weight reconstruction could potentially still be applied.

Alternative methods exist for weight smoothing There are different methods to achieve weight smoothing, either during model training (e.g., weight decay, dropout) or post hoc approaches directly applied to pre-trained model weights (e.g., frequency filtering, modulate singular value). Our proposed approach focuses on modifying pretrained weights. In Appendix A, we explore alternatives for directly applying weight smoothing, either through modulating singular value or Fourier transformation and filtering. The downsides of these methods are that they are very sensitive to the right parameter setting or require checks against the validation dataset. Regularization-based methods usually come with a generalization-accuracy trade-off. Our method doesn't have these restrictions and we show below that we can iteratively increase accuracy of already-optimized models just using the reconstruction loss.

Reducing model storage is not as big of a concern as reducing the memory Training a predictor network doesn't necessarily mean requiring more memory. With a small predictor, we can learn through batch training of the original weights and, therefore, require significantly less memory compared to the original model (same for reconstruction of original weight). Compression is just one of the potential benefits of the proposed method. The reconstruction setup with a larger predictor aims at improving model performance rather than compression. During inference time, we will have a model of the same memory footprint but with improved performance. Moreover, we show the reconstructed weight has other benefits, such as better generalization for downstream tasks (see response to reviewer GbWd).

Problem setup in this paper is not clear We will update the paper to clarify the points raised by the reviewer. Specifically, we always use the test split for all model evaluations. Also for the reconstruction tasks, the predictor is trained to predict the model weight directly so no data split is involved. Only the training split is used for the knowledge distillation setup. We also updated the caption of Figure 1 and the description in the text of section 3.1.

Figure 1 (left) is confusing, and the slightly increased accuracy (Figure 1 right) may be due to variance We agree that the left figure is confusing, so we will remove it from the final version and have adjusted the text describing the figure. We also changed the caption of the figure to describe the meaning of the dots, i.e., representing results from predictors with different hidden layer sizes. Additionally, we provide the mean and standard deviation across three runs, demonstrating that the observed behavior is not due to variance in the results.

Comparison with distillation We designed three experiments to highlight the advantages of our proposed training strategies. First, we compare our predictor network with a distilled network trained using conventional KD. Here, a student network is trained from scratch using teacher guidance from ResNet50. The results show that our decoupled training achieves an accuracy of 73.95%, outperforming the conventional KD approach, 73.60%. However, the advantages of the proposed method are not limited to this performance gain; it also allows for further iterative improvement in each case.

Second, we applied the progressive-reconstruction process on the distilled network for two rounds. Both result in improved performance beyond the target accuracy. This behavior demonstrates that our method is not constrained by the initial training techniques and can effectively refine an already optimized baseline.

Third, we also investigated whether progressive-reconstruction can enhance our best-performing model (achieved through decoupled training with a high-performing teacher, as shown in Table 4). The results confirm improvements over the target accuracy in each round, emphasizing the effectiveness and flexibility of the proposed recipe.

Low reconstruction error We understand that "low reconstruction error" may seem broad in general contexts. In our study, reconstruction error serves as an important analytical metric, as it is directly related to the performance of the reconstructed network. We added more context in Section 3.1 and Figure 1. For example, achieving zero reconstruction error would theoretically result in performance identical to that of the original network. While one might expect performance to consistently decrease as reconstruction error increases, we observed cases where the reconstructed network's performance exceeds the original network's performance. To clarify, we measured reconstruction error and the corresponding performance computed by 3 runs across seven different predictor hidden sizes, with each point on the graph representing the original performance and a specific hidden size from left to right: original, 750, 680, 510, 360, 280, and 220.

Reasoning/intuition is lacking To explain the observed performance improvement, we introduce the $S_{ratio}$ metric for two key reasons: 1) it clearly distinguishes the behaviors of predicted and original weights, and 2) it supports the weight smoothing hypothesis, where progressive-reconstruction promotes weight smoothing, as evidenced by higher $S_{ratio}$ values (indicative of lower-frequency components), thereby leading to performance improvement. Figure 2 shows that Round 1 weights exhibit higher $S_{ratio}$, particularly in the later layers (i.e., closer to the decision layer). This trend continues across additional rounds of progressive-reconstruction (Figure 3 (b), up to Round 5). This suggests that progressive-reconstruction smooths weights by suppressing lower singular values.

To further validate this hypothesis, we conducted additional analyses to explicitly test the relationship between weight modulation and performance improvement. 1) Frequency modulation (Figure 7): Applying a low-pass filter to the weight matrix demonstrated its impact on performance. 2) Singular value modulation (Figure 8): Scaling down less significant singular values showed their contribution to performance. 3) Singular values vs frequency (Figure 9). This analysis is particularly important as modulating singular values does not inherently guarantee the removal of high-frequency components. It shows that the weights after low-pass filtering exhibit higher $S_{ratio}$ values, suggesting a potential link between frequency and singular value-based modulations. We believe our analysis provides important reasoning and intuition behind the performance improvement of the proposed method. This aligns with findings from many prior works, further supporting the validity of our explanation.

Figure 5's position We moved Figure 5 to the beginning of the section.

Extend beyond CNN-based architecture Our current work focuses on effectively performing NeRN's reparameterization on CNNs. We have carefully considered the challenges and requirements for extending NeRN to transformers. Generalizing NeRN to transformers would require significant modifications to its current setup, primarily in designing a suitable coordinate system and addressing the larger size and complexity of weight matrices in transformers. For example, the coordinate system would need to capture aspects such as layer index, head index, weight type (query, key, value), and block indices for submatrices (if decomposed weight matrices into smaller submatrices). These additions would maintain NeRN's core principle of functional weight representation but would introduce new challenges during training due to the complex structure.

Additionally, although NeRN does not impose architecture-specific assumptions, it relies on INR, which typically assumes that the input coordinate system represents a smooth and continuous space and that the learned function exhibits some level of smoothness. However, in transformers, the coordinates are discrete, and the corresponding target weight lacks continuity, making convergence and accuracy more challenging. NeRN addresses this for CNNs by introducing permutation-based smoothness, which promotes kernel smoothness by reordering pre-trained weights without altering their values. However, in transformers, the increased size and complexity of both the coordinate system and weight matrices would require further innovations to simplify the prediction task and ensure smooth learning.