Differential Model Scaling using Differential Topk

Dear reviewer,

Thank you for your comments. We have addressed all of them below.

W1 "The image classification baselines are too weak. baselines should have 90%+ top-1 accuracy."

Thanks for your suggestion. We agree with you that a strong baseline is very important.

However, in this paper, our main target is to compare the search efficiency of our method and other search methods. Hence, we select our baselines from the related work for a fair comparison. All of the most related work, including ScaleNet, ModelAmplification, and so on, use EfficientNet as their baselines.

Since we think our baselines are suitable for our target rather than too weak. We will consider using stronger baselines in our future work if we need to build a SOTA image classification model.

W2 "Shown in Table 6, the performance gain is marginal.", Q1 "accuracy vs MACs Plot with Table 6. The performance gain seems marginal from the table."

Sorry for the confusion. There are two reasons why our performance gain seems 'marginal'.

1. Compared with the Multi-Shot and One-shot methods, our methods have a much lower search cost, up to several tens, even hundreds of times lower. Therefore, our performance gain is not marginal because we may use much lower search costs to achieve a higher performance.
2. Compared with zero-shot NAS methods, they all use much stronger training recipes than ours, such as distillation and mixup. Therefore, the real improvement gain is more significant than the table shows.

Here, we provide a new table to make the comparison clearer (We only list some methods that may be confusing; please refer to Table 5 in our revised paper for full comparison.)

(+) means the model is trained using much stronger training tricks, such as distillation and mixup. As some methods do not report their search costs, we use "High" and "Low" to provide a rough comparison.

Thanks for your comments. We updated the table (Note the index is 5 in our revised paper, while it's 6 in the original paper). We also added an accuracy vs MACs plot to the paper, notated as Figure 3 in our revised paper.

Q3 "explain how this loss_resource is designed.

Resource constraint loss is widely used in the field of NAS and pruning [1,2]. We just follow the common practice in the field.

Take constraining the number of parameters of a fully connected layer as an example. Suppose the layer has $f_{in} \in R$ input features and $f_{out} \in R$ output features. The structure parameters for its input features and output features are $a_{in} \in [0,1]$ and $a_{out} \in 0,1$, respectively. The resource constraint loss for the layer is defined as $f_{in} \times a_{in} \times f_{out} \times a_{out}$. The resource constraint loss of the total model is the sum of all parameter losses of all layers.

We also detailed the resource constraint loss in Appendix A.1.2. Please refer to it for more details.

[1] Li, Yanyu, et al. "Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization."
[2] Gao, Shangqian, et al. "Disentangled differentiable network pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Thanks for your comments again. If you have any further comments, please let us know. Besides, we want to emphasize the value of our method. We build a general and flexible NAS method. It can improve various tasks stably with a reasonable search cost, while prior methods are usually designed for a specific task and use much more resources. Our method pushes the boundary of NAS and makes it more practical for real-world applications. Therefore, we think our method is valuable and worth publishing. I hope you can raise our paper's score if our responses are convincing. Thank you very much.

Dear reviewer:

Thanks for your constructive comments. We have posted our responses to your comments. We expect your feedback about whether our responses address your concerns, or if you have any further questions. We are glad to answer them and improve our paper.

Best,

Authors

Dear review:

Thanks for your constructive comments. We address your comments as follows:

W1: "Different element importance methods are not studied. Some comparisons should be presented to underscore the DMS method."

Thanks for your suggestion. Here, we provide a new ablation study to show the influence of element importance.

"Index" means assigning an importance value to each element according to its index in the sequence statically. For example, we assign 0 to the first element, and 1.0 to the last.

Obversely, Taylor importance works better than index importance, as it is able to detect the importance of each element dynamically. Moving averaging further makes Taylor importance more stable.

We also added the ablation study to Appendix A.5.3 in our revised paper.

W2: "More types of cost losses should be considered, such as latency or memory cost. Latency is a superior indicator compare to FLOPs."

Thanks for your suggestion. In this paper, we use two different resource constraint losses, including Flops loss and number of parameters loss.

In this paper, our main target is to show the high search efficiency of our method rather than design a model for a specific platform. Therefore, we think any resource indicator is acceptable as long as we use the same resource indicator as our baselines for fair comparison.

Specifically, for memory cost constraint, we think it's similar to the number of parameters. Hence, we think there is no need to add a memory cost constraint to our paper.

For latency constraint, We agree that latency is a useful indicator. However, the relation between latency and structure hyperparameters is not easy to model. It means latency is not easy to be converted to a differentiable loss. We think it's also an interesting research direction. We will take it in future research.

Therefore, we think the constraints on flops and the number of parameters are enough to evaluate our method, and other constraints should be left for future research, such as some work that focuses on a specific hardware platform.

W3: "As far as I know, gumbel top-k method is also differentiable, why you develop a new differentiable top-k methd?"

We are sorry for the confusion. Gumbel top-k [1,2] is completely different from our method. It may be confusing because we use the same term, "top-k", in our paper.

Gumbel top-k is learned to sample 'k' different elements from a set of n elements, where 'k' is a constant. Our differential topk is learning the 'k' itself, which is a learnable parameter. Gumbel top-k cannot be used to search the width and depth of a network, while our method can.

We also talk about another method, "Gumbel softmax", which is also usually used in pruning and NAS [3]. "Gumbel softmax" is able to work as a learnable gate to select elements for width and depth. However, as we described in our paper, it models depth and width hyperparameter searching as an element selection problem, making it hard to optimize.

[1] Tan, Haoxian, et al. "Mutually-aware Sub-Graphs Differentiable Architecture Search." arXiv preprint arXiv:2107.04324 (2021).
[2] Li, Jiaoda, Ryan Cotterell, and Mrinmaya Sachan. "Differentiable subset pruning of transformer heads." Transactions of the Association for Computational Linguistics 9 (2021): 1442-1459.
[3] Herrmann, Charles, Richard Strong Bowen, and Ramin Zabih. "Channel selection using gumbel softmax." European Conference on Computer Vision. Cham: Springer International Publishing, 2020.

W4: "The open source of the code will help to understand the paper."

Considering the double-blind review policy, we can not open-source our code before the paper is accepted. We will open-source our code after the paper is accepted.

Q5: "Besides, in p.5, the authors demonstrate that "Intuitively, $c_i'$ indicates the portion of $c$ values larger than $c_i$ Here should be "smaller" instead of "larger".

Thanks for your correction. We have updated the paper to correct it.

Thanks for your effort in reviewing our paper. We have addressed all your comments in above. If you have any further questions, please let us know.

Dear reviewer:

Thanks for your constructive comments. We have posted our responses to your comments. We expect your feedback about whether our responses address your concerns, or if you have any further questions. We are glad to answer them and improve our paper.

Best,

Authors

W5.1 "The paper's ablation study is not comprehensive enough to demonstrate the advantages of the proposed method. The paper only compares the differential topk with a uniform scaling baseline, but does not compare it with other model scaling methods, such as compound scaling or progressive scaling."

Thanks for your suggestion. We don't compare with compound scaling and progressive scaling in our ablation study because we have already compared them with our method in our main experiments. Where EfficientNet is a compound scaling method, and ModelAmplification is a progressive scaling method. Besides, we have also compared our method with other model scaling methods, such as differential scaling (JointPruning). Therefore, we think it is not necessary to compare with other scaling methods in our ablation study.

W5.2 " The paper also does not show how the differential topk performs on different network layers, such as convolutional layers or attention layers."

We apologize for our unclear description. We apply our differential topk to different layers by multiplying the mask, outputed by our differential topk operators, with the input to the layer.

For convolutional layers, suppose the input is $X \in \mathbb{R}^{B \times C \times H \times W}$, and the mask is reshaped as $m \in \mathbb{R}^{1 \times C \times 1 \times 1}$, $X \times m$ works as the new input to the layer.

For an attention layer, we search the head dims of q k v and the number of heads. Suppose our supernet has $H$ heads and $D$ dims in each head. We have a mask for qk head dim with $m_{qk}\in R^{1\times 1 \times 1 \times D}$, a mask for v head dim with $m_{v}\in R^{1\times 1 \times 1 \times D}$, and a mask for number of heads $m_{head}\in R^{1\times H \times 1 \times 1}$. Suppose the sequence length is $L$, and the q k v for self-attention is $Q,K,V \in R^{B \times H \times L \times D}$. We compute the output of the self-attention by $softmax(\frac{Q'K'^T}{\sqrt{}})V'$, where $Q'=Q\times m_{qk}\times m_{head}, K'=K\times m_{qk}\times m_{head}, V'=V\times m_{v}\times m_{head}$

We added above content to Appendix A.1.1

Q1 "In Section 3.1.3, you use a moving average to update the layerwise mask. What is the motivation and benefit of this technique? How does it affect the convergence and stability of the optimization?"

The moving average is used to make element importance measures more stable. As shown in the following table. With more stable element importance, we can get a better result.

Taylor importance with moving average is a common technique in the pruning field. Please refer to [1,2] for more details. We also added the ablation study to Appendix A.5.3 in our revised paper.

[1] Molchanov, Pavlo, et al. "Importance estimation for neural network pruning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[2] Humble, Ryan, et al. "Soft masking for cost-constrained channel pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Q2 "In Section 3.1.3, you adopt an L1-norm regularization term for the channel-wise mask. Why did you choose this norm over other alternatives, such as L2-norm or entropy? How does the choice of norm influence the sparsity and diversity of the channel-wise mask?"

Sorry for the confusion. We did not use L1-norm regularization term in the paper. We mentioned that "L1-norm" can be an alternative as an importance measure method. The "L1-norm" refers to a well-known channel importance measure method, which computes the L1-norm of each filter as the importance score. Please refer to [1] for more details.

[1] Li, Hao, et al. "Pruning filters for efficient convnets." arXiv preprint arXiv:1608.08710 (2016).

Thanks for diving into our paper and providing the constructive comments. We have revised our paper according to your suggestions. Please feel free to contact us if you have any further questions. Furthermore, we want to emphasize the value of our work. Our method. It's a general and flexible NAS method that can be applied in real-world scenarios. It improves performance stably, has only two hyperparameters that need turning, and costs much fewer resources than other NAS methods. Hence, we think our work is valuable and deserves to be published. If our responses have addressed your concerns, we would appreciate it if you could consider raising the score of our paper. Thank you for your time and consideration.

W4 "The paper introduces too many hyperparameters for the differential topk, such as the temperature, the sparsity ratio, and the regularization coefficient. The paper does not provide a systematic analysis of how these hyperparameters affect the performance and the stability of the method. It is also unclear how to choose these hyperparameters for different tasks and architectures."

Sorry for the confusion. We think our hyperparameters are not too many, as there are only two hyperparameters that need to be turned for different tasks and architectures, and the other hyperparameters are fixed.

We detail our hyperparameters as follows:

We first introduce the fixed hyperparameters.

1. Decay for Taylor importance. Taylor importance[1,2] is a well-known method to measure the importance of elements, and the decay of moving average is also widely used in the literature. Therefore, we directly use the value from prior work.
2. Temperature of our diffenretial topk. The temperature is used to polarize [3] the mask of elements. This is also a widely used approach for pruning. Directly selecting a value that can polarize the mask of elements is enough. Thanks to our importance normalization, the temperature can be directly computed by closed-form, detailed in section 3.1.2. Therefore, it's also fixed.
3. Supernet size and sparsity ratio. In our opinion, the supernet size and sparsity ratio are not hyperparameters. They are set according to the user's demand and their resource amount.

Only two hyperparameters need to be turned for different tasks and architectures.

1. Resource constraint loss weight, denoted by $\lambda_{resource}$
2. Learning rate for structure parameters denoted by $lr_{structure}$

They are used to control the update of the structure parameters. The update value of a structure parameter is computed by $lr_{strucutre}\times (g_{task} + \lambda_{resource}\times g_{resource})$, where $g_{task}$ and $g_{resource}$ is the gradient of structure parameters with respect to the task loss and resource constraint loss, We provide an ablation study as follows:

The head row is the $lr_{structure}$, while the head column is the $\lambda_{resource}$. '/' indicates that the model cannot reach our target resource constraint. Obviously,

1. Smaller $\lambda_{resource}$ is better, as long as the model can reach the target resource constraint. Smaller $\lambda_{resource}$ means that the task loss takes more control of the update of the structure parameter.
2. When $\lambda_{resource}$ is small, the model is not sensitive to the change of $lr_{structure}$. When $\lambda_{resource}$ is large, a relatively large $lr_{structure}$ is better. This is because reaching the target resource constraint can reduce the influence of the resource constraint loss, as resource constraint loss is zero when the model reaches the target resource constraint.

Therefore, the setting of $\lambda_{resource}$ and $lr_{structure}$ is not difficult. We first fix $lr_{structure}$ and turn $\lambda_{resource}$ to a small value and ensure the model can reach the target resource constraint. Then, we turn $lr_{structure}$ to a relatively large value, which makes the model reach the target resource constraint in the first hundreds of iterations. Only observing the resource decrease in the first epoch is enough to set these two hyperparameters.

Compared with other NAS methods, our method uses fewer hyperparameters. For example, ModelAmplification must turn at least five hyperparameters for different tasks and models.

We added the ablation study to Appendix A.5.4 in our revised paper.

[1] Molchanov, Pavlo, et al. "Importance estimation for neural network pruning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[2] Humble, Ryan, et al. "Soft masking for cost-constrained channel pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
[3] Neuron-level Structured Pruning using Polarization Regularizer

Dear reviewer:

Thanks for your comments. Below are our responses to address your concerns.

W1 "The paper does not provide the code or the implementation details of the proposed method, which makes it difficult to reproduce and verify the results."

Considering the double-blind review policy, we can not open-source our code before the paper is accepted. However, we will open-source our code after the paper is accepted.

W2 "The paper does not explain how the layerwise mask affects the channel-wise mask in the differential topk. It is unclear how the two masks interact and whether they can be jointly optimized in an efficient way."

The layerwise mask and the channel-wise mask are independent. For example, we have a $layer$ and an input $x$. We use $m_{L_i} \in [0,1]$ to denote the layerwise mask and $m_{C}\in [0,1]^{N}$ for the channel-wise mask. The forward process is as follows: $y= m_C \times x+m_{L_i} \times layer(m_C \times x)$. After searching, we will prune layers and channels according to the layerwise and channel-wise masks, respectively.

We updated Appendix A.1.1 to make it more clear.

W3 "The paper lacks a proper control experiment to isolate the effect of the differential topk from other factors, such as the network architecture, the learning rate, and the data augmentation. It is possible that some of the improvements are due to these factors rather than the proposed method."

We think we have controlled the factors you mentioned. For architecture, we evaluate our method always with the same architecture as our baselines. For learning rate and data augmentation, without extra explanation, we use the same training setting for our method and the baseline methods. We detail the training setting in Appendix a.6. If you think where we have not controlled the factors, please let us know.

Sorry, we misunderstood your question.

In our view, your question can be divided into two parts.

1. Why do we use Taylor importance with moving average as the element importance measure method?

Anwser: Our reason is simple. Our main contribution is to propose a method to search the number of elements, rather than measuring the importance of each element. Hence, we directly select a simple and commonly used element importance metric, Taylor importance, as the element importance measure method.

2. What's the advantage of Taylor importance with moving average, compared with other metrics?

Anwser:

To be honest, we do not conduct a comprehensive comparison between different metrics, as this is not our main contribution. As a comparison, ZiCo must compare with other metrics, as they propose a new metric.

Besides, For structural pruning, some work argues that the difference between different importance metrics is not significant [1]. Note the conclusion of Zico about the importance metrics is not suitable for us, as the search granularity is different. ZiCo modifies the importance metrics to select models, while we search structure elements (channels/layers).

Although we do not conduct a comprehensive comparison, we will still provide an ablation study to compare different metrics.

[1] Huang, Zhongzhan, et al. "Rethinking the pruning criteria for convolutional neural network." Advances in Neural Information Processing Systems 34 (2021): 16305-16318.

Q8 "The gains over prior architecture search methods seem to be relatively minor, such as 0.7% accuracy on ImageNet. What would be your argument against this?"

There are two reasons.

1. The comparison is unfair, as Zico uses a stronger training script, such as distillation and mix-up. Therefore, the real improvement should be larger.
2. Our main target is not to provide SOTA ImageNet performance but to provide a general and flexible architecture search method. Our method is almost compatible with most network architectures and training processes. Therefore, we only change the depth and width of our baselines and do not use any tricks to improve performance. For our method, improving various tasks stably with limited search costs and limited hyperparameter tuning is more critical, as it is more important for real-world applications. Note, as far as we know, we are the first method that achieves 2.0% performance improvement on yolo-v8, a high-performance and lightweight model.

Therefore, we think 0.7% improvement is not minor.

Thank you for your comments again. We are looking forward to hearing more feedback from you. If you have any further questions, please feel free to contact us. We are active until the rebuttal deadline. Besides, we want to emphasize the value of our work. Our method is novel and effective, and it outperforms the state-of-the-art NAS methods using much fewer resources. It pushes the boundary of NAS and makes it more practical. Our work is valuable and deserves to be introduced to the community. Would you consider raising the score if our responses have resolved your concerns? Thank you!

Q1: "How sensitive is the method to the element importance measure $a$? It's computed as a moving average with a specific hyperparameter. It seems like the gradient estimate of $a$ depends on this being stable."

Yes, the moving average is used to make the training more stable. Here, we provide an ablation study of the element importance measure.

"Index" means assigning an importance value to each element according to its index in the sequence statically. For example, we assign 0 to the first element, 1.0 to the last. As shown in the above Table, the Taylor importance and the moving average improve the performance. As Taylor importance computes importance by mini-batch, the moving average is used to make the importance measure more stable, also making the gradient computation of $a$ more stable.

We also added the ablation study to our revised paper in Appendix A.5.3.

Q6: "In Section 3.2 you say "Compared with training-based pruning, our method eliminates the need for time-consuming pretraining since we think searching from scratch is more efficient thatn from a pretrained model...". How does that save resources? Typically pretrained models are available for free, but training one yourself from scratch is extremely expensive?"

Here, we compare the pipeline of our method with standard pruning methods as follows.

We guess that you may think the fintune stage of pruning methods is more efficient than the Retrain stage (train from scratch) of our method. However, most structural pruning methods directly use the same training setting as pretraining in fintuning stage [1,2]. Because this setting achieves the best performance. Therefore, our method is more efficient by omitting the most resource-consuming pretraining stage and does not introduce any extra cost.

Besides, the pretrained models are not free but expensive. They are free only because someone has paid for them and shared them with us. Imagine two situations:

1. We want to search for a model with a big private dataset.
2. We want to design a new model architecture.

In both situations, we can not find any free pretrained model.

Here, we provide the performance and search cost comparison between searching with pretrained models and without pretrained models.

(The unit of cost is $G MACs \times Epochs$)
Searching without pretrained models achieves 0.8% higher top-1 accuracy and only uses nearly 1/4 of the total cost of searching with pretrained models. As a result, we think our method is more efficient than standard pruning methods in both performance and cost.

[1] Li, Bailin, et al. "Eagleeye: Fast sub-net evaluation for efficient neural network pruning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.
[2] Wang, Huan, and Yun Fu. "Trainability preserving neural structured pruning." arXiv preprint arXiv:2207.12534 (2022).

Q7 "I don't understand what Table 4 means because the rows and columns refer to search and retraining, but one can't have both a pretrained model and a model that is retrained?"

We apologize for the confusion. We update the table as follows:

This table shows different initialization schemes for the search stage and retrain stage. The search stage searches a structure, and then the structure is retrained. Hence, in the retrain stage, the model is initialized randomly or with the weights from the search stage. So there is no pretrained initialization in the retrain stage.

Dear reviewer:

Thanks for your helpful comments. Here are our responses to your comments.

W1: "In Section 3.2 the authors mention that this method bears some resemblance to pruning works, "Our DMS follows a pipeline similar to training-based model pruning." This implies that the model should be compared to pruning based methods in experiments. However, the comparisons appear to be made to NAS methods, such as JointPruning. A comparison to state of the art sparse methods, such as RIGL would make the experiments more robust."

Thanks for your suggestion. We compared our method with SOTA structure pruning methods in our original paper, detailed in Appendix A.4. Our method also outperforms SOTA structure pruning methods.

There are two types of pruning methods: unstructural pruning and structural pruning. Unstructural pruning methods, such as RIGL, prune the model by setting the weights to zero. Structural pruning methods, such as our method, prune the model by removing the whole dimensions of the weight tensor. The two types of pruning methods have different pruning granularity. Unstructural pruning needs special hardware support to achieve acceleration, while structural pruning can be accelerated by general hardware. Therefore, RIGL can not be compared with our method directly. Please refer to [1] for more details.

Besides, we want to talk about the boundary between structure pruning and NAS. In fact, they are not two independent fields. The classic NAS method, darts [2], is very similar to pruning, and the pruning work, EagleEye[3], also searches model architectures. Our method can be interpreted as a structure pruning method and a NAS method simultaneously. Propose it as a nas method because we want to emphasize our method's high structure search efficiency rather than discovering better initialization from pretrained models.

[1] Li, Hao, et al. "Pruning filters for efficient convnets." arXiv preprint arXiv:1608.08710 (2016).
[2] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "Darts: Differentiable architecture search." arXiv preprint arXiv:1806.09055 (2018).
[3] Li, Bailin, et al. "Eagleeye: Fast sub-net evaluation for efficient neural network pruning." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.

W2: "The function they have constructed for optimization is smooth but saturates outside of the active regions illustrated in Figure 2. This may cause vanishing gradient information. Any experiment to investigate whether this happens during training, or why it doesn't happen would be valuable."

Through our deduction and observation, we think gradient vanishing does not happen in our method. We show our deduction as follows.

Figure 2 illustrates the gradient of $a$ with respect to element mask $m_i$. Here, we denote it as $\frac{\partial m_i}{\partial a}$. It's indeed that $\frac{\partial m_i}{\partial a}$ may equal to 0 when $m_i$ is close to 0 or 1. But the gradient of $a$ with respect to the $task\_loss$ is $\sum_{i=0}^{N}{\frac{\partial task \_loss}{\partial m_i}\frac{\partial m_i}{\partial a}}$. As shown in Figure 2, $\frac{\partial m_i}{\partial a}$ will not be 0 for all $i \in [0,N]$. Therefore, the gradient of $a$ has a low probability of being 0.

Besides, we also do not observe gradient vanishing in our experiments. Therefore, we think gradient vanishing does not happen in our method.

W3: "The relationship between resource constraint and the top-k parameters is in the appendix but it's extremely important to the overall algorithm."

We apologize for the inconvenience. As many papers have used the resource constraint[1,2], we move the section about the resource constraint to the appendix when the paper is too long.

Thanks for your advice. We will consider moving the section to the main paper.

[1] Li, Yanyu, et al. "Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization."
[2] Gao, Shangqian, et al. "Disentangled differentiable network pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

W4: "The top-k operator as described leads the reader to assume would be fixed but in practice it's not constrained and can be any value. Really it's just a binary mask that has a soft constraint to sum to a low enough value to meet the resource constraints."

We are sorry for the confusion. We agree that the 'k' is usually fixed. The reason we use the term "top-k" is because we just follow prior work. For example, [1] uses "soft top-k" in their method with an unfixed 'k'. We will consider more carefully when using the term "top-k" in the future.

[1] Gao, Shangqian, et al. "Disentangled differentiable network pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Dear reviewer:

Thanks for your constructive comments. We have posted our responses to your comments. We expect your feedback about whether our responses address your concerns, or if you have any further questions. We are glad to answer them and improve our paper.

Best,

Authors

I apologise for not replying sooner.

W1: I missed Appendix A.4 when reading this paper for the first time. It does appear that the method compares well in these experiments.

W2: Figure 2 does show regions of zero gradient. Other methods that use this kind of relaxation add noise (most famously Gumbel softmax) to smooth out the gradient (with noise sometimes it will land in the high gradient region). This appears to involve a fixed mask, perhaps the noise from SGD is enough to maintain gradient information. If, as you say, "we also do not observe gradient vanishing in our experiments" then it may be worthwhile adding those results, for example a gradient hook showing the norm of the gradient passing through the soft mask versus the gradient norm due to the resource loss would be valuable. It seems like the method requires these two parts to be balanced carefully via $\lambda_{resource}$.

Q1: I find it concernging that the Taylor importance is not critical to the performance of the method. Why would it still work within 1% accuracy without it? The importance signal tells the model which parts are important. Is the method in this analogous to Nested Dropout, ie the network reacts to the importance of different elements prescribed by the mask and not the other way around?

Q6: I don't understand how you can end up with a trained model here without training it; you're comparing pretrained models to your search method and the cost is 116 for the larger model using your search method? And it performs better? I apologize I don't understand what is happening here and I will update my confidence in my review.

Q7: See response to response Q6 above.

Q8: Unfortunately, I can't conclude that the method would have worked better if it had been trained differently. It would be necessary to replicate ZiCo and train it without the extra tricks they used for a fair comparison.

Dear reviewer:

Thanks for your nice comments and suggestions. We have revised the paper according to your suggestions. The following are our responses to your comments.

W1: "Search time comparison in some cases seems unfair/confusing (refer to questions)"
Q1: "Search time comparison -> Since the observation from section 5 show that initialization from pre-trained models is very useful for differential scaling, did the authors include this in the search time computation. If a method relies on pre-trained models, then ideally the pre-training cost is a part of the total cost incurred. Could the authors clarify the intialization scheme used in each of the tables ie. table 1,2,3."

We are sorry for the confusion. To clarify, we did not load pretrained models by default. We only load pretrained models in Table 3 (LLM experiments), Table 4 (ablation study), and Table 8 (comparison with other pruning methods). Therefore, we did not include the pretraining cost in Table 1 for our method as we did not use pretraining, and we think the comparison is fair for other methods. We chose this setting in experiments because, based on our ablation study, we found searching from a large, randomly initialized supernet reaches higher accuracy but lower cost than searching from a small pretrained supernet. Details are in the table below.

We have updated this table to section 5 to make our conclusion more clear.

W2: "Since the problem is cast as a NAS problem the search spaces used are not the ones very traditional to NAS (refer to questions)"
Q2: "In the appendix the authors compare with one-shot methods like OFA [1] . The comparison in my opinion is unfair since the search is performed on different search spaces. Could the authors evaluated the method on the exact same search space as OFA? This would help differentiable the gains of the search-space v/s the method itself? Similarly could a comparison be made with the AutoFormer [2] by evaluating the method on its exact search space [2]?"

There are two main differences between the search space of our method and that of OFA and Autoformer.

1. What is more fine-grained in OFA's search space: We (and Autoformer) only search for the depth and width of networks, while OFA also searches for the resolution and kernel size. This is because we want to build a general method that can be applied to NLP and CV tasks. The resolution and kernel size are not necessary for NLP tasks.
2. What is more fine-grained in our search space: We can search width and depth with step 1 due to our high search efficiency. However, OFA and Autoformer have to use large steps and min-max limitations to compress their search space. This is an advantage of our method, as our search space is easier to define and much larger.

Our current submission uses the default setting directly, causing different search spaces. To make the comparison more fair, we select Autoformer-T as our baseline and use the exact same supernet and search space from Autoformer-T. The performance and estimated search cost are shown below.

Using Autoformer's search space, our method outperforms Autoformer-T by 0.5% with less than 10% of the search cost. It proves the gain of our search method is from itself.

We have also added this ablation study to Appendix A.5.2 in our revised paper.

W3: The search needs to be repeated for every resource constraint and obtaining a Pareto-Front of objectives might be very expensive (unlike methods like OFA[1] which directly approximate the whole Pareto-front).

Yes, we agree with your opinion. Our method is not suitable for searching for the Pareto-Front.

However, we want to emphasize our target is to build a general and flexible NAS method. There are several situations where our method is more suitable than OFA.

1. Searching for a small number of models, which is the usual case in the real world.
2. Searching extremely large models. For example, we prepare to train an LLM with 100B parameters using all our GPUs for 30 days. It's impossible to train a supernet (by OneShot NAS, like OFA), which needs much over 30 days, such as 60 or even 300 days. But using much less than 30 days, such as 1 day or 3 days , to search for it with our method is acceptable.

Last, thank you for your helpful reviews! Looking forward to hearing back from you!

Dear reviewer:

Thanks for your constructive comments. We have posted our responses to your comments. We expect your feedback about whether our responses address your concerns, or if you have any further questions. We are glad to answer them and improve our paper.

Best,

Authors