Powering Neural Architecture Search with Robust Masked Autoencoders

Q1: There is a lack of discussion and understanding of how NAS and MAE synergize together.

A1: The motivation comes from a crucial observation: supervised neural architecture search often yields the final models that overfit the training data. In other words, regardless of how we optimize $alpha$ and $w$ in Equation 1, the searched models consistently achieve near-zero training errors. However, the ultimate goal of NAS is to identify architectures that exhibit strong generalization performance on the validation set. This presents an inherent contradiction in supervised NAS. With this perspective in mind, we propose leveraging MAE as the proxy task for NAS. In this way, MAE-NAS seeks to discover models with enhanced generalization capabilities in the unsupervised paradigm.

our work is the first to explore the MAE paradigm for unsupervised NAS. In theory, it can serve as the proxy task for any NAS methods.

Q2: Some doubts why this particular decoder design works the best.

A2: Please refer A3 to Reviewer PEu7.

Q1: The motivation for adopting masked autoencoders to achieve unsupervised NAS is unclear.

A1: Please refer A1 to Reviewer jWWf.

Q2: The methods combined with mask autoencoders are old.

A2: DARTS is the pioneering work of differentiable NAS, and many subsequent methods are its variations. This method is clean and elegant, and combination with it can fully reflects the true effect of the proposed proxy task. Additionally, we conducted detailed comparison with recent SOTA differentiable NAS including supervised and unsupervised methods in Table 2 and Table 3, which demonstrate our approach is competitive enough in terms of performance.

Q3: Some claims are not well supported in terms of the performance collapse.

A3: The root cause of performance collapse lies in the design of the search algorithm and the search space, and the specific reasons have been widely studied before. It is attributed to unfair competition between skip connections and other operations, resulting in unstable training of the supernet. Additionally, the experiments in Table 8 can support our claim.

Q4: This phenomenon is conflict to the claim “when the mask ratio is less than 0.5, the probability of collapse is significantly high”.

A4: This is not contradictory. The experiments in Table 7 are conducted with HD enabled by default, so there is no performance collapse issue at all, and the test error is within the normal fluctuation range. In Table 8, at the setting of the small mask ratio without HD, the training error and the number of skip connections both increase, which is a clear signal of training collapse.

Q5: The writing needs improvements.

A5: Thanks! We will fix the format of the references in the revised version.

Q1: Technical novtiy is not significant and improvements are marginal.

A1: Please refer A1 and A2 to Reviewer qtpr.

Q2: Supporting experiments to confirm training collapse; How HD addresses this issue.

A2: Training collapse is a long-standing issue in the DARTS-based methods. Prior to our work, many methods such as PDARTS and RLNAS have been proposed to address the problem. Specifically, the algorithms will converge to the local optima and the searched architectures have lots of skip connection operations. As shown in Table 8, MAE-NAS without the HD module leads to serious performance collapse (more than 5 skip connections) in the setting of small mask ratio. By increasing the mask ratio or applying HD, the training collapse issue can be resolved.

Q3: What is the design logic behind the proposed HD?

A3: In depth, we think that single-scale algorithm cannot learn multi-scale features well. The multi-scale structure has been a universal paradigm in computer vision. For instance, the pyramid networks cope with variations in object scales by its hierarchical design. Masked modeling is originally applied to transformers in a single-scale manner. Simply transferring it to convnets will lose the advantage of model hierarchy. Given that this work explores convnet-style search spaces, hierarchical decoder becomes a natural choice. On the one hand, it helps stabilize training by enhancing gradient propagation. In Figure 3 (supp.), the hierarchical decoder leads to the smoother convergence curve and the lower training loss. On the other hand, it allows to learn more robust hierarchical features without intensive hyper-parameters tuning, enabling us to discover stronger vision backbones. In Table 8, the searched models achieve the lower Top-1 error at different settings by applying hierarchical decoder.

Q4: Minor: The format of the citation should be revised.

A4: Thank you. We will fix the format of the references in the revision version.

Q1: The novelty or contribution may not be sufficient.

A1: To our best knowledge, we are the first to explore the MAE paradigm for NAS in unsupervised setting, and it's not straightforward to make it work. Direcly applying it suffers from the performance collapse issue like DARTS. We couple the masked autoencoder's objective with our proposed Hierarchical Decoder to address the collapse issue in DARTS and its variants (Table~8).

While other unsupervised proxy metrics each have their limitations and constraints in application scenarios, our method is neat and doesn't have these limitations.

Q2: The performance gain on top of existing methods has been limited.

A2: The relative improvements brought by MAE-NAS are comparable to the SOTA NAS methods (RLNAS and DARTS-). Moreover, we would like to emphasize that the performance of NAS algorithms depends on the search space. After a period of development, existing methods have nearly reached the upper limit of performance within the DARTS search space. Under such circumstances, we believe that significant improvements are unrealistic.

In addition, various indicators such as Hessian eigenvalues have been proposed as a signal to stop searching before the performance collapses. However, these indicator-based methods tend to easily reject good architectures if the thresholds are inappropriately set. In contrast, we propose a new unsupervised paradigm, which is more elegant. Beyond addressing the issue of performance collapse, the primary advantage of our approach is that it requires no labels during the search phase. This significantly 060 broadens the potential use cases and the scope of MAE-NAS.

Q3: MAE-NAS works for convolutional search space only and has not shown promise in searching in transformers spaces.

A3: We emphasize that this work doesn't care the problem of pre-training efficiency. Our motivation is to discover superior model architectures by taking MAE as the proxy task of NAS. Coincidentally, the mainstream search spaces are primarily based on CNNs. Prior to this, Spark has demonstrated that MAE can be effectively applied to CNNs and performs well.

Q4: Presentation. Section 3 includes 1 subsection 3.1 only.

A4: Thanks! We will fix the issue in the revision.

Q5: Is it possible to design a new search space on top of MAE training so that masked patches can still be dropped during pre-training?

A5: In our implementation, the positions of masked patches are simply set to 0, which is actually equivalent to discarding these patches like the transformer. To further make the convolution efficient, these masked patches can be completely discarded without participating in any computations by applying sparse convolution. We will leave it as the future work.

Q6: Does the method apply to other domains such as 3D networks ?

A6: Currently, we have only conducted experiments on 2D CNNs and have not yet performed experiments on 3D networks. We believe it is promising and will consider it as the future work.