LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order

W1. The study primarily evaluates a low-level task, specifically classification, which raises the question of whether similar performance trends would hold for more complex tasks. It would be helpful if the authors could provide results for additional tasks, such as segmentation or regression, to gauge the method’s broader applicability.

W2. While the evaluation results on ImageNet show an acceptable performance drop relative to the baseline, Table 1 indicates a more substantial performance drop on the CIFAR-100 dataset. This suggests that performance may correlate with task complexity, though the current study does not provide sufficient insight into this relationship.

W3. I find it challenging to conceive of grounded examples for S2. This may be due to my limited expertise in distributed systems. Could the authors provide examples illustrating practical applications of the approach? Specifically, how does the robustness in arbitrary layers contribute to these applications?

W4. Some captions lack sufficient explanatory information. For instance, in Figure 1(b), it’s unclear what “ReducedRetrain” represents—is it the original model? Additionally, in Table 1, does the italicized font indicate the second-best performance?

1. The concept is interesting; however, the argument that the computing machine can fail at a time does not convince me to develop this method. The paper has weak motivation.
2. The paper evaluation is not sufficient.
3. There are several Transformer architectures which have not been evaluated. Currently, the evaluation is only done on ViT/DeiT, which is not convincing to me.
4. Hyperparameter study is not done.
5. Accuracy loss is significant (20%). This accuracy gap is very substantial.
6. I have particular concerns about Figure 1c. It is unfair to compare layer-drop or reduced depth with layer shuffle configuration at deployment because the two baselines are not trained with layer shuffling order. This figure is entirely misleading and should be dropped.
7. The only change made in the transformer block is the layer index, which is not a significant contribution.
8. Please include several recent transformer-based baselines such as Swin, EfficientVit, etc.

Although there are some spotlights in the paper, there are still some unclearnesses that need to be clarified:

• Novelty is limited. The key idea of the paper is to shuffle the layer order of a model during training so that the model is robust to different layer orders.
• Model performance drops. Line 285 says the model performance with LayerShuffle is “slightly” lower than the one without LayerShuffle. However, the degradation is over 28% on the simple classification task according to Table 1 (CIFAR, sequential, LS-pred).
• LS and LS-Pos are almost the same according to Table 1. LS-pred is always the worst. Is it necessary to have three variants without performance improvement?
• More experiments on different model architectures. It would be great if the authors could conduct experiments with 2 more sota models.
• More experiments on different tasks. It would be great if the authors could conduct experiments on 2 more tasks.
• More experiments on different datasets. It would be great if the authors could conduct experiments on 2 more datasets.

I carefully reviewed the paper, and rather than addressing an extensive list of less critical issues, I will concentrate on the most significant concerns that influence my rating. My focus will be on the following two points:

1. The reduction in performance, which is around or above 20%, is substantial. This limits the useability and effectiveness of the method, making motivation unclear. In what practical real-world application is it preferable to run vision transformers, in particular the ones used in the paper (ViT and DeiT) in distributed setups where one could tolerate 20% loss? Energy consumption optimization does not seem to be a motivator since there is an added cost of data transfer across nodes in a distributed setup. Instead of layer shuffling, layer dropping or just using smaller models might be more advantageous alternatives. Why would one use "any sequence of layer execution"?

2. Robustification to changing the order of execution and shuffling has been investigated in the past, for instance, as discussed in the paper by instance-wise layer reordering (ICLR 2020), permutation invariance (many works here), LayerDrop (ICLR 2020), RandConv (ICLR 2021), and ShuffleTransformer (last two can be added to related work). As mentioned, auto-regressive models leveraging self-attention across tokens, including LLMs, are very robust to changing the positions of adjacent layers or the underlying layers from the model. After patch embedding, ViT also follows a similar architecture. Besides, a recent work, ShuffleMamba discusses a similar stochastic layerwise shuffle regularization in training. Considering this, the novelty of the method becomes borderline. I am open to reconsidering my rating based on a satisfactory rebuttal from the authors.