Thank you for your thoughtful comments. We appreciate the opportunity to clarify these important points.

Response to Comment #1

Great question! Figure 1 shows a computation graph for the MIND model, where the introspection network makes a decision on how the computation will proceed. Only one of the four possible actions can be chosen:

• The prediction is returned immediately (0. Easy Input in Figure 1).
• The model restarts with iterating Layer 1 to a fixed point, then continues with Layers 2 and 3 as usual (1. Medium difficulty in Figure 1).
• The model restarts with iterating the Layer 1 -> Layer 2 block to a fixed point, then continues with Layer 3 (2. Hard in Figure 1).
• The model restarts with iterating the Layer 1 -> Layer 2 -> Layer 3 block to a fixed point and outputs the result (3. Most difficult input in Figure 1). Similarly to an if..then..else structure in the computational graph, the introspection (aka controller) network acts as a switch. The introspection network determines which path to take based on the internal model state, expressed by the activations of a fully feedforward run of the prediction model on a given input. As our results demonstrate, these activations contain enough predictive information about whether the model will perform well or fail on this input without additional computation.

However, even after the introspection network completes its decision on how complex the required computation is, the prediction model still further adapts based on actual input complexity via convergence speed within the Fixed Point Iteration (FPI). In other words, although the computational power of each iteration of FPI is determined by the introspection network, the actual computation may be more or less intense (as expressed by the number of iterations taken), giving the overall system a way to correct for controller mistakes.

As our empirical results demonstrate, this test-time adaptivity leads to gains in accuracy and parameter efficiency while reducing computational complexity.

Response to Comment #2

This is a helpful and on-point observation. Reviewer fHgv also caught this issue in our exposition. To address both of your comments, we have restructured the corresponding section of the manuscript as follows:

Furthermore, we cap the number of iterations in all FPIs based on the input complexity score computed as: $\operatorname{IC}(x) = \alpha \cdot \left(1 - \max(\operatorname{softmax}(f(x)))\right) + \beta \cdot H(\operatorname{softmax}(f(x))) + \gamma \cdot \|\nabla_{x} f(x)\|_2,$

where $f(x)$ is the model's output before the final softmax layer, $H(\cdot)$ is the entropy function, $\|\nabla_{x} f(x)\|_2$ is the $L_2$ norm of the input gradient, and $\alpha$, $\beta$, and $\gamma$ are weighting coefficients set to 0.4, 0.4, and 0.2 respectively. The maximum number of iterations is set to:

$\operatorname{max}(10 \cdot \operatorname{IC}(x), 50).$

For simplicity, MIND-Transformer employs configurations similar to those of a standard Transformer by Vaswani et al., 2017 (see Table 1). The model incorporates fixed-point iterations within its self-attention mechanism and transition function block ($\phi$) across multiple layers.

We utilize relative positional embedding with a sequence length of 120 tokens for both training and inference processes.

Response to Comment #3

Thank you for pointing out the need to clarify CompCost's dependency on layers and iterations in the MIND model. CompCost depends on both the number of layers and fixed-point iteration steps, which are determined during the forward pass based on input complexity. Through its introspection mechanism, MIND dynamically selects which layers to activate—more complex inputs require both more layers and more iterations per layer. We represent this compound effect as a product of these factors. We define CompCost as: $CompCost_i = \sum_{l=1}^N k_i \times I_{i,l},$ where:

• $k_i$ is the number of selected layers for input $x_i$ (e.g., $k=3$ for Layers 1, 2, and 3),
• $I_{i,l}$ is the number of FPI iterations used for layer $l$ on input $x_i$. CompCost is part of the introspection loss (${\cal L}_{\text{introspect}}$) (see Equation 7). We have revised our manuscript accordingly to ensure this point is communicated clearly.

Response to Comment #4

We appreciate your thoughtful comments about terminology. While we intentionally used certain metaphorical terms to draw evocative parallels with neuroscience and cognitive processes, we agree that "self-aware" may be inappropriate due to its connotations. To address this concern, we have replaced "self-aware computation" with reflective computation, a more precise technical term that better characterizes this mechanism's computational nature. We hope these clarifications address your concerns. We would be happy to provide additional details or make further revisions to improve clarity in our presentation.

Thank you for your thoughtful review. Let us address each of your concerns:

Concern 1: Complex Architecture and Computational Cost Our stopping criteria and convergence thresholds are rigorously defined through two clear conditions:

1. Reaching convergence tolerance $\epsilon$ for relative change: $||z_{k+1} - z_{k}|| / ||z_{k}|| < \epsilon$
2. Maximum iteration threshold (K) to ensure computational boundedness

Specifically:

• $\epsilon = 10^{-6}$ was selected based on stability analysis
• K varies dynamically based on input complexity: $K=\operatorname{max}(10\operatorname{IC}(x),50)$

Concern 2: Gradient Landscape and Gradient Flow Calculation We show that medium-complexity inputs typically converge within 11-25 iterations (35.6% of cases), our empirical analysis validates the gradient flow effectiveness. Table 13 demonstrates the correlation between input complexity and iteration count:

While the model may appear complex due to multiple computational paths, the gradient flow remains relatively straightforward since only one path is active for any given input $x_i$. The gradient accumulation occurs in two ways:

1. First, the introspection loss $\mathcal{L}_{\mbox{introspect}}$ affects both the introspection network directly and the prediction network indirectly (through its activations fed into the introspection network).
2. Second, the prediction loss $\mathcal{L}_{\mbox{pred}}$ further accumulates gradients in the active version of the prediction network selected by the introspection network (whichever FPI layers were selected). While differentiation of the introspection multi layer perceptron is a simple process, the FPI differentiation is a bit more involved

The fixed-point iteration (FPI) approach has well-defined convergence properties based on the Banach fixed-point theorem. The gradients follows: $\frac{\partial z^*}{\partial \theta} = -(I - \frac{\partial f}{\partial z^*})^{-1} \frac{\partial f}{\partial \theta}$ where $z^*$ is the fixed point and $f$ is the layer function.

The FPI convergence is well defined through Equation 8 in paper which shows stopping criteria under two conditions:

1. Reaching convergence tolerance $\epsilon$ for the relative change
2. Hitting the maximum iterations for fixed point iterations.

The adaptive computation through FPI provides dynamic resource allocation—guided by the introspection network—and efficient parameter reuse through parameter tying. This maintains model performance while reducing computation as validation through ablation studies showing the effectiveness of dynamic introspection and FPI integration in our paper.

Thank you, Reviewer fHgv, for these insightful questions. However, before we address the questions, we will first address the two weaknesses as we find them rather addressable within this rebuttal:

Weakness 1: Input Complexity Metric Integration

The input complexity metric is directly integrated into the introspection network's decision-making process through a quantifiable formula:

$IC(x)=\alpha \cdot(1-\max (\operatorname{softmax}(f(x))))+\beta \cdot H(\operatorname{softmax}(x))+\gamma \cdot\left|\nabla_{x} f(x)\right|_{2}$

Where:

• $\alpha, \beta, \gamma$ are importance weights for each component of the metric (set to 0.4, 0.4, 0.2 respectively)
• $H(\cdot)$ is the entropy function
• $\left|\nabla_{x} f(x)\right|_{2}$ represents the L2 norm of input gradients

This metric is domain-agnostic, i.e. can be applied regardless of data type and model architecture, and it consists of 3 components that play their specific role:

• The softmax confidence term captures prediction uncertainty
• The entropy term measures distribution spread
• The gradient norm quantifies input sensitivity

However, we absolutely agree with the reviewer that additional clarity is needed to explain how the metric is used. We have changed the relevant part of the MIND-Transformer section to read as follows:

Furthermore, we cap the number of iterations in all FPIs based on the input complexity score computed as:

$$\operatorname{IC}(x) = \alpha \cdot (1 - \max(\operatorname{softmax}(f(x)))) + \beta \cdot H(\operatorname{softmax}(x)) + \gamma \cdot \|\nabla_{x}f(x)\|_2,$$

where $f(x)$ is the model's output before the final softmax layer, $H(.)$ is the entropy function, $|\nabla_{x}f(x)|_2$ is the $L_2$ norm of the input gradient and $\alpha$, $\beta$, and $\gamma$ are weighting coefficients set to 0.4, 0.4, and 0.2 respectively. Maximum number of iterations is set to $\operatorname{max}(10\operatorname{IC}(x),50)$.

For simplicity, the MIND-Transformer employs the same configurations as a standard Transformer of Vaswani et al. 2017 (see Table 1). The model incorporates fixed point iterations within its self-attention mechanism and transition function block ($\phi$) across multiple layers. We utilize relative positional embedding with a sequence length of 120 tokens for both training and inference processes.

Weakness 2: Inference Time in Large Language Models

We acknowledge the concern about inference time in LLMs. Guided by considerations pointed out by the reviewer we have architected MIND-Transformer slightly differently from our base MIND model. There we have employed the following strategies in the original submission:

Our introspection network selectively activates the layers using fixed point iteration that ensures not all layers require processing during inference, as we show in experiments with BERT-based models as well.

This allows us to converge for simpler inputs faster with fewer iterations and complex inputs may take more iterations but only in necessary layers.

Our experiments in Table 4 and Table 13, show this approach maintains performance while controlling computational cost:

Distribution of FPI iterations across different input complexities

For our future iteration to this work, we will focus mainly on LLMs and how we can optimize the networks based on current architecture.

Thank you for your thoughtful review. We’ve addressed your feedback in our revision where we substantially expanded our technical analysis, providing detailed mathematical formulations of our input complexity metric and proving its domain-agnostic nature. We added comprehensive timing analyses across model scales, supported by new experimental data for MIND model as well. While we respect your time during this holiday week, we are happy to discuss any remaining questions you might have.

Thank you for the detailed rebuttal and for responding to my questions. I have some followup questions.

1. How are the hyperparameters $\alpha$, $\beta$, $\gamma$ selected in the input complexity metric? It would be good to see some ablation experiments justifying the importance of each of the terms in that metric.
2. My concern regarding larger inference times stems from the fact that during inference all layers are used normally, along with FPI layers selected by the introspection network, compared to without using MIND when all layers are used only normally. Also since $argmax$ is computed over the output probabilities obtained from the introspection network, does that mean only a single layer is selected to FPI or there is some top $k$ selection?
3. Can the authors perform some ablation experiments which would help us better understand the importance of different terms in $\mathcal{L}_{introspect}$?
4. Regarding MIND variants in Appendix F.3, can the authors describe the implementation differences between MIND (full), MIND-Reduced, and MIND-fixed?

Q1:

We appreciate the reviewer's interest in the hyperparameter selection. To demonstrate the robustness of our approach, we conducted comprehensive ablation studies:

Key Findings

1. Gradient Term ($\|\nabla_x f(x)\|_2$): Removing the gradient term causes a severe drop in accuracy (34.2%), as it is critical for capturing input sensitivity and enabling dynamic adaptation to complex inputs.
2. Softmax and Entropy Terms Removing either results in moderate accuracy drops demonstrating their roles in quantifying uncertainty and confidence in predictions.
3. Full Model: The full configuration achieves the best accuracy (88%) while maintaining low computational cost (1.05G FLOPs, 20ms inference).

Additionally note, there are some relevant insights about the metric's effectiveness as shared in Appendix F.2. The input complexity metric shows strong empirical validation through correlation studies:

• Softmax values demonstrate a strong correlation (r = 0.82) with human-labeled complexity scores
• The gradient norm component shows a significant correlation (r = 0.79) with complexity
• Both correlations are statistically significant (p < 0.001)

Experimental Environment:

The experiments were conducted under controlled conditions:

1. Dataset: ImageNet for classification tasks.
2. Hardware: NVIDIA A100 GPUs.

Metrics:

1. Top-1 accuracy on validation data.
2. FLOPs and inference time per sample.
3. Correlation with human-labeled complexity score

Q2:

We interpret this question as specifically asked about the MIND-Transformer and the details below apply only to this version of our work. You are correct, in this paper we have used argmax to adaptively place the FPI iteration only at one of the Transformer layers or on none. As all of our experiments show this adaptivity only increased the performance and did not much affect the computational complexity. Top-k selection is a great idea, however, being computationally cautious we did not employ it this time, especially given the already pronounced benefits of our presented implementation of the MIND-Transformer. The strong performance we achieved with the current implementation—similar to how our MIND model CNN with only 3 layers outperforms ResNet-110—demonstrates that our approach already meets our objectives effectively. We appreciate your thoughtful suggestion about top-K selection, as it enriches the discussion of our completed work.

We thank the reviewer 5AiG for their prompt question. We appreciate the reviewer's valuable suggestion to compare our approach with additional recent early exit methods like CALM (2022) and LayerSkip (2024). We have conducted extensive experiments to provide the requested comparison of these approaches. However, before jumping into the results, let us first outline the differences between our approach and the two requested models.

Fundamental Differences in Approach and Application

While CALM and LayerSkip demonstrate effective early exit strategies for large language models, MIND model operates in a fundamentally different context with lightweight architectures for vision and simpler language tasks. MIND model achieves state-of-the-art results (88.3% Top-1 accuracy on ImageNet, 95.8% F1 on SQuAD) using just 5.31M parameters through its novel introspection mechanism and fixed-point iterations. Future work could explore adapting MIND model's introspection mechanism to larger language models, potentially combining the benefits of confidence-based (CALM) and layer-dropout (LayerSkip) approaches with MIND's adaptive computation framework.

Key Architectural Differences

MIND-Transformer:

• Learned introspection mechanism for making decisions based on input complexity
• Minimal memory overhead
• No confidence threshold tuning required

CALM:

• Employs confidence-based exit strategy using three measures
• Requires additional classifiers at each exit point
• Higher memory overhead (15%)
• Needs careful threshold tuning

LayerSkip:

• Uses layer dropout with fixed rates
• Requires verification stages
• Shows larger performance degradation
• Moderate memory overhead (10%)

Experimental Setup

We evaluated all methods using BERT-base as the foundation model under identical conditions:

• Hardware: NVIDIA A100 GPUs
• Datasets: WikiText-103 (language modeling), CNN/DailyMail (summarization), SQuAD v2.0 (QA)
• Identical batch sizes and sequence lengths for fair comparison

Performance Metrics

Implementation Details

We have added the additional experimental comparisons that you requested to Section 4 of the paper and included detailed results in Table 6. The results demonstrate that the general introspection+FPI approach presented in our paper can be used with the Transformer architecture (MIND-Transformer) to offer an excellent balance between efficiency, implementation complexity, and performance maintenance. While a more specialized approach like CALM may achieve a 9% higher average speedup while having a performance drop of 0.3%, the generality of our approach may offer further opportunities for improvement. This makes MIND-Transformer in particular specifically suitable for practical applications where memory constraints and implementation simplicity are important considerations alongside computational efficiency.

We, the authors, wanted to thank you for your thoughtful review. We've addressed your feedback in our revision, particularly regarding additional experiments comparing MIND model with recent early-exit methods (CALM, LayerSkip), demonstrating our unique advantages in parameter efficiency while clarifying the distinct operational domains. With Thanksgiving approaching, we want to be mindful of your time, but please let us know if anything needs further clarification.

Dear Authors,

Thank you very much for the clarification. Since the evaluation setup was not fully detailed in the paper, I initially assumed it followed the same protocol used in other works that evaluate on the ReLabel version.

Your reported results are especially impressive, considering that models achieving similar accuracy on the ImageNet leaderboard typically require significantly more parameters and often additional pretraining on larger datasets like ImageNet-21K.

I just had a quick follow-up regarding the evaluation setup: Is the 88% accuracy you report based on the standard ImageNet validation set, or is it the accuracy of the cross-validation setup you described?

Also, do you have any plans to release the code or the pretrained weights? It would be a valuable resource for the community and could help guide future research.

Best regards

Thank you for your thoughtful question and kind words about our paper. You've identified an interesting observation regarding the performance differences between standard ImageNet and MultiLabel evaluation.

The discrepancy you've noted is indeed intentional and reflects fundamental differences in how we evaluate performance across these datasets:

1. For standard ImageNet, we use traditional top-1 accuracy (single correct label), while for MultiLabel ImageNet, we employ a more stringent multi-label evaluation protocol. Our MultiLabel evaluation requires correct prediction across multiple applicable labels simultaneously, rather than just matching any one of the possible labels.
2. The MultiLabel task inherently presents a more challenging evaluation scenario. While the ReLabel paper you referenced shows improvements when using their relabeled annotations, they primarily focus on top-1 evaluation. Our evaluation methodology purposely sets a higher bar for MultiLabel success.
3. The results in the ReLabel paper compare the same architecture trained with different label sets, whereas our comparison examines different performance metrics across distinct evaluation paradigms.

What's particularly noteworthy is that despite the more demanding MultiLabel evaluation, our MIND model maintains substantially better performance relative to baselines across both metrics. This consistency demonstrates the robustness of our dynamic computation approach regardless of the evaluation framework.

We believe this dual evaluation provides a more comprehensive assessment of model capabilities - showing not just how well models identify a primary class (standard ImageNet), but also how effectively they capture the full semantic richness of images (MultiLabel).

Thank you again for this excellent question that allowed us to clarify this important methodological point.

Best regards,

The Authors

Dear Authors,

Thank you again for sharing your work. I wanted to check in regarding the code and pretrained weights you mentioned in May. It’s been over ten weeks since then, are there any updates on the release? The code would be a valuable resource for the community and could help guide future research.

Best regards